Training method of centroid labeling model and training method of object segmentation model

Abstract

The application discloses a centroid labeling model training method and an object segmentation model training method, and relates to the technical field of artificial intelligence. The method comprises the following steps: inputting a centroid training image into an initial centroid labeling model which is not trained completely, to obtain output predicted centroid data; determining an offset loss function based on the predicted centroid data and standard centroid data, and determining a centroid loss function based on the offset loss function; adjusting model parameters of the initial centroid labeling model based on the centroid loss function, until the centroid loss function converges, to obtain a target centroid labeling model which is trained completely; and the offset loss function comprises a distance loss function and / or a direction loss function. The embodiment of the application solves the problem of low efficiency of manual centroid labeling, improves the accuracy of centroid labeling, and further improves the training effect of a subsequent object segmentation model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a training method of a centroid labeling model and a training method of an object segmentation model. BACKGROUND

[0002] The centroid refers to a hypothetical point on a material system where the mass is considered to be concentrated, and the centroid plays an important role in determining the stability of an object, the inertia of an object, the mechanical properties of an object, and the segmentation of an object in engineering applications.

[0003] At present, the centroid of a labeled object in an image is labeled by relying on manual labeling, but manual centroid labeling consumes a lot of labeling time and has poor accuracy. SUMMARY

[0004] The embodiments of the present application provide a training method of a centroid labeling model and a training method of an object segmentation model to solve the problem of low efficiency of manual centroid labeling, improve the accuracy of centroid labeling, and further improve the training effect of a subsequent object segmentation model.

[0005] According to an embodiment of the present application, a training method of a centroid labeling model is provided, which comprises:

[0006] inputting a centroid training image into an initial centroid labeling model that is not trained to obtain output predicted centroid data;

[0007] determining an offset loss function based on the predicted centroid data and standard centroid data, and determining a centroid loss function based on the offset loss function;

[0008] adjusting model parameters of the initial centroid labeling model based on the centroid loss function until the centroid loss function converges, and obtaining a target centroid labeling model that is trained;

[0009] The offset loss function comprises a distance loss function and / or a direction loss function, the distance loss function represents a loss amount of an offset distance of a centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents a loss amount of an offset direction of the centroid candidate point pair corresponding to the predicted centroid data.

[0010] According to another embodiment of the present application, a training method of an object segmentation model is provided, which comprises:

[0011] inputting a centroid test image into a target centroid labeling model that is trained in advance to obtain output target centroid data;

[0012] acquire at least one object training image block in the centroid test image based on target centroid coordinates corresponding to at least one centroid target point in the target centroid data;

[0013] input each of the object training image blocks into an initial object segmentation model that is not trained completely to obtain a predicted object image output;

[0014] adjust model parameters of the initial object segmentation model based on the predicted object image and a standard object image to obtain a target object segmentation model that is trained completely;

[0015] The target centroid labeling model is obtained by using the training method of the centroid labeling model according to any one of the embodiments of the present application, and the number of labeled objects contained in the standard object image is less than or equal to the number of segmented objects in the predicted object image.

[0016] According to another embodiment of the present application, a training device of a centroid labeling model is provided, which comprises:

[0017] a predicted centroid data output module configured to input a centroid training image into an initial centroid labeling model that is not trained completely to obtain predicted centroid data output;

[0018] a centroid loss function determination module configured to determine an offset loss function based on the predicted centroid data and standard centroid data, and determine a centroid loss function based on the offset loss function;

[0019] a target centroid labeling model determination module configured to adjust model parameters of the initial centroid labeling model based on the centroid loss function, until the centroid loss function converges, to obtain a target centroid labeling model that is trained completely;

[0020] The offset loss function comprises a distance loss function and / or a direction loss function, the distance loss function represents a loss amount of an offset distance of a centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents a loss amount of an offset direction of the centroid candidate point pair corresponding to the predicted centroid data.

[0021] According to another embodiment of the present application, a training device of an object segmentation model is provided, which comprises:

[0022] a target centroid data output module configured to input a centroid test image into a target centroid labeling model that is trained completely to obtain target centroid data output;

[0023] The object training image patch acquisition module is used to acquire at least one object training image patch in the centroid test image based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data.

[0024] The predicted object image output module is used to input each of the object training image blocks into the untrained initial object segmentation model to obtain the output predicted object image;

[0025] The target object segmentation model determination module is used to adjust the model parameters of the initial object segmentation model based on the predicted object image and the standard object image to obtain the trained target object segmentation model.

[0026] The target centroid annotation model is obtained by training the centroid annotation model according to any embodiment of the present invention, and the number of annotations of the standard object image is less than or equal to the number of segments of the annotation objects in the predicted object image.

[0027] According to another embodiment of the present invention, an electronic device is provided, the electronic device comprising:

[0028] At least one processor; and

[0029] A memory communicatively connected to the at least one processor; wherein,

[0030] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the centroid annotation model and / or the training method of the object segmentation model according to any embodiment of the present invention.

[0031] According to another embodiment of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions, the computer instructions being configured to cause a processor to execute and implement the training method of the centroid annotation model and / or the training method of the object segmentation model according to any embodiment of the present invention.

[0032] The technical scheme of the embodiment of the present application determines the offset loss function based on the standard center data and the predicted center data output by the initial center annotation model which is not trained, and determines the center loss function based on the offset loss function, adjusts the model parameters of the initial center annotation model based on the center loss function, and obtains the target center annotation model which is trained until the center loss function converges, wherein the offset loss function includes a distance loss function and / or a direction loss function, the distance loss function represents the loss amount of the offset distance of the center candidate point pair corresponding to the predicted center data, and the direction loss function represents the loss amount of the offset direction of the center candidate point pair corresponding to the predicted center data, thereby solving the problem of low efficiency of manual center annotation, improving the accuracy of center annotation, and further improving the training effect of the subsequent object segmentation model.

[0033] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present application, nor is it used to limit the scope of the present application. Other features of the present application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0035] Figure 1 A flowchart of a center annotation model training method provided by an embodiment of the present application;

[0036] Figure 2 A flowchart of another center annotation model training method provided by an embodiment of the present application;

[0037] Figure 3 A flowchart of an object segmentation model training method provided by an embodiment of the present application;

[0038] Figure 4 A model framework diagram of an object segmentation model training method provided by an embodiment of the present application;

[0039] Figure 5 A structure schematic diagram of a center annotation model training device provided by an embodiment of the present application;

[0040] Figure 6 A structure schematic diagram of an object segmentation model training device provided by an embodiment of the present application;

[0041] Figure 7A structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0042] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative efforts should fall within the scope of protection of the present application.

[0043] It should be noted that the terms "centroid", "object" and "region" and the like in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product or device.

[0044] Figure 1 A flowchart of a training method of a centroid labeling model provided by an embodiment of the present application. The embodiment can be applicable to the training of a centroid labeling model for automatically labeling the centroid of a labeled object in an image. The method can be executed by a centroid labeling model training device, which can be realized in the form of hardware and / or software, and can be configured in a terminal device. As shown in the figure, the method comprises: Figure 1

[0045] S110, input the centroid training image into the initial centroid labeling model which is not trained completely, to obtain the output predicted centroid data.

[0046] For example, the image type of the centroid training image can be a medical image, an environmental image, an article image or a human image, etc. The labeled objects in the environmental image include but are not limited to vehicles, building components and machine tools, etc. The acquisition device of the medical image includes but is not limited to an ultrasonic device, a magnetic resonance device, a CBCT (Cone beam CT) device and an infrared device, etc. The labeled objects in the medical image include but are not limited to teeth, cells, lesions, etc. The image type of the training image is not limited here.​

[0047] The network architecture of the initial centroid labeling model includes, for example, but is not limited to, a VoteNet network, a CNN (Convolutional Neural Networks), a U-Net network, and the like. The network architecture of the initial centroid labeling model is not limited herein.

[0048] As an optional embodiment, the initial centroid labeling model adopts a model principle of obtaining, by a farthest point sampling algorithm, at least one centroid reference point corresponding to each of the at least two labeled objects in the centroid training image, and then calculating, by a non-maximum suppression algorithm, an IoU (Intersection over Union) score corresponding to each of the centroid reference points. For each labeled object, the centroid reference point with the highest IoU score among the centroid reference points corresponding to the labeled object is taken as a centroid candidate point of the labeled object.

[0049] In this embodiment, the predicted centroid data contains predicted centroid coordinates corresponding to the at least two centroid candidate points. Specifically, each of the centroid candidate points corresponds to a labeled object in the centroid training image. Correspondingly, the centroid training image contains at least two labeled objects, that is, the predicted centroid data contains predicted centroid coordinates of the centroid candidate points corresponding to each of the labeled objects.

[0050] In an optional embodiment, when the centroid training image is a three-dimensional image, the predicted centroid coordinates are three-dimensional coordinates, and when the centroid training image is a two-dimensional image, the predicted centroid coordinates are two-dimensional coordinates.

[0051] In S120, an offset loss function is determined based on the predicted centroid data and the standard centroid data, and a centroid loss function is determined based on the offset loss function.

[0052] Specifically, the standard centroid data contains standard centroid coordinates corresponding to the at least two centroid real points, and each of the centroid real points corresponds to a labeled object, that is, the standard centroid data contains standard centroid coordinates of the centroid real points corresponding to each of the labeled objects.

[0053] In this embodiment, the offset loss function includes a distance loss function and / or a direction loss function. The distance loss function represents a loss amount of an offset distance of a centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents a loss amount of an offset direction of the centroid candidate point pair corresponding to the predicted centroid data.

[0054] As an optional embodiment, when the offset loss function includes the distance loss function, the offset loss function L de satisfies the formula:

[0055] L de = κL D

[0056] wherein κ represents a distance weight coefficient corresponding to the distance loss function L D , and exemplarily, κ can be 1, and herein the distance weight coefficient is not limited, and can be set according to actual requirements.

[0057] As an optional embodiment, when the offset loss function comprises a direction loss function, the offset loss function L de satisfies the formula:

[0058] L de = λL R

[0059] wherein λ represents a direction weight coefficient corresponding to the direction loss function L R , and exemplarily, λ can be 1, and herein the direction weight coefficient is not limited, and can be set according to actual requirements.

[0060] As an optional embodiment, when the offset loss function comprises a distance loss function and a direction loss function, the offset loss function L de satisfies the formula:

[0061] L de = κL D + λL R

[0062] wherein exemplarily, κ and λ can both be 0.3, and herein the specific parameter values of the two weight coefficients are not limited, and can be set according to actual requirements.

[0063] As an optional embodiment, based on the offset loss function, the centroid loss function is determined, comprising: taking the offset loss function as the centroid loss function. Exemplarily, the centroid loss function L ce = L de .

[0064] As another optional embodiment, based on the offset loss function, the centroid loss function is determined, comprising: obtaining a basic loss function corresponding to an initial centroid labeling model, and determining the centroid loss function based on the basic loss function and the offset loss function.

[0065] wherein specifically, the basic loss function represents a preset loss function adopted by the initial centroid labeling model, and exemplarily, the centroid loss function L ce = L base + L de , wherein L base represents the basic loss function.

[0066] The advantage of such an arrangement is that the offset loss function provided by the embodiments of the present application can be mounted on the basic loss function of any centroid annotation model, and is applicable to centroid annotation models of any network architecture, thereby improving the model performance of the centroid annotation model and the scalability of the embodiments of the present application.

[0067] S130, based on the centroid loss function, adjusting the model parameters of the initial centroid annotation model until the centroid loss function converges, to obtain a trained target centroid annotation model.

[0068] On the basis of the above-mentioned embodiments, optionally, the method further comprises: inputting the region training image into the untrained initial region segmentation model to obtain an output predicted region image; wherein the predicted region image comprises a region image of at least one annotated object corresponding to the region of interest; based on the predicted region image and the standard region image, adjusting the model parameters of the initial region segmentation model to obtain a trained target region segmentation model; inputting the region test image into the target region segmentation model to obtain an output centroid training image.

[0069] Specifically, the region training image and the region test image comprise a region image of an interference region and a region image of at least one annotated object corresponding to the region of interest, and the target region segmentation model is used to crop the region image of the interference region in the region training image and the region test image, and retain the region image of at least one annotated object corresponding to the region of interest.

[0070] Taking the region training image and the region test image as oral images and the annotated object as teeth as an example, the oral images usually comprise teeth, tongue, salivary glands, gums, etc., wherein the objects other than teeth constitute the region image of the interference region in the oral images, and the predicted region image and the centroid training image comprise a region image of at least one tooth corresponding to the region of interest.

[0071] Exemplarily, the network architecture of the initial region segmentation model includes but is not limited to a SegNet network, a DeepLab network, a FCN (Fully Convolutional Networks), a U-Net network, a ToothNet network, etc., and the network architecture of the initial region segmentation model is not limited herein.

[0072] Exemplarily, the loss function of the initial region segmentation model can be a Softmax cross-entropy loss function, and the loss function of the initial region segmentation model is not limited herein and can be customized according to actual needs.

[0073] The advantage of such arrangement is that the influence of irrelevant areas in the area test image on the accuracy of the centroid labeling of the labeled object is effectively filtered out, and the accuracy of the centroid labeling is further improved.

[0074] The technical scheme of the embodiment determines the offset loss function based on the standard centroid data and the predicted centroid data output by the initial centroid labeling model that is not trained completely, determines the centroid loss function based on the offset loss function, adjusts the model parameters of the initial centroid labeling model based on the centroid loss function, and obtains the target centroid labeling model that is trained completely when the centroid loss function converges, wherein the offset loss function includes the distance loss function and / or the direction loss function, the distance loss function represents the loss amount of the offset distance of the centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents the loss amount of the offset direction of the centroid candidate point pair corresponding to the predicted centroid data, thereby solving the problem of low efficiency of manual centroid labeling, improving the accuracy of centroid labeling, and further improving the training effect of the subsequent object segmentation model.

[0075] Figure 2 The flowchart of another training method of a centroid labeling model provided by an embodiment of the application, which further refines the "determining the offset loss function based on the predicted centroid data and the standard centroid data" in the above embodiment. As shown in the figure, the method comprises: Figure 2

[0076] S210, inputting the centroid training image into the initial centroid labeling model that is not trained completely to obtain the output predicted centroid data.

[0077] S210 in the embodiment corresponds to S110 in the above embodiment, and the details are not repeated here. Figure 1

[0078] S220, determining the predicted parameter data based on the predicted centroid coordinates corresponding to the at least two centroid candidate points in the predicted centroid data.

[0079] In the embodiment, the predicted parameter data includes a predicted distance matrix and / or a predicted normal vector set. The predicted distance matrix represents the predicted centroid distance corresponding to the at least one centroid candidate point pair in the predicted centroid data, and the predicted normal vector set includes the predicted normal vector of the at least one centroid candidate point pair in the predicted centroid data.

[0080] ​​As an optional embodiment, the prediction parameter data is determined based on the predicted centroid coordinates corresponding to at least two centroid candidate points in the predicted centroid data, including: for each pair of centroid candidate points corresponding to the predicted centroid data, the predicted centroid distance is determined based on the two predicted centroid coordinates corresponding to the pair of centroid candidate points, and the matrix parameter value in the prediction distance matrix corresponding to the pair of centroid candidate points is set as the predicted centroid distance; and / or, the predicted normal vector is determined based on the two predicted centroid coordinates corresponding to the pair of centroid candidate points, and the predicted normal vector is added to the predicted normal vector set.

[0081] For example, the candidate point set consisting of at least two centroid candidate points adopts P. c It means that P c ={pc1,pc2,…,pc n}, where n represents the total number of centroid candidate points in the predicted centroid data or the total number of labeled objects in the centroid training image.

[0082] In this embodiment of the invention, the centroid training image is used as a three-dimensional image, and the centroid candidate point pairs are based on the centroid candidate points pc. i and centroid candidate points pc j For illustrative purposes, other centroid candidate point pairs are compared with centroid candidate point pc. i and centroid candidate points pc j The corresponding ones are the same or similar, where i,j∈[1,n].

[0083] For example, the centroid candidate point pc i The predicted centroid coordinates are centroid candidate point pc j The predicted centroid coordinates are Correspondingly, the candidate centroid point pc i and centroid candidate points pc j The corresponding predicted centroid distance D p (pc i ,pc j Satisfies the formula:

[0084]

[0085] In this example, the predicted distance matrix D p The dimension is n×n.

[0086] For example, the centroid candidate point pc i and centroid candidate points pc j Corresponding prediction normal vector Satisfying the formula:

[0087]

[0088] S230, determine the standard parameter data based on the standard centroid coordinates corresponding to the at least two centroid real points in the standard centroid data.

[0089] In this embodiment, the standard parameter data includes a standard distance matrix and / or a standard normal vector set. The standard distance matrix represents the standard centroid distances corresponding to the at least two centroid real point pairs in the standard centroid data, and the standard normal vector set represents the standard normal vectors corresponding to the at least two centroid real point pairs in the standard centroid data.

[0090] As an optional embodiment, the standard parameter data is determined based on the standard centroid coordinates corresponding to the at least two centroid real points in the standard centroid data, including: for each centroid real point pair in the standard centroid data, determining a standard centroid distance based on the two standard centroid coordinates corresponding to the centroid real point pair, and setting the matrix parameter value corresponding to the centroid real point pair in the standard distance matrix as the standard centroid distance; and / or, determining a standard normal vector based on the two standard centroid coordinates corresponding to the centroid real point pair, and adding the standard normal vector to the standard normal vector set.

[0091] For example, the candidate point set composed of the at least two centroid real points adopts Q c , n represents the total number of centroid real points in the standard centroid data or the total number of labeled objects in the centroid training image. c = {qc1, qc2, …, qc n n represents the total number of centroid real points in the standard centroid data or the total number of labeled objects in the centroid training image.

[0092] For example, the standard centroid coordinates of the centroid real point qc i are The standard centroid coordinates of the centroid real point qc j are Correspondingly, the standard centroid distance D q (qc i , qc j ) corresponding to the centroid real point qc i and the centroid real point qc j satisfies the formula:

[0093]

[0094] In this example, the dimension of the standard distance matrix D q is n x n.

[0095] For example, the standard normal vector corresponding to the centroid real point qc i and the centroid real point qc j satisfies the formula:

[0096]

[0097] S240, determining the offset loss function based on the predicted parameter data and the standard parameter data.

[0098] As an optional embodiment, determining the offset loss function based on the predicted parameter data and the standard parameter data comprises: for each pair of the center candidate points corresponding to the predicted center data, obtaining a predicted center distance and a standard center distance in the predicted distance matrix and the standard distance matrix respectively corresponding to the pair of the center candidate points; determining an offset distance based on the predicted center distance and the standard center distance; and determining a distance loss function based on the offset distance corresponding to at least one pair of the center candidate points.

[0099] For example, the distance loss function L D satisfies the formula:

[0100]

[0101] wherein D p (pc i ,pc j ) represents a predicted center distance in the predicted distance matrix D p corresponding to the center candidate point pc i and the center candidate point pc j , and D q (qc i ,qc j ) represents a standard center distance in the standard distance matrix D q corresponding to the center real point qc i and the center real point qc j .

[0102] Specifically, represents an offset distance of the center candidate point pc i and the center candidate point pc j , and when the predicted center coordinates corresponding to the center candidate point pc i and the center candidate point pc j are both correct, the offset distance is 0.

[0103] As an optional embodiment, determining the offset loss function based on the predicted parameter data and the standard parameter data comprises: for each pair of the center candidate points corresponding to the predicted center data, obtaining a predicted normal vector and a standard normal vector in the predicted normal vector set and the standard normal vector set respectively corresponding to the pair of the center candidate points; determining an offset direction based on the predicted normal vector and the standard normal vector; and determining a direction loss function based on the offset direction corresponding to at least one pair of the center candidate points.

[0104] In this embodiment, the offset angle of two normal vectors is used to represent the offset direction, and the exemplary center point candidate pc i and the offset direction θ j of the center point candidate pc ij satisfies the formula:

[0105]

[0106] The direction loss function L R satisfies the formula:

[0107]

[0108] Wherein, 0≤θ ij <180.

[0109] S250, determine the center loss function based on the offset loss function.

[0110] S260, adjust the model parameters of the initial center labeling model based on the center loss function, until the center loss function converges, and obtain the trained target center labeling model.

[0111] S250-S260 in this embodiment correspond to S120-S130 in Figure 1 , which are the same or similar, and will not be repeated here.

[0112] The technical scheme of this embodiment determines the prediction parameter data based on the prediction center coordinates corresponding to at least two center point candidates in the prediction center data, determines the standard parameter data based on the standard center coordinates corresponding to at least two center point true points in the standard center data, and determines the offset loss function based on the prediction parameter data and the standard parameter data. This increases the learning ability of the center labeling model to the spatial position correlation between the center point candidates, solves the problem of low efficiency of manual center labeling, improves the accuracy of center labeling, and further improves the training effect of the subsequent object segmentation model.

[0113] Figure 3 A flowchart of an object segmentation model training method provided by an embodiment of the present application, which can be applied to the case of training an object segmentation model for automatically segmenting labeled objects in an image. The method can be executed by an object segmentation model training device, which can be realized in the form of hardware and / or software, and can be configured in a terminal device. As shown in Figure 3 , the method comprises:

[0114] S310, input the centroid test image into the pre-trained target centroid labeling model to obtain output target centroid data.

[0115] Specifically, the image type of the centroid test image corresponds to the image type of the centroid training image in the above embodiment, which is not repeated here.

[0116] In this embodiment, the target centroid labeling model is obtained by using the training method of the centroid labeling model in the above embodiment of the present application, and the target centroid data includes target centroid coordinates corresponding to at least one centroid target point. Specifically, each centroid target point corresponds to an annotation object in the centroid test image, and correspondingly, the centroid test image includes at least one annotation object.

[0117] S320, based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data, obtain at least one object training image block in the centroid test image.

[0118] As an optional embodiment, based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data, obtaining at least one object training image block in the centroid test image includes: for each centroid target point in the target centroid data, based on the preset cropping data and the target centroid coordinates corresponding to the centroid target point, obtaining the object training image block in the centroid test image; wherein the preset cropping data is a preset cropping size or a preset cropping number based on distance.

[0119] In one embodiment, the preset cropping data is a preset cropping size, and correspondingly, the center point coordinates of the object training image block are the target centroid coordinates. For example, the preset cropping size can be 3mm*2mm*3mm, which is not limited here and can be customized according to actual needs.

[0120] In another embodiment, the preset cropping data is a preset cropping number based on distance. Specifically, the image point distance corresponding to the image point coordinates of each image point in the centroid test image and the target centroid coordinates is obtained, and each image point distance is sorted. The image block composed of the image points with smaller image point distances in the preset cropping number is taken as the object training image block.

[0121] For example, assuming that the preset cropping number is 100, when the sorting method is descending, the image block composed of the image points corresponding to the last 100 image point distances is taken as the object training image block, and when the sorting method is ascending, the image block composed of the image points corresponding to the first 100 image point distances is taken as the object training image block.

[0122] S330, input each object training image block into the initial object segmentation model which is not trained completely, and obtain the output prediction object image.

[0123] For example, the network architecture of the initial object segmentation model includes but is not limited to SegNet network, DeepLab network, FCN (Fully Convolutional Networks), U-Net network and ToothNet network, etc., and the network architecture of the initial object segmentation model is not limited here.

[0124] As an optional embodiment, the prediction object image includes a prediction instance image and / or a prediction edge image, wherein the prediction instance image includes a background region and at least one labeled object corresponding to a spatial entity region, and the prediction edge image includes a background region and at least one labeled object corresponding to an edge contour region.

[0125] S340, based on the prediction object image and the standard object image, adjusting the model parameters of the initial object segmentation model to obtain a trained target object segmentation model.

[0126] For example, the loss function of the initial object segmentation model can be a Softmax cross-entropy loss function, and the loss function of the initial object segmentation model is not limited here, and can be customized according to actual needs.

[0127] Figure 4 For an embodiment of the application, a model framework diagram of an object segmentation model training method is provided, specifically, the initial object segmentation model training method includes three network models, which are a region segmentation model, a centroid labeling model and an object segmentation model.

[0128] Figure 4 Taking the region image as the oral image and the labeled object as the tooth as an example, when the region image is a region training image, the region segmentation model is an initial region segmentation model, and when the region image is a region test image, the region segmentation model is a target region segmentation model. The centroid image is a centroid training image or a centroid test image. When the centroid image is a centroid training image, the centroid labeling model is an initial centroid labeling model, and the centroid point is a centroid candidate point. When the centroid image is a centroid test image, the centroid labeling model is a target centroid labeling model, and the centroid point is a centroid target point.

[0129] When the center-of-mass image is the center-of-mass test image, the target center-of-mass labeling model obtains target center-of-mass coordinates of a target center-of-mass point output by the center-of-mass test image, obtains an object training image block in the center-of-mass test image based on the target center-of-mass coordinates, inputs the object training image block into the initial object segmentation model, and obtains a predicted object image output by the initial object segmentation model. In this example, the predicted object image includes a predicted instance image and a predicted edge image.

[0130] Image segmentation refers to a technology of dividing an image into a plurality of mutually disjoint regions according to features such as gray scale, color, spatial texture, geometric shape, and the like, so that the features show consistency or similarity within the same region and show obvious difference between different regions. With the continuous development of artificial intelligence technology, neural network models begin to be applied to image segmentation, but the training process of the neural network model depends on a large amount of labeled data of labeled objects.

[0131] The technical scheme of the embodiment, by first using the target center-of-mass labeling model, obtaining target center-of-mass coordinates respectively corresponding to each labeled object in the center-of-mass test image, and then based on each target center-of-mass coordinate, obtaining at least one object training image block in the center-of-mass test image, inputting each object training image block into the initial object segmentation model which is not trained, obtaining a predicted object image output by the initial object segmentation model, adjusting the model parameters of the initial object segmentation model based on the predicted object image and the standard object image, and obtaining the target object segmentation model which is trained, when the number of labeled objects contained in the standard object image is equal to the number of segmented objects contained in the predicted object image, the embodiment can solve the problem of poor training effect of the traditional object segmentation model under the assistance of the target center-of-mass data, thereby improving the segmentation quality of the labeled objects. When the number of labeled objects contained in the standard object image is less than the number of segmented objects contained in the predicted object image, the embodiment can achieve the purpose of weakly supervised training of the object segmentation model under the assistance of the target center-of-mass data, solve the problem that the training of the traditional object segmentation model depends on a large amount of object labeled data, and reduce the dependence of the object segmentation model on the number of labeled objects in the standard object image while ensuring the training effect of the object segmentation model.

[0132] The following is an embodiment of a center-of-mass labeling model training device provided by the embodiment, which belongs to the same inventive concept as the center-of-mass labeling model training method described above. Details not described in the embodiment of the center-of-mass labeling model training device can be referred to the content about the center-of-mass labeling model training method in the above embodiment.

[0133] Figure 5 The structure diagram of a center-of-mass labeling model training device provided by an embodiment of the present application. As shown in Figure 5As shown, the apparatus comprises: a predicted centroid data output module 410, a centroid loss function determination module 420, and a target centroid labeling model determination module 430.

[0134] The predicted centroid data output module 410 is configured to input the centroid training image into the initial centroid labeling model which is not trained completely, to obtain output predicted centroid data.

[0135] The centroid loss function determination module 420 is configured to determine an offset loss function based on the predicted centroid data and the standard centroid data, and determine a centroid loss function based on the offset loss function.

[0136] The target centroid labeling model determination module 430 is configured to adjust the model parameters of the initial centroid labeling model based on the centroid loss function, until the centroid loss function converges, to obtain the target centroid labeling model which is trained completely.

[0137] The offset loss function comprises a distance loss function and / or a direction loss function, the distance loss function represents a loss amount of an offset distance of a centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents a loss amount of an offset direction of the centroid candidate point pair corresponding to the predicted centroid data.

[0138] The technical scheme of the embodiment solves the problem of low efficiency of manual centroid labeling, improves the accuracy of centroid labeling, and further improves the training effect of a subsequent object segmentation model.

[0139] As an optional embodiment, the centroid loss function determination module 420 comprises:

[0140] The predicted parameter data determination unit is configured to determine predicted parameter data based on predicted centroid coordinates corresponding to at least two centroid candidate points in the predicted centroid data.

[0141] The standard parameter data determination unit is configured to determine standard parameter data based on standard centroid coordinates corresponding to at least two centroid real points in the standard centroid data.

[0142] The offset loss function determination unit is configured to determine the offset loss function based on the predicted parameter data and the standard parameter data.

[0143] When the predicted parameter data comprises a predicted distance matrix, the standard parameter data comprises a standard distance matrix, and the offset loss function comprises the distance loss function; when the predicted parameter data comprises a predicted normal vector set, the standard parameter data comprises a standard normal vector set, and the offset loss function comprises the direction loss function.

[0144] As an optional embodiment, the predicted parameter data determination unit is specifically configured to:

[0145] For each centroid candidate point pair corresponding to the predicted centroid data, a predicted centroid distance is determined based on the two predicted centroid coordinates corresponding to the centroid candidate point pair, and a matrix parameter value in the predicted distance matrix corresponding to the centroid candidate point pair is set as the predicted centroid distance.

[0146] And / or,

[0147] A predicted normal vector is determined based on the two predicted centroid coordinates corresponding to the centroid candidate point pair, and the predicted normal vector is added to the predicted normal vector set.

[0148] As an optional embodiment, the offset loss function determination unit comprises:

[0149] The distance loss function determination sub-unit is configured to, for each centroid candidate point pair corresponding to the predicted centroid data, obtain a predicted centroid distance and a standard centroid distance in a predicted distance matrix and a standard distance matrix corresponding to the centroid candidate point pair, respectively.

[0150] An offset distance is determined based on the predicted centroid distance and the standard centroid distance.

[0151] As an optional embodiment, the offset loss function determination unit comprises:

[0152] The direction loss function determination sub-unit is configured to, for each centroid candidate point pair corresponding to the predicted centroid data, obtain a predicted normal vector and a standard normal vector in a predicted normal vector set and a standard normal vector set corresponding to the centroid candidate point pair, respectively.

[0153] An offset direction is determined based on the predicted normal vector and the standard normal vector.

[0154] The direction loss function is determined based on the offset direction corresponding to at least one centroid candidate point pair.

[0155] As an optional embodiment, the device further comprises:

[0156] The centroid training image output module is configured to input the region training image into the initial region segmentation model which is not trained completely, to obtain an output predicted region image; wherein the predicted region image comprises a region image of the region of interest corresponding to at least one labeled object.

[0157] The model parameters of the initial region segmentation model are adjusted based on the predicted region image and the standard region image, to obtain a target region segmentation model which is trained completely.

[0158] The region test image is input into the target region segmentation model, to obtain an output centroid training image.

[0159] The training device of the center-of-mass labeling model provided in the embodiments of the present application can execute the training method of the center-of-mass labeling model provided in any of the embodiments of the present application, and has the function modules and beneficial effects corresponding to the execution method.

[0160] The following is an embodiment of the training device of the object segmentation model provided in the embodiments of the present application. The device and the training method of the object segmentation model in the above embodiments belong to the same inventive concept. Details not described in the embodiment of the training device of the object segmentation model can be referred to the content about the training method of the object segmentation model in the above embodiments.

[0161] Figure 6 The structure diagram of a training device of an object segmentation model provided in an embodiment of the present application is shown in FIG. 5. Figure 6 As shown in the figure, the device comprises a target center-of-mass data output module 510, an object training image block acquisition module 520, a predicted object image output module 530, and a target object segmentation model determination module 540.

[0162] The target center-of-mass data output module 510 is configured to input a center-of-mass test image into a target center-of-mass labeling model that has been pre-trained to obtain output target center-of-mass data.

[0163] The object training image block acquisition module 520 is configured to acquire at least one object training image block in the center-of-mass test image based on the target center-of-mass coordinates corresponding to at least one center-of-mass target point in the target center-of-mass data.

[0164] The predicted object image output module 530 is configured to input each object training image block into an initial object segmentation model that has not been trained to obtain output predicted object images.

[0165] The target object segmentation model determination module 540 is configured to adjust the model parameters of the initial object segmentation model based on the predicted object images and standard object images to obtain a target object segmentation model that has been trained.

[0166] The target center-of-mass labeling model is obtained by using the training method of the center-of-mass labeling model in the above embodiments of the present application. The number of labeled objects contained in the standard object images is less than or equal to the number of segmented objects contained in the predicted object images.

[0167] The technical solution of the present embodiment solves the problem of poor training effect of the conventional object segmentation model, and reduces the dependence of the object segmentation model on the number of labeled objects in the standard object images while ensuring the training effect of the object segmentation model.

[0168] As an optional embodiment, the object training image block acquisition module 520 is specifically configured to:

[0169] For each target mass data point in the target mass data, based on the preset cropping data and the target mass center coordinates corresponding to the target mass point, an object training image block in the mass test image is obtained.

[0170] The preset cropping data is a preset cropping size or a preset cropping number based on distance.

[0171] The training device of the object segmentation model provided in the embodiments of the present application can execute the training method of the object segmentation model provided in any embodiment of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0172] Figure 7 A structure diagram of an electronic device is provided for an embodiment of the present application. The electronic device 10 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular telephones, smart phones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the present application described and / or claimed in this document.

[0173] As shown in Figure 4 The electronic device 10 includes at least one processor 11, and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11, wherein the memory stores a computer program executable by the at least one processor. The processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or loaded from the storage unit 18 into the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0174] A plurality of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a magnetic disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network, such as the Internet, and / or various telecommunications networks.

[0175] The processor 11 can be various general and / or special purpose processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs various methods and processes described above, such as the training method of the centroid labeling model and / or the training method of the object segmentation model provided by the above-described embodiments.

[0176] In some embodiments, the training method of the centroid labeling model and / or the training method of the object segmentation model provided by the above-described embodiments can be implemented as a computer program tangibly embodied in a computer readable storage medium, such as the storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed onto the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded onto the RAM 13 and executed by the processor 11, one or more steps of the training method of the centroid labeling model and / or the training method of the object segmentation model described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the training method of the centroid labeling model and / or the training method of the object segmentation model by any other appropriate means, such as by means of firmware.

[0177] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0178] Computer programs for implementing the methods of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the computer program, when executed, enables the functions / acts specified in the flowcharts and / or block diagrams to be implemented. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a standalone software package and partially on a remote machine or entirely on a remote machine or server.

[0179] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0180] To provide for interaction with a user, the systems and techniques described here can be implemented on an electronic device having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the electronic device. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0181] The systems and techniques described herein can be implemented in a computing system that includes a back end component, e.g., as a data server, or that includes a middleware component, e.g., an application server, or that includes a front end component, e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described herein, or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication, e.g., a communication network. Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0182] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. A server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0183] It should be understood that the various forms of flow shown above can be re-ordered, added to, or deleted from without departing from the scope of the present disclosure. For example, the steps recited in the present disclosure can be performed in parallel, in series, or in a different order, and the present disclosure is not limited in this regard.

[0184] The specific embodiments described above are not intended to limit the scope of the present disclosure. Those skilled in the art will understand that various modifications, combinations, sub-combinations, and alternatives can be made to the specific embodiments without departing from the spirit and principles of the present disclosure. Any further modifications, equivalents, and / or alternatives come within the scope of the present disclosure as recited by the claims.

Claims

1. A training method for a centroid annotation model, characterized in that, include: The centroid training image is input into the untrained initial centroid annotation model to obtain the output predicted centroid data. Based on the predicted centroid data and the standard centroid data, the offset loss function is determined, and based on the offset loss function, the centroid loss function is determined. Based on the centroid loss function, the model parameters of the initial centroid labeling model are adjusted until the centroid loss function converges, thus obtaining the trained target centroid labeling model. The offset loss function includes a distance loss function and / or a direction loss function. The distance loss function represents the amount of loss in offset distance of the centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents the amount of loss in offset direction of the centroid candidate point pair corresponding to the predicted centroid data. The step of determining the offset loss function based on the predicted centroid data and the standard centroid data includes: Based on the predicted centroid coordinates corresponding to at least two candidate centroid points in the predicted centroid data, the prediction parameter data is determined. Based on the standard centroid coordinates corresponding to at least two true centroid points in the standard centroid data, standard parameter data are determined. Based on the predicted parameter data and the standard parameter data, determine the offset loss function; Wherein, when the prediction parameter data includes a prediction distance matrix, the standard parameter data includes a standard distance matrix, and the offset loss function includes a distance loss function; when the prediction parameter data includes a set of prediction normal vectors, the standard parameter data includes a set of standard normal vectors, and the offset loss function includes a direction loss function.

2. The method according to claim 1, characterized in that, The step of determining prediction parameter data based on the predicted centroid coordinates corresponding to at least two candidate centroid points in the predicted centroid data includes: For each pair of centroid candidate points corresponding to the predicted centroid data, the predicted centroid distance is determined based on the two predicted centroid coordinates corresponding to the pair of centroid candidate points, and the matrix parameter value in the predicted distance matrix corresponding to the pair of centroid candidate points is set as the predicted centroid distance. And / or, Based on the two predicted centroid coordinates corresponding to the candidate centroid points, the predicted normal vector is determined and added to the predicted normal vector set.

3. The method according to claim 2, characterized in that, The step of determining the offset loss function based on the predicted parameter data and the standard parameter data includes: For each pair of centroid candidate points corresponding to the predicted centroid data, obtain the predicted centroid distance and standard centroid distance corresponding to the pair of centroid candidate points in the predicted distance matrix and the standard distance matrix, respectively. The offset distance is determined based on the predicted centroid distance and the standard centroid distance; The distance loss function is determined based on the offset distances corresponding to at least one pair of centroid candidate points.

4. The method according to claim 2, characterized in that, The step of determining the offset loss function based on the predicted parameter data and the standard parameter data includes: For each pair of centroid candidate points corresponding to the predicted centroid data, obtain the predicted normal vector and the standard normal vector corresponding to the pair of centroid candidate points in the predicted normal vector set and the standard normal vector set, respectively. The offset direction is determined based on the predicted normal vector and the standard normal vector; The directional loss function is determined based on the offset direction corresponding to at least one pair of centroid candidate points.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: The region training image is input into the untrained initial region segmentation model to obtain the output predicted region image; wherein the predicted region image contains a region image corresponding to at least one labeled object. Based on the predicted region image and the standard region image, the model parameters of the initial region segmentation model are adjusted to obtain the trained target region segmentation model. The region test image is input into the target region segmentation model to obtain the output centroid training image.

6. A training method for an object segmentation model, characterized in that, include: The centroid test image is input into the pre-trained target centroid annotation model to obtain the output target centroid data; Based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data, at least one object training image patch in the centroid test image is obtained; Each of the object training image blocks is input into the untrained initial object segmentation model to obtain the output predicted object image; Based on the predicted object image and the standard object image, the model parameters of the initial object segmentation model are adjusted to obtain the trained target object segmentation model. The target centroid annotation model is obtained by training the centroid annotation model according to any one of claims 1-5, and the number of annotations of the standard object image is less than or equal to the number of segments of the annotation objects in the predicted object image.

7. The method according to claim 6, characterized in that, The step of obtaining at least one object training image patch in the centroid test image based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data includes: For each centroid target point in the target centroid data, based on preset cropping data and the target centroid coordinates corresponding to the centroid target point, obtain the object training image patch in the centroid test image; The preset cutting data refers to a preset cutting size or a preset cutting quantity based on distance.

8. A training device for a centroid-annotated model, characterized in that, include: The predicted centroid data output module is used to input the centroid training image into the untrained initial centroid annotation model to obtain the output predicted centroid data. The centroid loss function determination module is used to determine the offset loss function based on the predicted centroid data and the standard centroid data, and to determine the centroid loss function based on the offset loss function. The target centroid annotation model determination module is used to adjust the model parameters of the initial centroid annotation model based on the centroid loss function until the centroid loss function converges, thereby obtaining the trained target centroid annotation model. The offset loss function includes a distance loss function and / or a direction loss function. The distance loss function represents the amount of loss in offset distance of the centroid candidate point pair corresponding to the predicted centroid data, and the direction loss function represents the amount of loss in offset direction of the centroid candidate point pair corresponding to the predicted centroid data. The centroid loss function determination module includes: The prediction parameter data determination unit is used to determine prediction parameter data based on the prediction centroid coordinates corresponding to at least two centroid candidate points in the prediction centroid data. The standard parameter data determination unit is used to determine standard parameter data based on the standard centroid coordinates corresponding to at least two true centroid points in the standard centroid data. The offset loss function determination unit is used to determine the offset loss function based on the predicted parameter data and the standard parameter data; Wherein, when the prediction parameter data includes a prediction distance matrix, the standard parameter data includes a standard distance matrix, and the offset loss function includes a distance loss function; when the prediction parameter data includes a set of prediction normal vectors, the standard parameter data includes a set of standard normal vectors, and the offset loss function includes a direction loss function.

9. A training device for a centroid-annotated model, characterized in that, include: The target centroid data output module is used to input the centroid test image into the pre-trained target centroid annotation model to obtain the output target centroid data. The object training image patch acquisition module is used to acquire at least one object training image patch in the centroid test image based on the target centroid coordinates corresponding to at least one centroid target point in the target centroid data. The predicted object image output module is used to input each of the object training image blocks into the untrained initial object segmentation model to obtain the output predicted object image; The target object segmentation model determination module is used to adjust the model parameters of the initial object segmentation model based on the predicted object image and the standard object image to obtain the trained target object segmentation model. The target centroid annotation model is obtained by training the centroid annotation model according to any one of claims 1-5, and the number of annotations of the standard object image is less than or equal to the number of segments of the annotation objects in the predicted object image.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the training method of the centroid annotation model according to any one of claims 1-5, and / or the training method of the object segmentation model according to any one of claims 6-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute and implement the training method of the centroid annotation model according to any one of claims 1-5, and / or the training method of the object segmentation model according to any one of claims 6-7.

Images