Data annotation method, device and storage medium based on fuzzy comprehensive evaluation method

Abstract

The present application discloses a data annotation method, device and storage medium based on the fuzzy comprehensive evaluation method. Among them, the method includes: obtaining the evaluation results of N users for M factors of the image to be annotated, where each factor corresponds to Y evaluation types; respectively counting the distribution of the evaluation results of each factor; determining the fuzzy comprehensive evaluation matrix of the image to be annotated according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×N matrix; obtaining the weights corresponding to each of the M factors; combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the image to be annotated; the weight vector of the evaluation set is a 1×N matrix; determining the evaluation result of the image to be annotated according to the Y values included in the weight vector of the evaluation set. Using this method can improve the accuracy of the nodule marking result.

Description

Technical Field

[0001] This application relates to the field of medicine, and in particular, to a data annotation method, device, and storage medium based on the fuzzy comprehensive evaluation method. Background Art

[0002] With the rise of deep learning technology, this technology has been more and more widely used in the field of medical image analysis. For example, in the early screening and prevention of lung cancer, deep learning algorithms can be used to detect and segment lung nodules from chest CT images, and identify and classify the benign and malignant of lung nodules. The accuracy of using deep learning technology to identify lung nodule classification can even rival that of professional physicians. However, deep learning algorithms require a large amount of accurately labeled data for training, and usually use the crowdsourcing method to label a large amount of data by multiple institutions or individuals.

[0003] However, medical images are different from ordinary natural images and need to be labeled by professional radiologists, resulting in low labeling efficiency. In addition, due to the complexity of lesions, many lesions do not have very clear and quantifiable definitions, and usually require radiologists to judge based on their own experience, resulting in a low consistency rate of labeling. When the same lesion is labeled multiple times by one physician or simultaneously labeled by multiple physicians, the labeling results often have ambiguities. Summary of the Invention

[0004] The embodiments of this application provide a data annotation method, device, and storage medium based on the fuzzy comprehensive evaluation method, which can improve the accuracy of medical image marking results.

[0005] In a first aspect, the embodiments of this application provide a data annotation method based on the fuzzy comprehensive evaluation method, including:

[0006] Obtain the evaluation results of N users for M factors of the image to be annotated, and each factor corresponds to Y evaluation types;

[0007] Statistically analyze the distribution of the evaluation results of each factor respectively;

[0008] Determine the fuzzy comprehensive evaluation matrix of the image to be annotated according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×Y matrix;

[0009] Obtain the weights corresponding to each of the M factors;

[0010] Combine the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the image to be annotated; the weight vector of the evaluation set is a 1×Y matrix;

[0011] Determine the evaluation result of the image to be annotated according to the Y values included in the weight vector of the evaluation set.

[0012] In a possible implementation, before obtaining the evaluation results of M factors of the to-be-annotated image by N users, the method further includes:

[0013] Determine a to-be-annotated data set; the to-be-annotated data set includes X to-be-annotated images;

[0014] Send the to-be-annotated data set to N terminals respectively; the N terminals correspond to N users;

[0015] Receive the evaluation result v of each factor of each of the X to-be-annotated images in the to-be-annotated data set for each terminal j = δ(k, x, u i ), to obtain an evaluation result set; where k represents the number of the terminal, and the value range of k is a positive integer from 1 to N, x represents the number of the to-be-annotated image, and the value range of x is a positive integer from 1 to X, u i represents the number of the factor, and the value range of i is a positive integer from 1 to M, v j represents the evaluation result, and the value range of j is a positive integer from 1 to Y;

[0016] The obtaining the evaluation results of M factors of the to-be-annotated image by N users includes:

[0017] Obtain the evaluation results of M factors of the to-be-annotated image by N users from the evaluation result set according to the number of the to-be-annotated image.

[0018] In a possible implementation, the respectively counting the distribution of the evaluation results of each factor includes:

[0019] Count the distribution of the evaluation results of factor u of the to-be-annotated image x by the N users i ; the distribution is that there are n ij users whose evaluation results of factor u of the to-be-annotated image x are v i , and j And

[0020]

[0021] The determining the fuzzy comprehensive evaluation matrix of the to-be-annotated image according to the distribution of the evaluation results of each factor includes:

[0022] Determine the membership degree r of factor u of the to-be-annotated image x for the Y evaluation types i = n ij / N; ij / N;

[0023] Determine the fuzzy comprehensive evaluation matrix R of the to-be-annotated image according to the membership degrees of each factor of the to-be-annotated image x to the Y evaluation types, where R = [r ij M×Y .

[0024] In a possible implementation manner, the weights corresponding to the M factors are represented by a 1×M matrix, and the 1×M matrix is the weight vector A = (a 1 , a 2 , …, a M ) of the factor set of the to-be-processed image;

[0025] Determining the weight vector of the evaluation set of the to-be-annotated image by combining the weights corresponding to the M factors and the fuzzy comprehensive evaluation matrix includes:

[0026] Through fuzzy transformation, convert the weight vector of the factor set of the to-be-processed image into the weight vector B of the evaluation set; the is the dominant factor prominent type synthesis operator.

[0027] In a possible implementation manner, the weight vector of the evaluation set includes the membership degrees corresponding to the Y evaluation types of the to-be-annotated image respectively;

[0028] Determining the evaluation result of the to-be-annotated image according to the Y values included in the weight vector of the evaluation set includes: determining the evaluation type with the highest membership degree among the membership degrees corresponding to the Y evaluation types of the to-be-annotated image as the evaluation result of the to-be-annotated image.

[0029] In a possible implementation manner, the M factors include at least two of the following: nodule diameter, presence of burrs, nodule type, nodule location.

[0030] In a possible implementation manner, each factor corresponding to the Y evaluation types includes at least two of the following: high risk, medium risk, low risk.

[0031] In a second aspect, an embodiment of the present application provides a data annotation device based on the fuzzy comprehensive evaluation method, including:

[0032] A first acquisition module, configured to acquire the evaluation results of N users for M factors of the to-be-annotated image, where each factor corresponds to Y evaluation types;

[0033] A statistics module, configured to respectively count the distribution of the evaluation results of each factor;

[0034] A first determination module, configured to determine the fuzzy comprehensive evaluation matrix of the to-be-annotated image according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×N matrix;​

[0035] A second acquisition module, configured to acquire the weight corresponding to each of the M factors.

[0036] A second determination module, configured to determine a weight vector of the evaluation set of the to-be-annotated image by combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix; the weight vector of the evaluation set is a 1×N matrix.

[0037] A third determination module, configured to determine an evaluation result of the to-be-annotated image according to Y values included in the weight vector of the evaluation set.

[0038] In a third aspect, an embodiment of the present application provides a computer storage medium, which stores multiple instructions, and the instructions are suitable for being loaded and executed by a processor to perform the method steps provided in the first aspect or any possible implementation manner of the first aspect of the embodiments of the present application.

[0039] In a fourth aspect, an embodiment of the present application provides an electronic device, which includes: a processor and a memory; wherein, the memory stores a computer program, and the computer program is suitable for being loaded and executed by the processor to perform the method steps provided in the first aspect or any possible implementation manner of the first aspect of the embodiments of the present application.

[0040] In the embodiment of the present application, by acquiring the evaluation results of N users for M factors of the to-be-annotated image, each factor corresponding to Y evaluation types; respectively counting the distribution of the evaluation results of each factor; determining the fuzzy comprehensive evaluation matrix of the to-be-annotated image according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×Y matrix; acquiring the weights corresponding to each of the M factors; determining the weight vector of the evaluation set of the to-be-annotated image by combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix; the weight vector of the evaluation set is a 1×Y matrix; determining the evaluation result of the to-be-annotated image according to the weight vector of Y values included in the evaluation set, the fuzzy comprehensive evaluation method can be used to quantify and fuse the annotation processes of multiple physicians, eliminate the influence of subjective differences between different physicians, obtain a relatively objective annotation result, solve the problem of inconsistent multi-person annotation caused by the ambiguity of problem definition, and improve the accuracy of nodule marking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required in the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application, and those of ordinary skill in the art can obtain other drawings without creative efforts based on these drawings.

[0042] Figure 1 Schematic diagram of the architecture of a data annotation system based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application;

[0043] Figure 2 Schematic diagram of the process of a data annotation method based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application;

[0044] Figure 3 Schematic diagram of the process of another data annotation method based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application;

[0045] Figure 4 Schematic diagram of a set of evaluation results provided by an embodiment of the present application;

[0046] Figure 5 Schematic diagram of the structure of a data annotation device based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application;

[0047] Figure 6 Schematic diagram of the structure of another data annotation device based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application;

[0048] Figure 7 Schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0049] Next, the technical solutions in the embodiments of the present application will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present application.

[0050] The terms "first", "second", "third", etc. in the specification, claims and accompanying drawings of the present application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally further includes steps or units not listed, or optionally further includes other steps or units inherent to these processes, methods, products or devices.

[0051] Please refer to Figure 1 , Figure 1 which is a schematic diagram of the architecture of a data annotation system based on the fuzzy comprehensive evaluation method provided by an embodiment of the present application.

[0052] As shown in Figure 1As shown in the figure, the data annotation system based on the fuzzy comprehensive evaluation method may include a first terminal 110, N second terminals 120, and users 130 corresponding to each second terminal 120. In the embodiments of the present application, the user 130 may be a physician, who is used to evaluate the nodules included in the to-be-annotated image (three-dimensional medical image of a single nodule) displayed on the second terminal 120, that is, to identify and classify the benign and malignant nature of the nodules included in the to-be-annotated image, that is, the risk type (such as high risk, medium risk, low risk). In the embodiments of the present application, the identification of the benign and malignant nature of lung nodules is taken as an example for illustration.

[0053] The first terminal 110 may include a deep learning algorithm for identifying medical images containing nodules with unknown risk types. This deep learning algorithm requires a large amount of accurate annotation data for training. In the embodiments of the present application, the first terminal 110 may also be used to communicate with N second terminals 120. Specifically, the first terminal 110 may send the to-be-annotated image to each second terminal 120, and the first terminal 110 may also receive the evaluation results of the to-be-annotated image returned by the second terminal 120, and determine the risk type of the nodules included in the to-be-annotated image according to the evaluation results of each to-be-annotated image returned by each second terminal 120.

[0054] The second terminal 120 may be used to receive the to-be-annotated image sent by the first terminal, and receive the evaluation results of each factor of the to-be-annotated image by the user 130, and send the evaluation results of each factor of the to-be-annotated image to the first terminal 110. In the embodiments of the present application, each to-be-annotated image includes M factors, such as but not limited to: nodule diameter, whether there is a burr, nodule type, nodule position, etc. These M factors may constitute a factor set U=(u 1 =nodule diameter, u 2 =whether there is a burr, u 3 =nodule type, u 4 =nodule position). Each factor corresponds to an evaluation set V=(v 1 =high risk, v 2 =medium risk, v 3 =low risk).

[0055] Data communication is carried out between the first terminal 110 and any one of the second terminals 120 through a network. The network may be a medium that provides a communication link between the first terminal 110 and any one of the second terminals 120, or may be the Internet including network devices and transmission media, which is not limited thereto. The transmission medium may be a wired link (such as but not limited to, coaxial cable, optical fiber, and digital subscriber line (DSL), etc.) or a wireless link (such as but not limited to, wireless fidelity (WIFI), Bluetooth, and mobile device network, etc.).

[0056] In the embodiments of the present application, the first terminal 110 and the second terminal 120 involved may be mobile phones, tablet computers, desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), wearable electronic devices, virtual reality devices, etc.

[0057] Next, in combination with Figure 1 the data annotation system shown based on the fuzzy comprehensive evaluation method, the data annotation method provided by the embodiments of the present application will be introduced. As Figure 2 shown, the data annotation method based on the fuzzy comprehensive evaluation method may at least include the following steps:

[0058] S201: Obtain the evaluation results of N users for M factors of the image to be annotated.

[0059] Among them, each factor corresponds to Y evaluation types.

[0060] Optionally, each factor corresponding to Y evaluation types includes at least two of the following: high risk, medium risk, low risk. In the embodiments of the present application, the case where the evaluation types corresponding to each factor include high risk, medium risk, and low risk will be taken as an example for description.

[0061] Specifically, the N users are Figure 1 the users corresponding to the second terminal 120 mentioned in

[0062] Optionally, the M factors include at least two of the following: nodule diameter, presence of burrs, nodule type, nodule location. In the embodiments of the present application, the case where the M factors include nodule diameter, presence of burrs, nodule type, and nodule location will be taken as an example for description.

[0063] Specifically, the nodule diameter is used to characterize the size of the nodules contained in the annotated image. Generally speaking, the larger the nodule diameter, the higher the corresponding risk level of the nodule.

[0064] Specifically, the presence of burrs is also an indicator for judging the risk level of nodules. Generally speaking, the risk level of nodules with burrs is higher than that of nodules without burrs.

[0065] Specifically, nodule types may include solid nodules, semi-solid nodules, ground-glass nodules, etc. Different types of nodules have different degrees of risk. Generally speaking, the degree of risk of ground-glass nodules is higher than that of solid nodules and semi-solid nodules. The degree of risk of solid nodules is higher than that of semi-solid nodules.

[0066] Specifically, the nodule location is used to characterize the specific location of the nodule in the lung, such as the lower lobe of the left lung, the middle lobe of the right lung, etc.

[0067] S202: Statistically analyze the distribution of the evaluation results of each factor respectively.

[0068] Specifically, each user will give corresponding evaluation results for each factor of the image to be annotated.

[0069] Exemplarily, in the embodiments of the present application, M = 4, Y = 3. The M factors of each image to be annotated constitute a factor set U = (u 1 = nodule diameter, u 2 = presence of spicules, u 3 = nodule type, u 4 = nodule location). Each factor respectively corresponds to an evaluation set V = (v 1 = high risk, v 2 = medium risk, v 3 = low risk).

[0070] For the factor u i of the image x to be annotated, count how many physicians evaluate it as v 1 , v 2 , v 3 , and denote them as n i1 , n i2 , n i3 . Among them:

[0071]

[0072] Then the membership degrees of the factor u i of the image x to be annotated to v 1 , v 2 , v 3 are respectively:

[0073] r i1 = n i1 / N, r i2 = n i2 / N, r i3 = n i3 / N.

[0074] S203: Determine the fuzzy comprehensive evaluation matrix of the image to be annotated according to the distribution of the evaluation results of each factor.

[0075] Among them, the fuzzy comprehensive evaluation matrix is a matrix of M×Y.

[0076] Specifically, according to the membership degrees of each factor of the to-be-annotated image x to the Y evaluation types, the fuzzy comprehensive evaluation matrix R of the to-be-annotated image is determined as R = [r ij M×Y . Wherein, i takes positive integers from 1 to M, and j takes positive integers from 1 to Y.

[0077] Exemplarily, if M = 4, Y = 3, the fuzzy comprehensive evaluation matrix R of the to-be-annotated image is as follows:

[0078]

[0079] S204: Obtain the weights corresponding to each of the M factors.

[0080] Specifically, for each element in the factor set, a weight parameter is given to obtain the weight vector of the factor set, denoted as A = (a 1 , a 2 , a 3 , a 4 ). That is to say, the weight of the factor of nodule diameter is a 1 , the weight of the factor of whether there are spicules is a 2 , the weight of the factor of nodule type is a 3 , and the weight of the factor of nodule position diameter is a 4 . It can be known that the sum of the weights is 1.

[0081] S205: Combine the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the to-be-annotated image.

[0082] Among them, the weight vector of the evaluation set is a 1×Y matrix. Each value in the weight vector is used to represent the membership degree of the to-be-annotated image to each evaluation result.

[0083] Specifically, for the pulmonary nodule image x, through fuzzy transformation, the weight vector of the factor set U is converted into the weight vector B = (b 1 , b 2 , b 3 ) = A°R of the evaluation set V, where ° is called the fuzzy evaluation composition operator. Here, the main factor prominent type composition operator is selected, that is ∨ is the conjunction operation, that is, take the maximum value of a i ×r ij .

[0084] For example, assume A = (0.3, 0.4, 0.2, 0.1), that is, a 1 ​= 0.3, a 2 = 0.4, a 3 = 0.2, a 4 = 0.1. Assume N is 100, and for the pulmonary nodule image x, the factor u 1 (nodule diameter) of the number of people (n 11 ) is 80, u 2 (whether there are spicules) of the number of people (n 21 ) is 50, u 3 (nodule type) of the number of people (n 31 ) is 60, u 4 (nodule location) of the number of people (n 41 ) is 30, then:

[0085] a 1 × r 11 = 0.3 × 80 / 100 = 0.24

[0086] a 2 × r 21 = 0.4 × 50 / 100 = 0.2

[0087] a 3 × r 31 = 0.2 × 60 / 100 = 0.12

[0088] a 4 × r 41 = 0.1 × 30 / 100 = 0.03

[0089] Then, That is to say, the membership degree of the image x to be labeled for high risk is 0.24.

[0090] It can be known that the above process of solving b 1 is also applicable to solving b 2 and b 3 , which will not be listed one by one here.

[0091] S206: Determine the evaluation result of the image to be labeled according to the Y values included in the weight vector of the evaluation set.

[0092] Specifically, the weight vector of the evaluation set includes the membership degrees corresponding to the Y evaluation types of the image to be labeled. The evaluation result of the image to be labeled is the evaluation type with the highest membership degree among the membership degrees corresponding to the Y evaluation types of the image to be labeled.

[0093] For example, if the membership degree of the image to be labeled for high risk is 0.24, the membership degree for medium risk is 0.5, and the membership degree for low risk is 0.27, it can be seen that among the membership degrees corresponding to the Y evaluation types of the image to be labeled, the evaluation type with the highest membership degree is medium risk. Then, the evaluation result of the image to be labeled is medium risk.

[0094] In the embodiment of the present application, by obtaining the evaluation results of N users for M factors of the image to be labeled, each factor corresponding to Y evaluation types; respectively counting the distribution of the evaluation results of each factor; determining the fuzzy comprehensive evaluation matrix of the image to be labeled according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is a matrix of M×Y; obtaining the weights corresponding to each of the M factors; combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the image to be labeled; the weight vector of the evaluation set is a matrix of 1×Y; and determining the evaluation result of the image to be labeled according to the weight vector of the evaluation set, the fuzzy comprehensive evaluation method can be used to quantify and fuse the labeling processes of multiple physicians, eliminate the influence of subjective differences between different physicians, obtain a relatively objective labeling result, solve the problem of inconsistent multi-person labeling caused by the ambiguity of problem definition, and improve the accuracy of nodule marking results.

[0095] Figure 3 Another data labeling method of the fuzzy comprehensive evaluation method provided by the embodiment of the present application is exemplarily shown, which may specifically include the following steps:

[0096] S301: Determine the data set to be labeled.

[0097] Among them, the data set to be labeled includes X images to be labeled.

[0098] Specifically, an injection can be constructed such that each x in the data set to be labeled X = {x} is uniquely mapped to an element in the labeling set Y = {high risk, medium risk, low risk}, where x is the three-dimensional image of a single lung nodule. In addition, the factor set U = (u 1 = nodule diameter, u 2 = whether there is a burr, u 3 = nodule type, u 4 = nodule position), and the evaluation set V = (v 1 = high risk, v 2 = medium risk, v 3 = low risk).

[0099] S302: Send the data set to be labeled to N terminals respectively.

[0100] Among them, the N terminals correspond to N users. It can be known that the N terminals receiving the data set here are Figure 1The second terminal mentioned in []. That is, N doctors are invited to participate in the annotation task, and the dataset X to be annotated is sent to these N doctors respectively. For each x, each doctor is required to select an evaluation from the evaluation set V for each factor in the factor set U to obtain v j = δ(k, x, u i ). v j represents the evaluation result of the k-th doctor on the factor u i of the pulmonary nodule image x. Among them, k represents the number of the terminal, and the value range of k is a positive integer from 1 to N. x represents the number of the image to be annotated, and the value range of x is a positive integer from 1 to X. u i represents the number of the factor, and the value range of i is a positive integer from 1 to M. v j represents the evaluation result, and the value range of j is a positive integer from 1 to Y. The annotation of a single factor of a single image to be annotated by the above doctor is recorded as one annotation action.

[0101] S303: Receive the evaluation results v j = δ(k, x, u i ) for each factor of the X images to be annotated in the dataset to be annotated from each terminal, and obtain an evaluation result set.

[0102] Specifically, the terminal here is Figure 1 the second terminal mentioned in []. After all N doctors have completed the annotation actions on the dataset X to be annotated, collect all k, x, u i , v j to obtain an evaluation result set.

[0103] Figure 4 Exemplarily shows a schematic diagram of an evaluation result set provided by an embodiment of the present application. As Figure 4 shown, taking the image x to be annotated as a reference, the evaluation results of each doctor k on the factor u i of the image x to be annotated can be obtained. For example, the evaluation result of the second doctor (k = 2) on the nodule diameter (u 1 ) of the image to be annotated 1 (x = 1) is medium risk (v 2 ), and the evaluation result of the 100th doctor (k = 100) on the nodule type (u 3 ) of the image to be annotated 2 (x = 2) is high risk (v 1 ), and the evaluation result of the 3rd doctor (k = 3) on whether there is a burr (u 2 ) of the image to be annotated 1000 (x = 1000) is high risk (v 1 ).

[0104] S304: Obtain the evaluation results of N users on the M factors of the image to be annotated from the evaluation result set according to the number of the image to be annotated.

[0105] Specifically, after determining the number of the image to be labeled, the evaluation results of N users for M factors of the image to be labeled can be obtained from the evaluation result set. The specific obtaining process is the same as that of S201 and will not be elaborated here.

[0106] S305: Statistically analyze the distribution of the evaluation results of each factor respectively.

[0107] Specifically, S305 is the same as S202 and will not be elaborated here.

[0108] S306: Determine the fuzzy comprehensive evaluation matrix of the image to be labeled according to the distribution of the evaluation results of each factor.

[0109] Specifically, S306 is the same as S203 and will not be elaborated here.

[0110] S307: Obtain the weights corresponding to each of the M factors.

[0111] Specifically, S307 is the same as S204 and will not be elaborated here.

[0112] S308: Combine the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the image to be labeled.

[0113] Specifically, S308 is the same as S205 and will not be elaborated here.

[0114] S309: Determine the evaluation result of the image to be labeled according to the Y values included in the weight vector of the evaluation set.

[0115] Specifically, S309 is the same as S206 and will not be elaborated here.

[0116] In the embodiment of the present application, by obtaining the evaluation results of N users for M factors of the image to be labeled, each factor corresponding to Y evaluation types; statistically analyzing the distribution of the evaluation results of each factor respectively; determining the fuzzy comprehensive evaluation matrix of the image to be labeled according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×Y matrix; obtaining the weights corresponding to each of the M factors; combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the image to be labeled; the weight vector of the evaluation set is a 1×Y matrix; determining the evaluation result of the image to be labeled according to the weight vector of the evaluation set, the fuzzy comprehensive evaluation method can be used to quantify and fuse the labeling processes of multiple physicians, eliminate the influence of subjective differences between different physicians, obtain a relatively objective labeling result, solve the problem of inconsistent multi-person labeling caused by the ambiguity of problem definition, and improve the accuracy of nodule marking results.

[0117] Figure 5 Exemplarily shown is a schematic structural diagram of a data annotation device provided by an embodiment of the present application based on the fuzzy comprehensive evaluation method. As Figure 5 shown, the data annotation device 50 based on the fuzzy comprehensive evaluation method at least includes:

[0118] A first acquisition module 501, configured to acquire evaluation results of N users for M factors of an image to be annotated, where each factor corresponds to Y evaluation types;

[0119] A statistical module 502, configured to respectively count the distribution of evaluation results of each factor;

[0120] A first determination module 503, configured to determine a fuzzy comprehensive evaluation matrix of the image to be annotated according to the distribution of evaluation results of each factor; the fuzzy comprehensive evaluation matrix is an M×N matrix;

[0121] A second acquisition module 504, configured to acquire weights corresponding to each of the M factors;

[0122] A second determination module 505, configured to combine the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine a weight vector of an evaluation set of the image to be annotated; the weight vector of the evaluation set is a 1×N matrix;

[0123] A third determination module 506, configured to determine an evaluation result of the image to be annotated according to Y values included in the weight vector of the evaluation set.

[0124] In some possible embodiments, as Figure 6 shown, the data annotation device 50 based on the fuzzy comprehensive evaluation method may further include:

[0125] A fourth determination module 507, configured to determine a dataset to be annotated; the dataset to be annotated includes X images to be annotated;

[0126] A sending module 508, configured to respectively send the dataset to be annotated to N terminals; the N terminals correspond to N users;

[0127] A receiving module 509, receiving evaluation results v j =δ(k,x,u i ) of each factor of the X images to be annotated in the dataset to be annotated from each terminal, and obtaining an evaluation result set; where k represents the number of the terminal, k ranges from 1 to N, x represents the number of the image to be annotated, x ranges from 1 to X, u i represents the number of the factor, i ranges from 1 to M, and v j represents the evaluation result, j ranges from 1 to Y;

[0128] The first acquisition module 501 is specifically configured to acquire the evaluation results of N users on M factors of the to-be-annotated image from the evaluation result set according to the number of the to-be-annotated image.

[0129] In some possible embodiments, the statistics module 502 is specifically configured to: count the distribution of the evaluation results of the N users on the factor u of the to-be-annotated image x i ; the distribution is that there are n ij users whose evaluation results on the factor u of the to-be-annotated image x i are v j , and

[0130]

[0131] The first determination module 503 includes:

[0132] The first determination subunit is configured to determine the membership degree r i of the factor u of the to-be-annotated image x to the Y evaluation types ij = n ij / N;

[0133] The second determination subunit is configured to determine the fuzzy comprehensive evaluation matrix R = [r ij M×Y of the to-be-annotated image according to the membership degrees of the respective factors of the to-be-annotated image x to the Y evaluation types.

[0134] In some possible embodiments, the weights corresponding to the M factors are represented by a 1×M matrix, and the 1×M matrix is the weight vector A = (a 1 , a 2 , …, a M ) of the factor set of the to-be-processed image;

[0135] The second determination module 505 is specifically configured to: convert the weight vector of the factor set of the to-be-processed image into a weight vector of the evaluation set through fuzzy transformation; the is the synthetic operator of the dominant factor prominent type.

[0136] In some possible embodiments, the weight vector of the evaluation set includes the membership degrees corresponding to the Y evaluation types of the to-be-annotated image respectively;

[0137] The third determination module 506 is specifically configured to: determine the evaluation type with the highest membership degree among the membership degrees corresponding to the Y evaluation types of the to-be-annotated image as the evaluation result of the to-be-annotated image.

[0138] In some possible embodiments, the M factors include at least two of the following: nodule diameter, presence of spicules, nodule type, and nodule location.

[0139] In some possible embodiments, each of the factors corresponds to Y evaluation types including at least two of the following: high risk, medium risk, and low risk.

[0140] In the embodiments of the present application, by using the fuzzy comprehensive evaluation method to quantify and fuse the annotation processes of multiple physicians, the influence of subjective differences between different physicians is eliminated, an objective annotation result is obtained, the problem of inconsistent multi-person annotation caused by the ambiguity of problem definition is solved, and the accuracy of nodule marking results is improved.

[0141] The division of each module in the above data annotation device based on the fuzzy comprehensive evaluation method is only for illustration. In other embodiments, the data erasure device can be divided into different modules as needed to complete all or part of the functions of the above data erasure device. The implementation of each module in the data erasure device provided in the embodiments of the present application can be in the form of a computer program. This computer program can run on a terminal or a server. The program module constituted by this computer program can be stored in the memory of the terminal or the server. When this computer program is executed by a processor, all or part of the steps of the data erasure method described in the embodiments of the present application are implemented.

[0142] Please refer to Figure 7 , Figure 7 which shows a schematic structural diagram of an electronic device provided in the embodiments of the present application.

[0143] As Figure 7 shown, the electronic device 700 may include: at least one processor 701, at least one network interface 704, a user interface 703, a memory 705, a touch screen 706, and at least one communication bus 702.

[0144] Among them, the communication bus 702 can be used to realize the connection and communication of the above-mentioned various components.

[0145] Among them, the user interface 703 may include buttons, and the optional user interface may further include a standard wired interface and a wireless interface.

[0146] Among them, the network interface 704 may optionally include a Bluetooth module, an NFC module, a Wi-Fi module, etc.

[0147] Among them, the processor 701 may include one or more processing cores. The processor 701 connects various parts within the entire electronic device 700 through various interfaces and lines, and executes various functions of the routing device 700 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 705, and by calling the data stored in the memory 705. Optionally, the processor 701 may be implemented in at least one hardware form of DSP, FPGA, or PLA. The processor 701 may integrate one or a combination of several of CPU, GPU, and modem, etc. Among them, the CPU mainly processes the operating system, user interface, application programs, etc.; the GPU is responsible for rendering and drawing the content to be displayed on the display screen; the modem is used to process wireless communication. It can be understood that the above-mentioned modem may not be integrated into the processor 701 and may be implemented separately by a single chip.

[0148] Among them, the memory 705 may include RAM and may also include ROM. Optionally, the memory 705 includes a non-transitory computer-readable medium. The memory 705 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 705 may include a program storage area and a data storage area. Among them, the program storage area may store instructions for implementing the operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store the data involved in the above-mentioned various method embodiments. Optionally, the memory 705 may also be at least one storage device located far from the aforementioned processor 701. As Figure 7 shown, the memory 705 as a computer storage medium may include an operating system, a network communication module, a user interface module, and application programs.

[0149] Specifically, the processor 701 may be used to call the application program stored in the memory 705 and specifically perform the following operations:

[0150] Obtain the evaluation results of M factors of N users for the image to be labeled, and each factor corresponds to Y evaluation types;

[0151] Statistically analyze the distribution of the evaluation results of each factor respectively;

[0152] Determine the fuzzy comprehensive evaluation matrix of the image to be labeled according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is a matrix of M×Y;

[0153] Obtain the weights corresponding to each of the M factors;

[0154] Determine the weight vector of the evaluation set of the to-be-annotated image by combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix; the weight vector of the evaluation set is a 1×Y matrix;

[0155] Determine the evaluation result of the to-be-annotated image according to the Y values included in the weight vector of the evaluation set.

[0156] In some possible embodiments, before the processor 701 obtains the evaluation results of N users for the M factors of the to-be-annotated image, it is further configured to execute:

[0157] Determine the to-be-annotated data set; the to-be-annotated data set includes X to-be-annotated images;

[0158] Send the to-be-annotated data set to N terminals respectively; the N terminals correspond to N users;

[0159] Receive the evaluation result v j =δ(k,x,u i ) of each factor of the X to-be-annotated images in the to-be-annotated data set for each terminal, and obtain an evaluation result set; where k represents the number of the terminal, the value range of k is a positive integer from 1 to N, x represents the number of the to-be-annotated image, the value range of x is a positive integer from 1 to X, u i represents the number of the factor, the value range of i is a positive integer from 1 to M, and v j represents the evaluation result, the value range of j is a positive integer from 1 to Y;

[0160] When the processor 701 obtains the evaluation results of N users for the M factors of the to-be-annotated image, it is specifically configured to execute:

[0161] Obtain the evaluation results of N users for the M factors of the to-be-annotated image from the evaluation result set according to the number of the to-be-annotated image.

[0162] In some possible embodiments, when the processor 701 respectively counts the distribution of the evaluation results of each factor, it is specifically configured to execute:

[0163] Count the distribution of the evaluation results of the factor u i of the N users for the to-be-annotated image x; the distribution is that there are n ij users whose evaluation results for the factor u i of the to-be-annotated image x are v j , and

[0164]

[0165] When the processor 701 determines the fuzzy comprehensive evaluation matrix of the to-be-annotated image according to the distribution of the evaluation results of each factor, it is specifically configured to execute:

[0166] Factor u that determines the image x to be labeled i Membership degree r to the Y evaluation types ij = n ij / N;

[0167] Determine the fuzzy comprehensive evaluation matrix R of the image x to be labeled according to the membership degrees of the Y evaluation types with respect to each factor of the image x to be labeled: R = [r ij M×Y .

[0168] In some possible embodiments, the weights corresponding to the M factors are represented by a 1×M matrix, and the 1×M matrix is the weight vector A=(a 1 , a 2 , …, a M ) of the factor set of the image to be processed;

[0169] When the processor 701 determines the weight vector of the evaluation set of the image x to be labeled in combination with the weights corresponding to the M factors and the fuzzy comprehensive evaluation matrix, it is specifically configured to perform:

[0170] Convert the weight vector of the factor set of the image to be processed into the weight vector B of the evaluation set through fuzzy transformation; the is the dominant factor prominent type composition operator.

[0171] In some possible embodiments, the weight vector of the evaluation set includes the membership degrees corresponding to the Y evaluation types of the image x to be labeled;

[0172] When the processor 701 determines the evaluation result of the image x to be labeled according to the Y values included in the weight vector of the evaluation set, it is specifically configured to perform: determine the evaluation type with the highest membership degree among the membership degrees corresponding to the Y evaluation types of the image x to be labeled as the evaluation result of the image x to be labeled.

[0173] In some possible embodiments, the M factors include at least two of the following: nodule diameter, presence of spicules, nodule type, nodule location.

[0174] In some possible embodiments, each factor corresponding to the Y evaluation types includes at least two of the following: high risk, medium risk, low risk.

[0175] By using the fuzzy comprehensive evaluation method in the embodiments of the present application to quantify and fuse the labeling processes of multiple physicians, the influence of subjective differences between different physicians is eliminated, a relatively objective labeling result is obtained, the problem of inconsistent multi-person labeling caused by the ambiguity of problem definition is solved, and the accuracy of the nodule marking result is improved. ​

[0176] An embodiment of the present application further provides a computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions run on a computer or a processor, the computer or the processor is caused to execute one or more steps in the above Figure 2 or Figure 3 shown embodiments. If each component module of the above electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in the computer-readable storage medium.

[0177] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, Digital Subscriber Line (DSL)) or a wireless manner (such as infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or a data center that includes one or more integrated available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a Digital Versatile Disc (DVD)), or a semiconductor medium (for example, a Solid State Disk (SSD)), etc.

[0178] Those of ordinary skill in the art can understand that all or part of the processes of implementing the methods in the above embodiments can be completed by instructing relevant hardware through a computer program. The program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above methods. The foregoing storage media include various media that can store program codes, such as ROM, RAM, magnetic disks, or optical discs. Without conflict, the technical features in this embodiment and the implementation solutions can be combined arbitrarily.

[0179] The embodiments described above are only described as preferred embodiments of the present application, and do not limit the scope of the present application. Without departing from the design spirit of the present application, various deformations and improvements made by those of ordinary skill in the art to the technical solutions of the present application shall fall within the protection scope determined by the claims of the present application.

Claims

1. A data annotation method based on the fuzzy comprehensive evaluation method, characterized in that, comprising: Determine the dataset to be annotated; the dataset to be annotated includes X images to be annotated; Send the dataset to be annotated to N terminals respectively; the N terminals correspond to N users; Receive the evaluation result v of each factor for each of the X images to be labeled in the dataset to be labeled j = δ(k, x, u i ), to obtain an evaluation result set; where k represents the number of the terminal, and the value range of k is a positive integer from 1 to N, x represents the number of the image to be labeled, and the value range of x is a positive integer from 1 to X, u i represents the number of the factor, and the value range of i is a positive integer from 1 to M, v j represents the evaluation result, and the value range of j is a positive integer from 1 to Y; Obtain the evaluation results of N users for M factors of the images to be annotated, and each factor corresponds to Y evaluation types; Statistically analyze the distribution of the evaluation results of each factor respectively; Determine the fuzzy comprehensive evaluation matrix of the images to be annotated according to the distribution of the evaluation results of each factor; the fuzzy comprehensive evaluation matrix is a matrix of M×Y; Obtain the weights corresponding to each of the M factors; Combine the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the images to be annotated; the weight vector of the evaluation set is a matrix of 1×Y; Determine the evaluation result of the images to be annotated according to the Y values included in the weight vector of the evaluation set; The step of statistically analyzing the distribution of the evaluation results of each factor respectively includes: Statistically analyze the factors u of the N users for the image x to be annotated i of the evaluation results; the distribution is such that there are n ij users whose evaluation results for the factor u of the image x to be annotated i are v j , and The step of determining the fuzzy comprehensive evaluation matrix of the images to be annotated according to the distribution of the evaluation results of each factor includes: Factor u that determines the image x to be labeled i Membership degree r to the Y evaluation types ij = n ij / N; Determine the fuzzy comprehensive evaluation matrix \(R = [r\) of the image \(x\) to be labeled according to the membership degrees of each factor of the image \(x\) to be labeled to the \(Y\) evaluation types ij M×Y .​ 2. The method according to claim 1, characterized in that, The step of obtaining the evaluation results of N users for M factors of the images to be annotated includes: Obtain the evaluation results of N users for M factors of the images to be annotated from the evaluation result set according to the numbers of the images to be annotated.

3. The method according to claim 1, characterized in that, The weights corresponding to the M factors are represented by a 1×M matrix, and the 1×M matrix is the weight vector A=(a 1 , a 2 , …, a M ) of the factor set of the image to be labeled; The step of combining the weights corresponding to each of the M factors and the fuzzy comprehensive evaluation matrix to determine the weight vector of the evaluation set of the images to be annotated includes: Through fuzzy transformation, the weight vector of the factor set of the image to be labeled is converted into the weight vector B of the evaluation set; the B=(b 1 ,b 2 ,b 3 ) = A°R, where ° is the main factor prominent type composition operator.

4. The method according to claim 3, characterized in that, The weight vector of the evaluation set includes the membership degrees corresponding to each of the Y evaluation types of the images to be annotated; The step of determining the evaluation result of the images to be annotated according to the Y values included in the weight vector of the evaluation set includes: determining the evaluation type with the highest membership degree among the membership degrees corresponding to each of the Y evaluation types of the images to be annotated as the evaluation result of the images to be annotated.

5. The method according to claim 1, characterized in that, The M factors include at least two of the following: nodule diameter, presence of burrs, nodule type, nodule location.

6. The method according to claim 1, characterized in that, Each factor corresponding to Y evaluation types includes at least two of the following: high risk, medium risk, low risk.

7. A data annotation device based on the fuzzy comprehensive evaluation method, characterized in that, comprising: A first acquisition module for determining the dataset to be annotated; the dataset to be annotated includes X images to be annotated; Send the dataset to be annotated to N terminals respectively; the N terminals correspond to N users; Receive the evaluation result v of each factor for each of the X images to be annotated in the dataset to be annotated for each terminal j = δ(k, x, u i ), to obtain an evaluation result set; where k represents the number of the terminal, and the value range of k is a positive integer from 1 to N, x represents the number of the image to be annotated, and the value range of x is a positive integer from 1 to X, u i represents the number of the factor, and the value range of i is a positive integer from 1 to M, v j represents the evaluation result, and the value range of j is a positive integer from 1 to Y; obtain the evaluation results of M factors of the images to be annotated by N users, and each factor corresponds to Y evaluation types; A statistical module for statistically analyzing the distribution of the evaluation results of each factor respectively; A first determination module for determining the fuzzy comprehensive evaluation matrix of the images to be annotated according to the distribution of the evaluation results of each factor; The fuzzy comprehensive evaluation matrix is a matrix of M×N; A second acquisition module for obtaining the weights corresponding to each of the M factors; A second determination module, configured to determine a weight vector of the evaluation set of the to-be-annotated image by combining the weights corresponding to the M factors respectively and the fuzzy comprehensive evaluation matrix; The weight vector of the evaluation set is a 1×N matrix; A third determination module, configured to determine an evaluation result of the to-be-annotated image according to Y values included in the weight vector of the evaluation set; The specific function of the statistical module is to: statistically analyze the distribution of the evaluation results of the N users on the factor u of the image x to be annotated i ; the distribution is such that there are n ij users whose evaluation results on the factor u i of the image x to be annotated are v j , and The first determination module is specifically configured to: determine a factor u of the image x to be labeled i Degree of membership r to the Y evaluation types ij = n ij / N; Determine the fuzzy comprehensive evaluation matrix \(R = [r\) of the image \(x\) to be labeled according to the membership degrees of the various factors of the image \(x\) to be labeled to the \(Y\) evaluation types ij M×Y .​ 8. A computer storage medium, characterized in that, the computer storage medium stores multiple instructions, and the instructions are adapted to be loaded and executed by a processor to perform the method steps of any one of claims 1-6.

9. An electronic device, characterized in that, comprising: a processor and a memory; wherein, the memory stores a computer program, and the computer program is adapted to be loaded and executed by the processor to perform the method steps of any one of claims 1-6.

Images