Multi-round medical image labeling method and device introducing cross validation mechanism

Abstract

The embodiment of the invention relates to a multi-round medical image annotation method and device introducing a cross validation mechanism, and the method comprises the steps: transmitting a first image to a plurality of expert annotation interfaces, and merging the feedback annotation sets of all interfaces to obtain a first annotation set; setting cross validation items of every two annotation boxes in the first annotation set and constructing a corresponding cross validation matrix; on the basis of a cross validation matrix, dispute-free annotation box extraction, dispute-free annotation box aggregation and consistency score estimation are carried out; if the score is lower than a threshold value, a round of multi-person labeling is initiated again, otherwise, when the dispute labeling box set is empty, the non-dispute labeling box set serves as a final labeling result, when the dispute labeling box set is not empty, reexamination is conducted on the dispute labeling box set, and a set of the reexamination result and the non-dispute labeling box set serves as the final labeling result. According to the method, the labeling error of each image can be reduced, the overall labeling quality of all the images is improved, and the stability of the overall labeling quality is guaranteed.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a multi-round medical image labeling method and device with a cross-validation mechanism. BACKGROUND

[0002] Medical image labeling is an important part of medical image analysis, and the labeled medical images can be used for clinical diagnosis and artificial intelligence model training. Artificial labeling based on expert opinions is currently a common labeling method. This conventional labeling method is mostly based on a single expert opinion for labeling in specific implementation.

[0003] However, through practical observation, it can be found that this single-person labeling method often has the problem of large labeling quality fluctuation. The main reasons are as follows: 1) not every expert has sufficient full-modal image labeling experience, and in the case of insufficient cross-modal labeling experience of experts, large labeling errors may occur, leading to a decrease in labeling quality; for example, an expert has rich experience in labeling CT images, but poor experience in labeling MRI images, and if the expert's experience is used to label MRI images, large errors will occur. 2) Lack of professional comparison and verification mechanism, after each expert completes the labeling, only the standardization of the labeling behavior is checked, and the professional nature is not compared and verified, which may produce labeling errors and lead to a decrease in labeling quality in the case of insufficient pathological knowledge of the expert; for example, after an expert labels the lesion area of a stomach CT image, the traditional method only checks whether the lesion type labeled by the expert belongs to the category of stomach diseases, but does not professionally verify whether the stomach disease type labeled by the expert is appropriate, and if the stomach disease type labeled by the expert does not match the actual situation, a labeling error will occur.

[0004] To solve the labeling quality fluctuation problem caused by the above two reasons, a multi-round labeling scheme with a cross-validation mechanism is designed: single-round multi-person labeling is used to replace the traditional single-round single-person labeling, and a cross-validation mechanism is introduced to aggregate non-disputed labeling boxes, identify disputed labeling boxes, and evaluate labeling consistency for each round of multi-person labeling results, and when the consistency evaluation does not meet the standard, a new round of multi-person labeling is initiated until the standard is met, and after the consistency evaluation meets the standard, a review is conducted again to verify the remaining disputed labeling boxes. Based on this design scheme, the labeling error of each image can be effectively reduced, the overall labeling quality can be improved, and the stability of the overall labeling quality can be effectively guaranteed. However, how to implement this design scheme is a technical problem to be solved by the present application. SUMMARY

[0005] The present application aims at the defects of the prior art, and provides a multi-round medical image labeling method and device introducing a cross-validation mechanism, electronic equipment and a computer readable storage medium. When labeling a medical image to be labeled, the present application first randomly selects multiple expert labeling interfaces from an expert library to independently label the current image in one round; then cross- validates the multi-labeling results in this round in a pairwise comparison manner to obtain a corresponding cross-validation matrix A; then performs controversial labeling box extraction, non-controversial labeling box aggregation and consistency score estimation based on the cross-validation matrix A; if the consistency score does not meet the standard, a new round of multi-labeling is initiated until the latest consistency score meets the standard; if the consistency score meets the standard, it is identified whether the latest controversial labeling box set is empty, if not, the latest non-controversial labeling box set is directly taken as the final labeling result, if yes, the controversial labeling box set is reviewed in one round, and the final labeling result is composed of the review feedback and the non-controversial labeling box set; and the corresponding labeling record composed of the original medical image and the final labeling result is added to the labeling database. The present application reduces the labeling error of each image, improves the overall labeling quality of all images through the multi-labeling mechanism, cross-validation mechanism, multi-labeling restart mechanism based on the consistency score and the final review mechanism, and effectively guarantees the stability of the overall labeling quality while improving the overall labeling quality.

[0006] To achieve the above object, the first aspect of the embodiment of the present application provides a multi-round medical image labeling method introducing a cross-validation mechanism, which comprises:

[0007] Step 1, taking an original medical image to be labeled as a first image;

[0008] Step 2, randomly selecting X expert labeling interfaces from a preset expert library to form a current interface set; sending the first image to all the expert labeling interfaces of the current interface set; and merging X expert labeling sets fed back by all the expert labeling interfaces to obtain a corresponding first labeling set; X is a preset positive integer; each expert labeling set is composed of one or more first labeling boxes; the first labeling box comprises a first center point coordinate, a first width, a first height and a first labeling text;

[0009] Step 3, counting the total number of labeling boxes in the first labeling set to obtain a corresponding total number M; calculating the center point distance, the intersection and union ratio and the text similarity of each two first labeling boxes in the first labeling set, and setting a corresponding cross-validation item based on the calculation result; and constructing a cross-validation matrix A with a shape of MxM based on all the obtained cross-validation items;

[0010] Step 4, performing controversial annotation box extraction and non-controversial annotation box aggregation based on the cross-validation matrix A, and estimating the consistency score based on the processing result to obtain a corresponding controversial annotation box set, a non-controversial annotation box set and a first score;

[0011] Step 5, identifying whether the first score is lower than a preset consistency score threshold; if yes, returning to step 2 to initiate a new round of expert annotation; if no, proceeding to step 6;

[0012] Step 6, identifying whether the controversial annotation box set is empty; if yes, taking the non-controversial annotation box set as a corresponding first label annotation set; if no, sending the first image and the controversial annotation box set to a preset annotation review interface, and merging a review annotation box set fed back by the annotation review interface with the non-controversial annotation box set to obtain a corresponding first label annotation set;

[0013] Step 7, adding a corresponding first annotation record composed of the first image and the first label annotation set to a preset annotation database.

[0014] Preferably, the cross-validation item is a binary data with a value of 0 or 1;

[0015] Each row or column of the cross-validation matrix A corresponds to a first annotation box; the cross-validation matrix A is composed of MxM matrix units a i,j , 1≤row index i≤M, 1≤column index j≤M; each unit value of the matrix unit a i,j is 0 or 1; the unit values of M diagonal line matrix units a i＝1,j＝1 on the matrix diagonal line from the top-left vertex matrix unit a i＝M,j＝M to the bottom-right vertex matrix unit a 1≤i≤M,j＝i of the cross-validation matrix A are all 1;

[0016] The controversial annotation box set consists of one or more second annotation boxes when not empty; the second annotation box includes a second center point coordinate, a second width, a second height and a second annotation text;

[0017] The non-controversial annotation box set consists of one or more third annotation boxes when not empty; the third annotation box includes a third center point coordinate, a third width, a third height and a third annotation text;

[0018] The review annotation box set consists of one or more fourth annotation boxes; the fourth annotation box includes a fourth center point coordinate, a fourth width, a fourth height and a fourth annotation text.

[0019] Preferably, the center point distance between every two first annotation boxes in the first annotation set, the intersection over union of the annotation boxes, and the text similarity are calculated, and the corresponding cross-validation items are set based on the calculation results, specifically including:

[0020] The M first annotation boxes in the first annotation set are combined in pairs to obtain corresponding N annotation box combinations; N = M(M-1) / 2;

[0021] Each annotation box combination is taken as a current combination, the straight-line distance between the two first center point coordinates of the current combination is calculated to obtain a corresponding first distance, and based on the two sets of first center point coordinates, the first width, and the first height of the current combination, two annotation box regions are constructed in the first image, which are referred to as first and second regions, the region intersection over union of the first and second regions is calculated to obtain a corresponding first intersection over union, and based on a pre-set medical text encoder, the two first annotation texts of the current combination are respectively processed for feature encoding to obtain first and second feature vectors, and based on a pre-set vector similarity algorithm, the vector similarity of the first and second feature vectors is calculated to obtain a corresponding first similarity, and a corresponding combination comparison result is composed of the first distance, the first intersection over union, and the first similarity of the current combination. Wherein, the medical text encoder is a conventional word embedding encoding model or a large language model that has been pre-trained based on a natural language corpus and fine-tuned based on a medical domain corpus; the word embedding encoding model includes a bag-of-words model, a TF-IDF model, and an N-gram model; the large language model includes a BERT series model and a GPT series model;

[0022] Each combination comparison result is taken as a current comparison result, and the first distance, the first intersection over union, and the first similarity of the current comparison result are identified, if the first distance is less than or equal to a pre-set first distance threshold, the first intersection over union is greater than or equal to a pre-set first intersection over union threshold, and the first similarity is greater than or equal to a pre-set first similarity threshold, the corresponding cross-validation item is set to 1, if the first distance is greater than the first distance threshold, the first intersection over union is less than the first intersection over union threshold, or the first similarity is less than the first similarity threshold, the corresponding cross-validation item is set to 0.

[0023] Preferably, based on all the obtained cross-validation items, an MxM-shaped cross-validation matrix A is constructed, specifically including:

[0024] An all-zero matrix with a shape of MxM is set as the initialized cross-validation matrix A, and M matrix elements a on the matrix diagonal line of the cross-validation matrix A are set to 1, and the rest of the matrix elements a are set to 0.i,j the cell value of the matrix cell a i,j is reset to 1.

[0025] Preferably, the controversial annotation frame extraction and non-controversial annotation frame aggregation processing based on the cross-validation matrix A are performed, and the consistency score is estimated based on the processing result to obtain the corresponding controversial annotation frame set, non-controversial annotation frame set and first score, which specifically includes:

[0026] a round of row-by-row traversal is performed on the cross-validation matrix A; and in the current round of traversal, the current row is taken as the current row; and the column index j of the matrix cell a i,j with a cell value of 1 in the current row is taken as a corresponding first identifier; and all the first identifiers corresponding to the current row form a corresponding first identifier set; and at the end of the current round of traversal, the M first identifier sets obtained are processed for deduplication;

[0027] a round of traversal is performed on all the first identifier sets after deduplication; and in the current round of traversal, the first identifier set currently being traversed is taken as the current identifier set; and it is identified whether the total number of identifiers of the current identifier set is 1; if yes, the first annotation frame corresponding to the only first identifier in the current identifier set is taken as a corresponding second annotation frame; if not, all the first annotation frames corresponding to the current identifier set are subjected to annotation frame aggregation processing to obtain a corresponding third annotation frame; and after the current round of traversal ends, all the second annotation frames obtained form the corresponding controversial annotation frame set, and all the third annotation frames obtained form the corresponding non-controversial annotation frame set;

[0028] The total number of annotation frames of the controversial annotation frame set and the non-controversial annotation frame set obtained this time is counted to obtain the corresponding first and second totals; and the first score is estimated based on the first and second totals, first score = second total / (first total + second total).

[0029] Further, the all first annotation frames corresponding to the current identifier set are subjected to annotation frame aggregation processing to obtain a corresponding third annotation frame, which specifically includes:

[0030] The mean value of all the first center point coordinates corresponding to the current identification set is calculated to obtain a corresponding average coordinate, the mean value of all the first widths is calculated to obtain a corresponding average width, and the mean value of all the first heights is calculated to obtain a corresponding average height; the average coordinate, the average width and the average height obtained are taken as a corresponding third center point coordinate, a third width and a third height; and an optional one of all the first annotation texts corresponding to the current identification set is taken as a corresponding third annotation text; and the third center point coordinate, the third width, the third height and the third annotation text obtained this time form a corresponding third annotation frame.

[0031] The second aspect of the embodiment of the application provides a device for implementing the multi-round medical image annotation method with the cross-validation mechanism introduced in the first aspect, and the device comprises an image receiving module, a multi-person annotation module, a cross-validation preprocessing module, a cross-validation post-processing module, a consistency compliance evaluation module, an annotation review module and an annotation record adding module.

[0032] The image receiving module is configured to take an original medical image to be annotated as a first image.

[0033] The multi-person annotation module is configured to randomly select X expert annotation interfaces from a preset expert library to form a current round interface set, send the first image to all the expert annotation interfaces in the current round interface set, and combine X expert annotation sets fed back by all the expert annotation interfaces to obtain a corresponding first annotation set; X is a preset positive integer; each expert annotation set comprises one or more first annotation frames; and the first annotation frame comprises a first center point coordinate, a first width, a first height and a first annotation text.

[0034] The cross-validation preprocessing module is configured to count the total number of annotation frames in the first annotation set to obtain a corresponding total number M, calculate the center point distance, the intersection-over-union ratio and the text similarity of each two first annotation frames in the first annotation set, set a corresponding cross-validation item based on the calculation result, and construct a cross-validation matrix A with a shape of MxM based on all the cross-validation items obtained.

[0035] The cross-validation post-processing module is configured to perform controversial annotation frame extraction and non-controversial annotation frame aggregation processing based on the cross-validation matrix A, and perform consistency scoring estimation based on the processing result to obtain a corresponding controversial annotation frame set, a non-controversial annotation frame set and a first score.

[0036] The consistency meeting evaluation module is configured to identify whether the first score is lower than a preset consistency score threshold, and if yes, return the multi-person labeling module to initiate a new round of expert labeling, and if no, go to the labeling review module;

[0037] The labeling review module is configured to identify whether the controversial annotation box set is empty, and if yes, take the non-controversial annotation box set as a corresponding first label annotation set, and if no, send the first image and the controversial annotation box set to a preset labeling review interface, and merge a review annotation box set fed back by the labeling review interface and the non-controversial annotation box set to obtain the corresponding first label annotation set.

[0038] The labeling record adding module is configured to add a corresponding first labeling record composed of the first image and the first label annotation set to a preset labeling database.

[0039] The third aspect of the embodiment of the application provides an electronic device, which comprises a memory, a processor and a transceiver.

[0040] The processor is configured to be coupled with the memory, read and execute instructions in the memory, so as to realize the method steps of the first aspect described above.

[0041] The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transceiving.

[0042] The fourth aspect of the embodiment of the application provides a computer readable storage medium, which stores computer instructions, and when the computer instructions are executed by a computer, the computer instructions make the computer execute the instructions of the method of the first aspect described above.

[0043] The embodiment of the present application provides a multi-round medical image labeling method and device introducing a cross-validation mechanism, electronic equipment and a computer readable storage medium. From the above content, it can be known that when the medical image to be labeled is labeled, the embodiment of the present application: first, a plurality of expert labeling interfaces are randomly selected from an expert library to independently label the current image in one round; then, the multi-person labeling results of this round are cross-validated in a pairwise comparison manner to obtain a corresponding cross-validation matrix A; then, the cross-validation matrix A is used for controversial labeling box extraction, non-controversial labeling box aggregation and consistency score estimation processing; if the consistency score does not meet the standard, a new round of multi-person labeling is initiated again until the latest consistency score meets the standard; if the consistency score meets the standard, it is identified whether the latest controversial labeling box set is empty, if not, the latest non-controversial labeling box set is directly used as the final labeling result, if yes, the controversial labeling box set is reviewed for one round, and the final labeling result is composed of the review feedback and the non-controversial labeling box set; and the corresponding labeling record composed of the original medical image and the final labeling result is added to the labeling database. The embodiment of the present application reduces the labeling error of each image, improves the overall labeling quality of all images through the multi-person labeling mechanism, the cross-validation mechanism, the multi-person labeling restart mechanism based on the consistency score and the final review mechanism, and simultaneously improves the guarantee ability of the overall labeling quality stability while improving the overall labeling quality. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1 A multi-round medical image labeling method introducing a cross-validation mechanism is provided for the embodiment one of the present application.

[0045] Figure 2 A matrix diagram of the cross-validation matrix A is provided for the embodiment one of the present application.

[0046] Figure 3 An acquisition example diagram of the first identification set is provided for the embodiment one of the present application.

[0047] Figure 4 A module structure diagram of a multi-round medical image labeling device introducing a cross-validation mechanism is provided for the embodiment two of the present application.

[0048] Figure 5 A structure diagram of an electronic equipment is provided for the embodiment three of the present application. DETAILED DESCRIPTION

[0049] In order to make the objects, technical solutions and advantages of the present application clearer, the following further describes the present application with reference to the accompanying drawings. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of the present application.

[0050] The embodiment one of the present application provides a multi-round medical image labeling method introducing a cross-validation mechanism. Figure 1 A multi-round medical image labeling method introducing a cross-validation mechanism provided by the embodiment one of the present application is shown in a schematic diagram, and the method mainly includes the following steps.

[0051] Step 1, taking the original medical image to be labeled as a first image.

[0052] Here, the original medical image of the embodiment of the present application can be any type of medical image, such as X-ray image, plain CT image, enhanced CT image, conventional MRI image, functional MRI image, conventional ultrasound image, color Doppler ultrasound image, PET image, SPECT image, etc.

[0053] Step 2, randomly selecting X expert labeling interfaces from a preset expert library to form a current round interface set; sending the first image to all expert labeling interfaces of the current round interface set; and merging X expert labeling sets fed back by all expert labeling interfaces to obtain a corresponding first labeling set.

[0054] Here, the number X of the embodiment of the present application is a preset positive integer.

[0055] The expert library of the embodiment of the present application is a pre-set medical image expert interface library, including a plurality of expert labeling interfaces, each expert labeling interface corresponding to a medical image expert.

[0056] Each expert labeling interface of the embodiment of the present application is used to send the first image to the corresponding medical image expert. After receiving the first image, the medical image expert first performs medical image labeling on the first image received this time through a remote labeling device or a cloud labeling platform to generate a corresponding expert labeling set, and then feeds back the expert labeling set through the corresponding expert labeling interface.

[0057] Each expert labeling set of the embodiment of the present application is composed of a plurality of first labeling boxes. The first labeling box includes a first center point coordinate, a first width, a first height and a first labeling text. Because the first labeling set is merged from X expert labeling sets, the first labeling set is also composed of a plurality of first labeling boxes.

[0058] Step 3, count the total number of the first annotation set to obtain the corresponding total number M; and calculate the center point distance between each two first annotation boxes, the intersection over union of the annotation boxes and the text similarity, and set the corresponding cross-validation items based on the calculation results; and based on all the cross-validation items obtained, an MxM cross-validation matrix A is constructed;

[0059] Specifically, step 31, count the total number of the first annotation set to obtain the corresponding total number M;

[0060] Step 32, and calculate the center point distance between each two first annotation boxes in the first annotation set, the intersection over union of the annotation boxes and the text similarity, and set the corresponding cross-validation items based on the calculation results;

[0061] Wherein, the cross-validation item is a binary data with a value of 0 or 1;

[0062] Specifically, step 321, the M first annotation boxes of the first annotation set are combined two by two to obtain N corresponding annotation box combinations;

[0063] Here, N=M(M-1) / 2;

[0064] Step 322, each annotation box combination is taken as the current combination; and the straight line distance between the two first center point coordinates of the current combination is calculated to obtain the corresponding first distance; and based on the two sets of first center point coordinates, the first width and the first height of the current combination, two annotation box regions are constructed in the first image, which are recorded as the first and second regions, and the region intersection over union of the first and second regions is calculated to obtain the corresponding first intersection over union; and the two first annotation texts of the current combination are respectively processed by the preset medical text encoder to obtain the corresponding first and second feature vectors, and the vector similarity of the first and second feature vectors is calculated based on the preset vector similarity algorithm to obtain the corresponding first similarity; and a corresponding combination comparison result is composed of a group of first distance, first intersection over union and first similarity corresponding to the current combination;

[0065] Here, the medical text encoder of the embodiment of the application is a conventional word embedding encoding model, or a large language model that has been pre-trained based on a natural language corpus and fine-tuned in the medical field based on a medical field corpus; the word embedding encoding model of the embodiment of the application includes a bag-of-words model, a TF-IDF model, and an N-gram model; the large language model of the embodiment of the application includes a BERT series model and a GPT series model; and the vector similarity algorithm of the embodiment of the application is the vector cosine similarity algorithm by default;

[0066] Step 323, taking each combination comparison result as a current comparison result; identifying a first interval, a first intersection-over-union ratio and a first similarity of the current comparison result; if the first interval is less than or equal to a preset first interval threshold, the first intersection-over-union ratio is greater than or equal to a preset first intersection-over-union ratio threshold, and the first similarity is greater than or equal to a preset first similarity threshold, setting a corresponding cross-validation item as 1; if the first interval is greater than the first interval threshold, the first intersection-over-union ratio is less than the first intersection-over-union ratio threshold, or the first similarity is less than the first similarity threshold, setting the corresponding cross-validation item as 0;

[0067] Here, the first interval threshold, the first intersection-over-union ratio threshold and the first similarity threshold are three preset threshold parameters;

[0068] Step 33, constructing a cross-validation matrix A with a shape of MxM based on all the obtained cross-validation items;

[0069] Wherein, each row or each column of the cross-validation matrix A corresponds to a first annotation box; the cross-validation matrix A is composed of MxM matrix units a i,j , 1≤row index i≤M, 1≤column index j≤M; each unit value of each matrix unit a i,j is 0 or 1; the unit values of M diagonal line matrix units a i＝1,j＝1 from the top-left vertex matrix unit a i＝M,j＝M to the bottom-right vertex matrix unit a 1≤i≤M,j＝i of the cross-validation matrix A on the matrix diagonal line are all 1; Figure 2 The matrix schematic diagram of the cross-validation matrix A provided by the embodiment one of the present application can be used for intuitive understanding of the matrix structure of the cross-validation matrix A; Figure 2

[0070] Specifically including: setting a zero matrix with a shape of MxM as an initialized cross-validation matrix A; resetting the unit values of M matrix units a i,j on the matrix diagonal line of the cross-validation matrix A as 1; taking each cross-validation item with a value of 1 as a corresponding first validation item; and resetting the unit values of two matrix units a i,j corresponding to each first validation item in the cross-validation matrix A as 1.

[0071] Step 4, performing controversial annotation box extraction and non-controversial annotation box aggregation processing based on the cross-validation matrix A, and performing consistency score estimation based on the processing result to obtain a corresponding controversial annotation box set, a non-controversial annotation box set and a first score;

[0072] ​When the set of disputed annotation boxes is not empty, it consists of one or more second annotation boxes; the second annotation box includes the coordinates of the second center point, the second width, the second height, and the second annotation text; when the set of undisputed annotation boxes is not empty, it consists of one or more third annotation boxes; the third annotation box includes the coordinates of the third center point, the third width, the third height, and the third annotation text.

[0073] Specifically, this includes: Step 41, performing a row-by-row traversal of the cross-validation matrix A; and during this traversal, taking the current traversed row as the current row; and setting the matrix cell a with a value of 1 in the current row as the current row. i,j The column index j is used as a corresponding first identifier; and all the first identifiers corresponding to the current row form a corresponding first identifier set; and at the end of this round of traversal, the M first identifier sets are deduplicated.

[0074] Here, with Figure 3 Taking the example of obtaining the first identifier set provided in Embodiment 1 of the present invention, it is known that the first annotation set has M = 5 first annotation boxes, denoted as boxes 1, 2, 3, 4, and 5, and the corresponding N = M(M-1) / 2 = 10 cross-validation items are as follows: cross-validation item (1,2) = 1, cross-validation item (1,3) = 0, cross-validation item (1,4) = 0, cross-validation item (1,5) = 0, cross-validation item (2,3) = 0, cross-validation item (2,4) = 0, cross-validation item (2,5) = 0, cross-validation item (3,4) = 0, cross-validation item (3,5) = 0, cross-validation item (4,5) = 1; then: the distribution of the values ​​of the 5×5 matrix units of the generated 5×5 cross-validation matrix A is as follows. Figure 3 As shown; for Figure 3 After traversing the cross-validation matrix A row by row, we get 5 first identifier sets: first identifier set {1,2}, first identifier set {1,2}, first identifier set {3}, first identifier set {4,5}, and first identifier set {4,5}. After removing duplicates from these 5 first identifier sets, we get 3 first identifier sets: first identifier set {1,2}, first identifier set {3}, and first identifier set {4,5}.

[0075] Step 42: Perform a traversal of all the deduplicated first identifier sets; during this traversal, the currently traversed first identifier set is taken as the current identifier set; and identify whether the total number of identifiers in the current identifier set is 1; if yes, then take the first annotation box corresponding to the unique first identifier in the current identifier set as a corresponding second annotation box; if no, then perform annotation box aggregation processing on all the first annotation boxes corresponding to the current identifier set to obtain a corresponding third annotation box; and after this traversal, the obtained second annotation boxes form the corresponding disputed annotation box set, and the obtained third annotation boxes form the corresponding undisputed annotation box set.

[0076] wherein, all the first annotation boxes corresponding to the current identification set are subjected to annotation box aggregation processing to obtain a corresponding third annotation box, specifically including: all the first center point coordinates corresponding to the current identification set are subjected to mean value calculation to obtain a corresponding average coordinate, all the first widths are subjected to mean value calculation to obtain a corresponding average width, and all the first heights are subjected to mean value calculation to obtain a corresponding average height; and the obtained average coordinate, average width and average height are taken as a corresponding third center point coordinate, third width and third height; and one of all the first annotation texts corresponding to the current identification set is selected as a corresponding third annotation text; and a corresponding third annotation box is composed of the obtained third center point coordinate, third width, third height and third annotation text;

[0077] For example, after deduplication, there are 3 first identification sets left to be traversed; when the current identification set is the first identification set {1, 2}, the aggregation of the annotation boxes in rows 1 and 2 will obtain a corresponding third annotation box (denoted as box 6); when the current identification set is the first identification set {3}, box 3 is directly taken as a corresponding second annotation box (denoted as box 7); when the current identification set is the first identification set {4, 5}, the aggregation of the annotation boxes in rows 4 and 5 will obtain a corresponding third annotation box (denoted as box 8); then, the finally obtained controversial annotation box set is {box 7}, and the non-controversial annotation box set is {box 6, box 8};

[0078] Step 43, and the total number of annotation boxes in the obtained controversial annotation box set and non-controversial annotation box set is counted to obtain a corresponding first total number and a second total number; and the first score is estimated based on the first total number and the second total number;

[0079] Here, the first score = the second total number / (the first total number + the second total number).

[0080] Here, the value range of the first score is between 0 and 1, and the higher the first score value is, the higher the consistency of this round of multi-person annotation is, and vice versa.

[0081] Step 5, whether the first score is lower than a preset consistency score threshold is identified; if yes, a new round of expert annotation is initiated by returning to step 2; if no, step 6 is turned to.

[0082] Here, the consistency score threshold is a pre-set threshold parameter, and the value range is between 0 and 1, and the default is a value greater than or equal to 0.5. If the first score is lower than the consistency score threshold, it means that the consistency evaluation result of the current multi-person labeling result is not up to standard, and at this time, it is necessary to return to step 2 to randomly select X experts to restart a round of multi-person labeling; if the first score is greater than or equal to the consistency score threshold, it means that the consistency evaluation result of the current multi-person labeling result is up to standard, and at this time, it can be transferred to the subsequent step 6.

[0083] Step 6, identifying whether the set of controversial annotation boxes is empty; if yes, the set of non-controversial annotation boxes is taken as the corresponding first label annotation set; if no, the first image and the set of controversial annotation boxes are sent to the pre-set annotation review interface, and the review annotation box set fed back by the annotation review interface and the set of non-controversial annotation boxes are merged to obtain the corresponding first label annotation set.

[0084] Here, the review annotation box set of the embodiment of the application is composed of one or more fourth annotation boxes; the fourth annotation box includes a fourth center point coordinate, a fourth width, a fourth height and a fourth annotation text.

[0085] Here, the annotation review interface of the embodiment of the application corresponds to one pre-set medical image expert or administrator; if it corresponds to one medical image expert, the expert should be a professional expert in the field who has deep annotation experience in the image type and disease type corresponding to the current medical image, and if it corresponds to one administrator, the administrator can call multiple senior experts to review and diagnose the current image through offline or online methods.

[0086] The marking review interface of the embodiment of the present application is used to send the first image and the set of controversial marking boxes to the corresponding medical image specialist or administrator; if it is sent to the medical image specialist, the medical image specialist reviews the set of controversial marking boxes received this time through the remote marking device or the cloud marking platform after receiving the first image and the set of controversial marking boxes, and makes marking revision when finding marking errors, and makes marking addition when finding missing markers, and after the review is completed, each marking box is taken as a corresponding fourth marking box, and the set of review marking boxes is composed of all the obtained fourth marking boxes, and the set of review marking boxes is fed back through the marking review interface; if it is sent to the administrator, the administrator convenes multiple senior experts to review and diagnose the set of controversial marking boxes of the current image through offline or online methods after receiving the first image and the set of controversial marking boxes, and reviews the set of controversial marking boxes through the remote marking device or the cloud marking platform during the review and diagnosis process, and makes marking revision when finding marking errors, and makes marking addition when finding missing markers, and after the review and diagnosis this time is completed, each marking box is taken as a corresponding fourth marking box, and the set of review marking boxes is composed of all the obtained fourth marking boxes, and the set of review marking boxes is fed back through the marking review interface.

[0087] Step 7, a corresponding first marking record composed of the first image and the first label marking set is added to the preset marking database.

[0088] Here, the marking database of the embodiment of the present application is a pre-set database, which is used to store a plurality of first marking records, each first marking record including a first image and a first label marking set. Each first marking record can be used for clinical diagnosis or artificial intelligence model training.

[0089] Figure 4 A module structure diagram of a multi-round medical image marking device with an introduced cross-validation mechanism is provided for the second embodiment of the present application. The device is a terminal device or a server for implementing the foregoing method embodiment, or a device capable of enabling the foregoing terminal device or server to implement the foregoing method embodiment, such as a device or a chip system of the foregoing terminal device or server. As shown in the figure, the device includes an image receiving module 201, a multi-person marking module 202, a cross-validation preprocessing module 203, a cross-validation post-processing module 204, a consistency compliance evaluation module 205, a marking review module 206, and a marking record adding module 207. Figure 4

[0090] The image receiving module 201 is used to take the original medical image to be marked as a first image.

[0091] ​The multi-person labeling module 202 is configured to randomly select X expert labeling interfaces from a preset expert library to form a current interface set, send the first image to all expert labeling interfaces in the current interface set, merge X expert labeling sets fed back by all expert labeling interfaces to obtain a corresponding first labeling set, X is a preset positive integer, each expert labeling set is composed of one or more first labeling boxes, and the first labeling box includes a first center point coordinate, a first width, a first height, and a first labeling text.

[0092] The cross-validation preprocessing module 203 is configured to count the total number of labeling boxes in the first labeling set to obtain a corresponding total number M, calculate the center point distance, the intersection-over-union ratio, and the text similarity of each two labeling boxes in the first labeling set, and set a corresponding cross-validation item based on the calculation result, and construct an MxM cross-validation matrix A based on all obtained cross-validation items.

[0093] The cross-validation post-processing module 204 is configured to perform controversial labeling box extraction and non-controversial labeling box aggregation based on the cross-validation matrix A, and estimate the consistency score based on the processing result to obtain a corresponding controversial labeling box set, a non-controversial labeling box set, and a first score.

[0094] The consistency evaluation module 205 is configured to identify whether the first score is lower than a preset consistency score threshold, if yes, return to the multi-person labeling module 202 to initiate a new round of expert labeling, and if no, go to the labeling review module 206.

[0095] The labeling review module 206 is configured to identify whether the controversial labeling box set is empty, if yes, take the non-controversial labeling box set as a corresponding first label labeling set, and if no, send the first image and the controversial labeling box set to a preset labeling review interface, and merge a review labeling box set fed back by the labeling review interface with the non-controversial labeling box set to obtain a corresponding first label labeling set.

[0096] The labeling record adding module 207 is configured to add a corresponding first labeling record composed of the first image and the first label labeling set to a preset labeling database.

[0097] The multi-round medical image labeling device provided by the embodiment of the application can execute the method steps in the method embodiment, and has similar implementation principles and technical effects, which will not be described here.

[0098] It should be noted that the division of the various modules of the above apparatus is only a logical functional division, and in actual implementation, all or part of them can be integrated into one physical entity, or can be physically separated. These modules can all be implemented in the form of software invoked by a processing element; all can be implemented in the form of hardware; or some modules can be implemented in the form of software invoked by a processing element, and some modules can be implemented in the form of hardware. For example, the image receiving module can be a separately established processing element, or can be integrated in a certain chip of the above apparatus, in addition, it can also be stored in the form of program code in the memory of the above apparatus, and the function of the above determining module is invoked and executed by a certain processing element of the above apparatus. The implementation of other modules is similar. In addition, all or part of these modules can be integrated together, or can be independently implemented. The processing element described herein can be an integrated circuit with signal processing capability. In the implementation process, each step of the above method or each module can be completed by the integrated logic circuit of the hardware in the processor element or the instructions in the form of software.

[0099] For example, the above modules can be one or more integrated circuits configured to implement the above method, such as one or more application specific integrated circuits (ASICs), or one or more digital signal processors (DSPs), or one or more field programmable gate arrays (FPGAs), etc. For another example, when a certain module above is implemented in the form of program code invoked by a processing element, the processing element can be a general purpose processor, such as a central processing unit (CPU) or other processor that can invoke program code. For another example, these modules can be integrated together to implement in the form of system on a chip (SOC).

[0100] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the foregoing method embodiments are generated. The computer described above can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The aforementioned computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the aforementioned computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, Digital Subscriber Line (DSL)) or wireless (e.g., infrared, wireless, Bluetooth, microwave, etc.) means. The aforementioned computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that integrates one or more available media. The aforementioned available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks (SSDs)).

[0101] Figure 5 This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. This electronic device can be a terminal device or server implementing the methods of the aforementioned embodiments, or it can be a terminal device or server connected to the aforementioned terminal device or server implementing the methods of the aforementioned embodiments. Figure 5 As shown, the electronic device may include: a processor 301 (e.g., CPU), a memory 302, and a transceiver 303; the transceiver 303 is coupled to the processor 301, and the processor 301 controls the transmission and reception operations of the transceiver 303. The memory 302 may store various instructions for performing various processing functions and implementing the processing steps described in the foregoing embodiments. Preferably, the electronic device involved in the embodiments of the present invention further includes: a power supply 304, a system bus 305, and a communication port 306. The system bus 305 is used to realize communication connections between components. The communication port 306 is used for communication between the electronic device and other peripherals.

[0102] exist Figure 5The system bus 305 mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or only one type of bus. The communication interface is used to realize the communication between the database access device and other devices (such as a client, a read-write library and a read-only library). The memory can include a Random Access Memory (RAM), and can also include a Non-Volatile Memory, such as at least one disk memory.

[0103] The processor mentioned above can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), a Graphics Processing Unit (GPU), etc.; can also be a Digital Signal Processor (DSP), an Application-Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0104] It should be noted that the embodiments of the present application also provide a computer readable storage medium, which stores instructions, and when the instructions run on a computer, the computer executes the method and processing procedure provided in the above embodiments.

[0105] The embodiments of the present application also provide a chip for running instructions, which is used to execute the processing steps described in the foregoing method embodiments.

[0106] The embodiment of the present application provides a multi-round medical image labeling method and device introducing a cross-validation mechanism, electronic equipment and a computer readable storage medium. From the above content, it can be known that when the medical image to be labeled is labeled, the embodiment of the present application: first, a plurality of expert labeling interfaces are randomly selected from an expert library to independently label the current image in one round; then, the cross-validation matrix A corresponding to the multi-person labeling result of the current round is obtained by comparing in pairs; then, the controversial labeling box extraction, the non-controversial labeling box aggregation and the consistency score estimation processing are performed based on the cross-validation matrix A; if the consistency score does not meet the standard, a new round of multi-person labeling is reinitiated until the latest consistency score meets the standard; if the consistency score meets the standard, whether the latest controversial labeling box set is empty is identified, if the latest controversial labeling box set is empty, the latest non-controversial labeling box set is directly taken as the final labeling result, if the latest controversial labeling box set is not empty, the controversial labeling box set is reviewed in one round, and the final labeling result is composed of the review feedback and the non-controversial labeling box set; and the corresponding labeling record composed of the original medical image and the final labeling result is added to the labeling database. The embodiment of the present application effectively reduces the labeling error of each image, improves the overall labeling quality of all images, and improves the guarantee ability of the overall labeling quality stability while improving the overall labeling quality.

[0107] Those skilled in the art should further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example have been described in the above description in general terms. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0108] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be implemented by hardware, a software module executed by a processor, or a combination of both. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0109] The above detailed description of the specific embodiments of the present application has been given to understand the purpose, technical solutions and beneficial effects of the present application. It should be understood that the above description is only a specific embodiment of the present application and is not used to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multi-round medical image annotation method incorporating a cross-validation mechanism, characterized in that, The method comprises: Step 1, the original medical image to be labeled as a first image; Step 2, randomly select X expert labeling interfaces from a preset expert library to form a round interface set; and send the first image to all the expert labeling interfaces of the round interface set; and merge the X expert labeling sets fed back by all the expert labeling interfaces to obtain a corresponding first labeling set; X is a preset positive integer; each expert labeling set is composed of one or more first labeling boxes; the first labeling box includes a first center point coordinate, a first width, a first height and a first labeling text; Step 3, count the total number of labeling boxes in the first labeling set to obtain a corresponding total number M; calculate the center point distance, the intersection-over-union ratio and the text similarity of each two first labeling boxes in the first labeling set, and set a corresponding cross-validation item based on the calculation result; and based on all the cross-validation items obtained, a cross-validation matrix A with a shape of MxM is constructed; Step 4, based on the cross-validation matrix A, perform controversial labeling box extraction and non-controversial labeling box aggregation processing, and estimate the consistency score based on the processing result to obtain a corresponding controversial labeling box set, a non-controversial labeling box set and a first score; Step 5, identify whether the first score is lower than a preset consistency score threshold; if yes, return to step 2 to initiate a new round of expert labeling; if no, go to step 6; Step 6, identify whether the controversial labeling box set is empty; if yes, take the non-controversial labeling box set as a corresponding first label labeling set; if no, send the first image and the controversial labeling box set to a preset labeling review interface, and merge the review labeling box set fed back by the labeling review interface with the non-controversial labeling box set to obtain a corresponding first label labeling set; Step 7, a corresponding first labeling record composed of the first image and the first label labeling set is added to a preset labeling database.

2. The multi-round medical image labeling method with cross-validation mechanism according to claim 1, wherein: The cross-validation item is a binary data with a value of 0 or 1; Each row or column of the cross-validation matrix A corresponds to one of the first annotation boxes; the cross-validation matrix A is composed of MxM matrix cells a i,j , 1≤row index i≤M, 1≤column index j≤M; each cell value of the matrix cells a i,j is 0 or 1; the cell values of the M diagonal line matrix cells a i＝1,j＝1 on the matrix diagonal line from the top-left vertex matrix cell a i＝M,j＝M to the bottom-right vertex matrix cell a 1≤i≤M,j＝i of the cross-validation matrix A are all 1; The controversial labeling box set is composed of one or more second labeling boxes when it is not empty; the second labeling box includes a second center point coordinate, a second width, a second height and a second labeling text; The non-controversial labeling box set is composed of one or more third labeling boxes when it is not empty; the third labeling box includes a third center point coordinate, a third width, a third height and a third labeling text; The review labeling box set is composed of one or more fourth labeling boxes; the fourth labeling box includes a fourth center point coordinate, a fourth width, a fourth height and a fourth labeling text.

3. The multi-round medical image annotation method incorporating cross-validation mechanism according to claim 2, wherein, The calculation of the center point distance, the intersection-over-union ratio and the text similarity of each two first labeling boxes in the first labeling set and the setting of the corresponding cross-validation item based on the calculation result specifically comprise: M first annotation boxes in the first annotation set are combined two by two to obtain corresponding N annotation box combinations; N=M(M-1) / 2; each annotation box combination is taken as a current combination, the distance between the straight lines of the two first center point coordinates of the current combination is calculated to obtain a corresponding first distance, two annotation box regions corresponding to the current combination are constructed based on the two sets of first center point coordinates, the first width and the first height of the current combination, and are recorded as first and second regions, the area intersection ratio of the first and second regions is calculated to obtain a corresponding first area intersection ratio, the two first annotation texts of the current combination are respectively processed by a preset medical text encoder to obtain corresponding first and second feature vectors, and the vector similarity of the first and second feature vectors is calculated based on a preset vector similarity algorithm to obtain a corresponding first similarity, and a corresponding combination comparison result is composed of a set of first distances, the first area intersection ratio and the first similarity corresponding to the current combination; wherein the medical text encoder is a conventional word embedding encoding model or a large language model that has been pre-trained based on a natural language corpus and fine-tuned based on a medical field corpus; the word embedding encoding model includes a bag-of-words model, a TF-IDF model, and an N-gram model; the large language model includes a BERT series model and a GPT series model; each combination comparison result is taken as a current comparison result, and the first distance, the first area intersection ratio and the first similarity of the current comparison result are identified; if the first distance is less than or equal to a preset first distance threshold, the first area intersection ratio is greater than or equal to a preset first area intersection ratio threshold, and the first similarity is greater than or equal to a preset first similarity threshold, the corresponding cross-validation item is set to 1; if the first distance is greater than the first distance threshold or the first area intersection ratio is less than the first area intersection ratio threshold or the first similarity is less than the first similarity threshold, the corresponding cross-validation item is set to 0.

4. The method of claim 2, wherein the introducing a cross-validation mechanism in a multi-round medical image annotation method is characterized by, Based on all the obtained cross-validation items, a cross-validation matrix A with a shape of MxM is constructed, specifically including: setting a full zero matrix of shape M x M as the initialized cross-validation matrix A; and resetting the cell values of M matrix cells a i,j on the matrix diagonal of the cross-validation matrix A to 1; and taking each cross-validation item with a value of 1 as a corresponding first validation item; and resetting the cell values of two matrix cells a i,j corresponding to each first validation item in the cross-validation matrix A to 1.

5. The method of claim 2, wherein the introducing a cross-validation mechanism in a multi-round medical image annotation method is characterized by, Based on the cross-validation matrix A, controversial annotation box extraction and non-controversial annotation box aggregation processing are performed, and consistency scoring is estimated based on the processing result to obtain a corresponding set of controversial annotation boxes, a set of non-controversial annotation boxes and a first score, specifically including: performing one round of row-by-row traversal on the cross-validation matrix A; and during the current round of traversal, taking the current traversal row as a current row; and taking the column index j of each matrix element a i,j with a value of 1 in the current row as a corresponding first identifier; and taking all the first identifiers corresponding to the current row as a corresponding first identifier set; and at the end of the current round of traversal, performing deduplication processing on the M first identifier sets obtained; performing a round of traversal on all the first identification sets after deduplication; during the current round of traversal, taking the first identification set being currently traversed as a current identification set; identifying whether the total number of identifications in the current identification set is 1; if yes, taking the first annotation box corresponding to the only first identification in the current identification set as a corresponding second annotation box; if not, performing annotation box aggregation processing on all the first annotation boxes corresponding to the current identification set to obtain a corresponding third annotation box; after the current round of traversal ends, taking all the obtained second annotation boxes to form a corresponding set of controversial annotation boxes, and taking all the obtained third annotation boxes to form a corresponding set of non-controversial annotation boxes; statistically obtaining first and second total numbers of annotation boxes in the set of controversial annotation boxes and the set of non-controversial annotation boxes respectively, and estimating a corresponding first score based on the first and second total numbers, the first score = the second total number / (the first total number + the second total number).

6. The multi-round medical image annotation method incorporating cross-validation mechanism according to claim 5, wherein, The annotation box aggregation processing on all the first annotation boxes corresponding to the current identification set to obtain a corresponding third annotation box specifically includes: performing mean value calculation on all the first center point coordinates corresponding to the current identification set to obtain corresponding average coordinates, performing mean value calculation on all the first widths to obtain corresponding average widths, and performing mean value calculation on all the first heights to obtain corresponding average heights; taking the obtained average coordinates, average widths, and average heights as a corresponding set of third center point coordinates, third widths, and third heights; randomly selecting one of all the first annotation texts corresponding to the current identification set as a corresponding third annotation text; and taking the obtained third center point coordinates, third widths, third heights, and third annotation text to form a corresponding third annotation box.

7. An apparatus for performing the multi-round medical image annotation method incorporating the cross-validation mechanism of any one of claims 1-6, characterized in that, The device comprises an image receiving module, a multi-person annotation module, a cross-validation preprocessing module, a cross-validation post-processing module, a consistency compliance evaluation module, an annotation review module, and an annotation record adding module. The image receiving module is configured to take an original medical image to be annotated as a first image. The multi-person annotation module is configured to randomly select X expert annotation interfaces from a preset expert library to form a current interface set, send the first image to all the expert annotation interfaces in the current interface set, and merge X expert annotation sets fed back by all the expert annotation interfaces to obtain a corresponding first annotation set; X is a preset positive integer; each expert annotation set comprises one or more first annotation boxes; the first annotation box comprises first center point coordinates, a first width, a first height, and a first annotation text. The cross-validation preprocessing module is configured to count the total number of the first annotation set to obtain a corresponding total number M; calculate the center point distance between each two first annotation boxes, the intersection over union of the annotation boxes, and the text similarity in the first annotation set, and set a corresponding cross-validation item based on the calculation results; and construct a cross-validation matrix A with a shape of MxM based on all the cross-validation items obtained. The cross-validation post-processing module is configured to perform controversial annotation box extraction and non-controversial annotation box aggregation processing based on the cross-validation matrix A, and perform consistency scoring estimation based on the processing results to obtain a corresponding controversial annotation box set, a non-controversial annotation box set, and a first score. The consistency compliance evaluation module is configured to identify whether the first score is lower than a preset consistency score threshold; if yes, return the multi-person annotation module to initiate a new round of expert annotation; if no, go to the annotation review module. The annotation review module is configured to identify whether the controversial annotation box set is empty; if yes, take the non-controversial annotation box set as a corresponding first label annotation set; if no, send the first image and the controversial annotation box set to a preset annotation review interface, and merge the review annotation box set fed back by the annotation review interface and the non-controversial annotation box set to obtain the corresponding first label annotation set. The annotation record adding module is configured to add a corresponding first annotation record composed of the first image and the first label annotation set to a preset annotation database.

8. An electronic device, comprising: Comprise: a memory, a processor and a transceiver; The processor is configured to be coupled with the memory, read and execute instructions in the memory to implement the method of any one of claims 1-6; The transceiver is coupled with the processor, and the transceiver is controlled by the processor to perform message transmission and reception.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions, when the computer instructions are executed by the computer, the computer executes the method of any one of claims 1-6. The computer readable storage medium stores computer instructions, when the computer instructions are executed by the computer, the computer executes the method of any one of claims 1-6.

Images