A data processing method and device, computer equipment and a storage medium

Abstract

Embodiments of the present application disclose a data processing method and device, computer equipment and a storage medium, wherein the method comprises the following steps: obtaining a target object report and a structured annotation result corresponding to the target object report; the structured annotation result comprises a plurality of annotation result sets; performing structured processing on the target object report to generate a structured prediction result corresponding to the target object report; the structured prediction result comprises a plurality of prediction result sets; performing comparison detection on the prediction result set of the structured prediction result and the annotation result set of the structured annotation result to generate a comparison detection result of the target object report; generating a detection parameter of the target object report according to the comparison detection result; and the detection parameter is used to measure the prediction effect of the structured prediction result. By using the present application, the detection efficiency can be improved, and the accuracy of the detection result can be ensured.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a data processing method, apparatus, computer equipment, and storage medium. Background Technology

[0002] Because medical reports contain numerous medical fields and have complex structural relationships, existing structured prediction models for medical reports lack a systematic automatic regression testing scheme. Existing solutions mainly compare the predicted results with the labeled text content character by character. This comparison is time-consuming and labor-intensive, and the predicted and labeled results cannot be intelligently matched. The different order of attribute information in the predicted and labeled results leads to different matching results, which cannot guarantee the accuracy of the detection results and greatly affects the efficiency of research and development. Summary of the Invention

[0003] This application provides a data processing method, apparatus, computer equipment, and storage medium, which can improve detection efficiency and ensure the accuracy of detection results.

[0004] One embodiment of this application provides a data processing method, which may include:

[0005] Obtain the target object report and the corresponding structured annotation results; the structured annotation results include multiple annotation result sets;

[0006] The target object report is processed in a structured manner to generate a structured prediction result corresponding to the target object report; the structured prediction result includes multiple prediction result sets;

[0007] The prediction result set of the structured prediction results and the annotation result set of the structured annotation results are compared and detected to generate the comparison and detection results of the target object report;

[0008] The detection parameters for generating the target object report are generated based on the comparative detection results; the detection parameters are used to measure the predictive effect of the structured prediction results.

[0009] In one feasible implementation, the step of performing structured processing on the target object report to generate a structured prediction result corresponding to the target object report includes:

[0010] Multiple nodes are generated based on the structure of the target object report, and a set of prediction results corresponding to each node is generated based on the attribute information corresponding to each node.

[0011] The set of prediction results corresponding to each node is determined as the structured prediction results corresponding to the target object report.

[0012] In one feasible implementation, the step of comparing and detecting the prediction result set of the structured prediction results and the annotation result set of the structured annotation results to generate the comparison and detection results of the target object report includes:

[0013] Obtain the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generate a similarity matrix;

[0014] The matching relationship between the predicted result set and the labeled result set is obtained based on the similarity matrix;

[0015] The predicted result set and the labeled result set corresponding to the matching relationship are compared and detected to generate the comparison and detection results of the target object report.

[0016] In one feasible implementation, obtaining the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generating a similarity matrix, includes:

[0017] Obtain the target prediction result set from the structured prediction results and the target annotation result set from the structured annotation results; the target prediction result set is any one of the prediction result sets in the structured prediction results, and the target annotation result set is any one of the annotation result sets in the structured annotation results;

[0018] The attribute information in the target prediction result set is compared with the attribute information in the target annotation result set to generate a similarity score;

[0019] A similarity matrix for the target object report is generated based on the similarity between each set of predicted results and each set of labeled results.

[0020] In one feasible implementation, obtaining the matching relationship between the prediction result set and the annotation result set based on the similarity matrix includes:

[0021] Obtain the maximum similarity in the similarity matrix and record the index of the maximum similarity in the similarity matrix; each index corresponds to a set pair, and the set pair includes a set of prediction results and a set of annotation results;

[0022] The row and column of the maximum similarity in the similarity matrix are reset to negative values;

[0023] After all similarities in the similarity matrix are reset to negative values, an index sequence is generated based on the indices corresponding to at least two maximum similarities.

[0024] The matching relationship between the predicted result set and the labeled result set is generated based on the set corresponding to the index sequence.

[0025] In one feasible implementation, it further includes:

[0026] If there is a set of unmatched prediction results in the structured prediction results, then it is determined that the structured prediction results reported by the target object contain redundant prediction results.

[0027] If there is a set of unmatched annotation results in the structured annotation results, it is determined that the structured prediction results reported by the target object are missing.

[0028] The step of generating a matching relationship between the predicted result set and the labeled result set based on the set corresponding to the index sequence includes:

[0029] Based on the set of unmatched prediction results, the set of unmatched annotation results, and the set pairs corresponding to the index sequence, a matching relationship between the set of prediction results and the set of annotation results is generated.

[0030] In one feasible implementation, the step of generating the detection parameters for the target object report based on the comparative detection results includes:

[0031] Based on the comparative detection results, the prediction category of each prediction result set in the structured prediction results is statistically analyzed;

[0032] Based on the prediction category of each prediction result set, detection parameters for the target object report are generated; the detection parameters include accuracy, recall, precision, and F1 score.

[0033] One embodiment of this application provides a data processing apparatus, which may include:

[0034] The annotation result acquisition unit is used to acquire the target object report and the structured annotation results corresponding to the target object report; the structured annotation results include multiple annotation result sets;

[0035] The prediction result acquisition unit is used to perform structured processing on the target object report and generate a structured prediction result corresponding to the target object report; the structured prediction result includes multiple prediction result sets;

[0036] The detection result generation unit is used to compare and detect the prediction result set of the structured prediction result and the annotation result set of the structured annotation result, and generate the comparison and detection result of the target object report;

[0037] The detection parameter generation unit is used to generate detection parameters for the target object report based on the comparative detection results; the detection parameters are used to measure the prediction effect of the structured prediction results.

[0038] In one feasible implementation, the prediction result acquisition unit is used for:

[0039] Multiple nodes are generated based on the structure of the target object report, and a set of prediction results corresponding to each node is generated based on the attribute information corresponding to each node.

[0040] The set of prediction results corresponding to each node is determined as the structured prediction results corresponding to the target object report.

[0041] In one feasible implementation, the detection result generation unit is used for:

[0042] Obtain the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generate a similarity matrix;

[0043] The matching relationship between the predicted result set and the labeled result set is obtained based on the similarity matrix;

[0044] The predicted result set and the labeled result set corresponding to the matching relationship are compared and detected to generate the comparison and detection results of the target object report.

[0045] In one feasible implementation, the detection result generation unit is used for:

[0046] Obtain the target prediction result set from the structured prediction results and the target annotation result set from the structured annotation results; the target prediction result set is any one of the prediction result sets in the structured prediction results, and the target annotation result set is any one of the annotation result sets in the structured annotation results;

[0047] The attribute information in the target prediction result set is compared with the attribute information in the target annotation result set to generate a similarity score;

[0048] A similarity matrix for the target object report is generated based on the similarity between each set of predicted results and each set of labeled results.

[0049] In one feasible implementation, the detection result generation unit is used for:

[0050] Obtain the maximum similarity in the similarity matrix and record the index of the maximum similarity in the similarity matrix; each index corresponds to a set pair, and the set pair includes a set of prediction results and a set of annotation results;

[0051] The row and column of the maximum similarity in the similarity matrix are reset to negative values;

[0052] After all similarities in the similarity matrix are reset to negative values, an index sequence is generated based on the indices corresponding to at least two maximum similarities.

[0053] The matching relationship between the predicted result set and the labeled result set is generated based on the set corresponding to the index sequence.

[0054] In one feasible implementation, it further includes:

[0055] The set matching detection unit is used to determine that there are redundant prediction results in the structured prediction results reported by the target object if there is an unmatched set of prediction results in the structured prediction results.

[0056] If there is a set of unmatched annotation results in the structured annotation results, it is determined that the structured prediction results reported by the target object are missing.

[0057] The detection result generation unit is used for:

[0058] Based on the set of unmatched prediction results, the set of unmatched annotation results, and the set pairs corresponding to the index sequence, a matching relationship between the set of prediction results and the set of annotation results is generated.

[0059] In one feasible implementation, the detection parameter generation unit is used for:

[0060] Based on the comparative detection results, the prediction category of each prediction result set in the structured prediction results is statistically analyzed;

[0061] Based on the prediction category of each prediction result set, detection parameters for the target object report are generated; the detection parameters include accuracy, recall, precision, and F1 score.

[0062] One embodiment of this application provides a computer-readable storage medium storing a computer program adapted to be loaded by a processor and executed by the above-described method steps.

[0063] One embodiment of this application provides a computer device, including: a processor, a memory, and a network interface; the processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the above-described method steps.

[0064] One embodiment of this application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method steps described above.

[0065] In this embodiment, by acquiring a target object report and its corresponding structured annotation results (which include multiple annotation result sets), the target object report is further processed in a structured manner to generate a structured prediction result, which also includes multiple prediction result sets. Furthermore, the prediction result sets of the structured prediction results and the annotation result sets of the structured annotation results are compared and detected to generate a comparison and detection result for the target object report. Finally, detection parameters for the target object report are generated based on the comparison and detection results, and these parameters are used to measure the prediction effect of the structured prediction results. Using this method, a structured prediction result including multiple prediction result sets can be generated. By comparing each prediction result set with multiple annotation result sets in the structured annotation results, the problem of different matching results caused by different attribute information order in the prediction and annotation results when comparing characters one by one is avoided, thus improving detection efficiency and ensuring the accuracy of the detection results. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0067] Figure 1 This is a network architecture diagram for data processing provided in an embodiment of this application;

[0068] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0069] Figure 3 This is a schematic diagram illustrating an example of a target object report structure provided in an embodiment of this application;

[0070] Figure 4 This is an example diagram illustrating a set of prediction results provided in an embodiment of this application;

[0071] Figure 5This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0072] Figure 6 This is an example diagram illustrating the generation of similarity scores provided in an embodiment of this application;

[0073] Figure 7a This is an example diagram illustrating the generation of a similarity matrix provided in an embodiment of this application;

[0074] Figure 7b This is an example diagram illustrating an update of the similarity matrix provided in an embodiment of this application;

[0075] Figure 7c This is an example diagram illustrating an update of the similarity matrix provided in an embodiment of this application;

[0076] Figure 7d This is an example diagram illustrating an update of the similarity matrix provided in an embodiment of this application;

[0077] Figure 8 This is an example diagram illustrating the generation of comparative detection results provided in an embodiment of this application;

[0078] Figure 9 This is an example diagram illustrating the generation of comparative detection results provided in an embodiment of this application;

[0079] Figure 10 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0080] Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0081] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0082] Please see Figure 1 , Figure 1This is a network architecture diagram for data processing provided in an embodiment of this application. The network architecture diagram may include a service server 100 and a user terminal cluster. The user terminal cluster may include user terminal 10a, user terminal 10b, ..., user terminal 10c. Communication connections may exist between user terminals in the cluster; for example, there is a communication connection between user terminal 10a and user terminal 10b, and a communication connection between user terminal 10b and user terminal 10c. Furthermore, any user terminal in the user terminal cluster may have a communication connection with the service server 100; for example, there is a communication connection between user terminal 10a and service server 100, and a communication connection between user terminal 10b and service server 100.

[0083] The aforementioned user terminal cluster (including user terminal 10a, user terminal 10b, and user terminal 10c) can all integrate and install the target application. Optionally, the target application may include an application with the function of displaying data information such as text, images, and videos. Database 10d stores the target object report and the structured annotation results corresponding to the target object report. In one feasible implementation, the user terminal obtains the target object report and the structured annotation results corresponding to the target object report. The structured annotation results include multiple annotation result sets. Further, the user terminal performs structured processing on the target object report to generate a structured prediction result corresponding to the target object report. The structured prediction result includes multiple prediction result sets. Further, the user terminal compares and detects the prediction result set of the structured prediction result and the annotation result set of the structured annotation result to generate a comparison detection result of the target object report. Finally, the user terminal generates detection parameters of the target object report based on the comparison detection results. The detection parameters are used to measure the prediction effect of the structured prediction result. Using the above method, a structured prediction result comprising multiple prediction result sets can be generated. By comparing each prediction result set with multiple annotation result sets in the structured annotation result, the problem of inconsistent matching results caused by different attribute information order in the prediction and annotation results during character-by-character comparison is avoided. This improves detection efficiency and ensures the accuracy of the detection results. Optionally, the user terminal mentioned above can be used in the above... Figure 1 Any user terminal selected in the user terminal cluster of the corresponding embodiment, for example, the user terminal can be the aforementioned user terminal 10b.

[0084] It is understood that the methods provided in this application embodiment can be executed by computer devices, including but not limited to terminals or servers. The business server 100 in this application embodiment can be a computer device, and the user terminals in the user terminal cluster can also be computer devices; this is not limited here. The aforementioned business server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The aforementioned terminals can include: smartphones, tablets, laptops, desktop computers, smart TVs, smart speakers, desktop computers, smartwatches, and other smart terminals with image recognition functions, but are not limited to these. The user terminals and the business server can be directly or indirectly connected via wired or wireless communication; this application does not impose any restrictions on this connection.

[0085] Furthermore, for ease of understanding, please refer to Figure 2 , Figure 2 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The method can be implemented by a user terminal (e.g., the one described above). Figure 1 The operation can be performed by the user terminal shown, or by the user terminal and the service server (as described above). Figure 1 The service server 100 in the corresponding embodiment performs the operation together. For ease of understanding, this embodiment uses the method executed by the aforementioned user terminal as an example for explanation. The data processing method may include at least the following steps S101-S104:

[0086] S101, Obtain the target object report and the structured annotation results corresponding to the target object report;

[0087] Specifically, the user terminal can obtain the target object report and the structured annotation results corresponding to the target object report. It is understood that in the specific implementation of this application, data such as user object reports are involved. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0088] The target report can be a user's medical report, surgical report, physical examination report, etc. These reports have been individually authorized by the user and can be obtained from the user's personal information. Furthermore, multiple nodes are generated based on the structure of the target report, and the structured annotation results corresponding to the nodes are obtained. Specifically, the structured annotation results can be a JSON structure. The following explanation uses an examination report as an example; please refer to [link to relevant documentation]. Figure 3 , Figure 3 This is an example diagram illustrating a structured target object report provided in an embodiment of this application, such as... Figure 3 The examination report, based on its structure, can be divided into three parts: basic information, visible description, and examination conclusion. The visible description can be further divided into organ visible description and nodule visible description. Each type of visible description consists of several attribute information entries. Each part can correspond to one or more nodes. Specifically, basic information can correspond to one node, and each detected site in the nodule visible description can correspond to one node, meaning the nodule visible description can correspond to multiple nodes. Each node includes several attribute information entries, which are the specific content of each node. Attribute information can include "location," "type," "size," etc. Each node and its corresponding attribute information constitute a set of annotation results. The structured annotation results include multiple sets of annotation results. It should be noted that the structured annotation results can be manually annotated. In automatic regression testing, the structured prediction results of the old version of the system can be used as the structured annotation results of the new version.

[0089] S102, perform structured processing on the target object report to generate a structured prediction result corresponding to the target object report;

[0090] Specifically, the user terminal performs structured processing on the target object report to generate a structured prediction result corresponding to the target object report. This can be understood as the user terminal employing a report structuring engine to generate multiple nodes based on the structure of the target object report. Each node generates a prediction result set based on its corresponding attribute information. Each node includes several attribute information entries, and each node and its corresponding attribute information constitute a labeled result set. The prediction result set corresponding to each node is then determined as the structured prediction result corresponding to the target object report. Specifically, the structured prediction result can be a JSON structure, and it includes multiple prediction result sets. Please refer to [link to relevant documentation]. Figure 4 , Figure 4 This is an example schematic diagram of a prediction result set provided in an embodiment of this application. The diagram includes three prediction result sets, each of which consists of several attribute information. The three prediction result sets can be represented as {"Type: Hypoechoic (Present)", "Location: Left nipple", "Trend of Change: Expansion", "Sound Transmission: Good Sound Transmission", "Multiple: Multiple"}, {"Type: Hypoechoic (Present)", "Location: Left nipple", "Size: 1.7cm*1.1cm", "Edge: Unclear Boundary", "CDFI: Blood Flow Signal Not Obvious"}, and {"Type: Nodule (None)", "Location: Right Breast"}.

[0091] The report structuring engine allows for batch input of multiple target object reports, yielding structured prediction results for each report. These results can then be parsed and stored in a table. Each row represents a single target object report's structured prediction result, improving readability and enabling users to quickly extract information from the results.

[0092] S103, compare and detect the prediction result set of the structured prediction result and the annotation result set of the structured annotation result to generate the comparison and detection result of the target object report;

[0093] Specifically, the structured prediction results include multiple prediction result sets, and the structured annotation results include multiple annotation result sets. The user terminal can obtain the similarity between each prediction result set and each annotation result set, generate a similarity matrix, and further obtain the matching relationship between the prediction result sets and the annotation result sets based on the similarity matrix, that is, determine the correspondence between the prediction result sets and the annotation result sets. The prediction result sets and annotation result sets corresponding to the matching relationship are compared and detected to obtain the differences between the prediction result sets and the annotation result sets. The differences include error information, missing prediction information, and redundant prediction information in the structured prediction results. Based on the differences, a comparison and detection result of the target object report is generated.

[0094] S104, Generate detection parameters for the target object report based on the comparison detection results; the detection parameters are used to measure the prediction effect of the structured prediction results.

[0095] Specifically, the user terminal statistically analyzes the prediction categories of each prediction result set in the structured prediction results based on the comparative detection results. These prediction categories include correct prediction (TP), missing prediction information (FN), and redundant prediction information (FP). Correct prediction means the prediction result set correctly predicts the information; missing prediction information means the prediction result set lacks prediction information; and redundant prediction information means the prediction result set contains redundant information. Based on the prediction category of each prediction result set, detection parameters for the target object report are generated. These detection parameters include accuracy, recall, precision, and F1 score, where accuracy = TP / (TP+FN+FP), recall = TP / (TP+FN), precision = TP / (TP+FP), and F1 score = 2 * recall * precision / (recall + precision). Using these detection parameters can measure the predictive effectiveness of the structured prediction results, facilitating improved development efficiency in automated regression testing.

[0096] In this embodiment, by acquiring a target object report and its corresponding structured annotation results (which include multiple annotation result sets), the target object report is further processed in a structured manner to generate a structured prediction result, which also includes multiple prediction result sets. Furthermore, the prediction result sets of the structured prediction results and the annotation result sets of the structured annotation results are compared and detected to generate a comparison and detection result for the target object report. Finally, detection parameters for the target object report are generated based on the comparison and detection results, and these parameters are used to measure the prediction effect of the structured prediction results. Using this method, a structured prediction result including multiple prediction result sets can be generated. By comparing each prediction result set with multiple annotation result sets in the structured annotation results, the problem of different matching results caused by different attribute information order in the prediction and annotation results when comparing characters one by one is avoided, thus improving detection efficiency and ensuring the accuracy of the detection results.

[0097] Please see Figure 5 , Figure 5 This is a flowchart illustrating the data processing method provided in an embodiment of this application. The method can be implemented by a user terminal (e.g., the one described above). Figure 1 The operation can be performed by the user terminal shown, or by the user terminal and the service server (as described above). Figure 1 The service server 100 in the corresponding embodiment performs the operation together. For ease of understanding, this embodiment uses the method executed by the aforementioned user terminal as an example for explanation. The data processing method may include at least the following steps S201-S206:

[0098] S201, Obtain the target object report and the structured annotation results corresponding to the target object report; the structured annotation results include multiple annotation result sets;

[0099] Wherein, step S201 of the embodiment of the present invention is referred to Figure 2 The specific description of step S101 in the illustrated embodiment will not be repeated here.

[0100] S202, perform structured processing on the target object report to generate a structured prediction result corresponding to the target object report; the structured prediction result includes multiple prediction result sets;

[0101] Wherein, step S202 of the embodiment of the present invention is referred to Figure 2 The specific description of step S102 in the illustrated embodiment will not be repeated here.

[0102] S203, obtain the similarity between each set of prediction results in the structured prediction results and each set of annotation results in the structured annotation results, and generate a similarity matrix;

[0103] Specifically, the user terminal obtains the target prediction result set from the structured prediction results and the target annotation result set from the structured annotation results. The target prediction result set is any one of the prediction result sets in the structured prediction results, and the target annotation result set is any one of the annotation result sets in the structured annotation results. Further, the attribute information in the target prediction result set is compared with the attribute information in the target annotation result set to generate a similarity score. Please refer to [link to relevant documentation]. Figure 6 , Figure 6 This is an example illustration of similarity generation provided by an embodiment of this application. The figure includes three sets of predicted results and three sets of labeled results. For example, to calculate the similarity between the first set of predicted results and the second set of labeled results, each attribute in the predicted results set is compared with each attribute in the labeled results set. For instance, in the figure, "Type: Low Echo (Present)" in the predicted results set matches "Type: Low Echo (Present)" in the labeled results set. Each successful match increases the similarity by 1. It should be noted that for more important attributes, higher weights can be assigned. For example, if "Location" and "Type" are more important, the weight of "Location" and "Type" is increased by 0.1, that is, a successful match of "Location" or "Type" increases the similarity by 1.1. Therefore, in the figure, the first set of predicted results and the second set of labeled results have three identical attributes, and two of these attributes are "Location" and "Type", so the similarity is 5.2. Using the above similarity calculation method, even if the order of different attributes in the labeled result set is different from that in the predicted result set, the similarity calculation can still be completed accurately. For example, the first set in the predicted result set corresponds to the second set in the labeled result set. Specifically, the third attribute "change trend" in the predicted result set corresponds to the fourth attribute "change trend" in the labeled result set, and the fourth attribute "sound transmission" in the predicted result set corresponds to the third attribute "sound transmission" in the labeled result set. Therefore, a similarity of 5.2 can be accurately calculated. Furthermore, fuzzy matching is supported during the matching process. For example, "TCM discharge diagnosis" and "discharge TCM diagnosis" will be detected as the same, and different date formats, such as June 16, 2021 and 2021-06-16, will be considered the same time.

[0104] Furthermore, the user terminal generates a similarity matrix for the target object report based on the similarity between each prediction result set and each labeled result set. See also... Figure 7a , Figure 7aThis is an example diagram illustrating the generation of a similarity matrix provided in this application embodiment. The structured prediction result includes 4 sets of prediction results, and the structured annotation result includes 4 sets of annotation results. A 4*4 similarity matrix can be generated based on the similarity.

[0105] S204, Obtain the matching relationship between the prediction result set and the annotation result set based on the similarity matrix;

[0106] Specifically, the user terminal obtains the maximum similarity in the similarity matrix and records the index of the maximum similarity in the similarity matrix. Each index corresponds to a set pair, which includes a prediction result set and a labeling result set. The row and column of the maximum similarity in the similarity matrix are then negatively reset, specifically by resetting the similarity to -1. After all similarities in the similarity matrix have been negatively reset, an index sequence is generated based on the indices corresponding to at least two maximum similarities. The matching relationship between the prediction result set and the labeling result set is then generated based on the set pairs corresponding to the index sequence. Please refer to [link to relevant documentation]. Figure 7a The maximum similarity score of 5.2 is found in the similarity matrix, and the index (1, 2) corresponding to this maximum score is determined. This index represents the matching set pair, indicating that the second predicted set in the structured prediction results and the third labeled set in the structured labeling results are the best-matching pair. Since these two sets have already matched successfully, they cannot participate in the matching of other sets. Therefore, the matrix needs to be updated. Thus, the entire row and column corresponding to the maximum score are set to -1 to ensure that it will not affect the matching of subsequent sets.

[0107] For the updated similarity matrix, please refer to [link / reference]. Figure 7b , Figure 7b This is an example diagram illustrating an update of the similarity matrix provided in an embodiment of this application, such as... Figure 7b As shown, the maximum similarity value of 3.1 is found in the similarity matrix, and the index (3, 1) corresponding to the maximum value is determined. This indicates that the fourth prediction result set in the structured prediction result and the second annotation result set in the structured annotation result are the best matching pair. Then, the entire row and column corresponding to the maximum value are set to -1 to update the similarity matrix.

[0108] Using the above method, it is known that all elements in the similarity matrix are reset to -1. Please refer to [link / reference]. Figures 7c-7d , Figures 7c-7dThis is an example diagram of updating a similarity matrix provided in an embodiment of this application. The index sequence of successful matching is [(1,2), (3,1), (2,3), (0,0)], that is, the 1st, 3rd, 2nd, and 0th prediction result sets in the structured prediction results match the 2nd, 1st, 3rd, and 0th annotation result sets in the structured annotation results, respectively.

[0109] It should be noted that if there are unmatched sets, it means that the number of prediction result sets is different from the number of annotation result sets. Specifically, if there are unmatched prediction result sets in the structured prediction results, it is determined that the structured prediction results reported by the target object have redundant prediction results; if there are unmatched annotation result sets in the structured annotation results, it is determined that the structured prediction results reported by the target object have missing prediction results. Based on the unmatched prediction result sets, the unmatched annotation result sets, and the set pairs corresponding to the index sequence, a matching relationship between the prediction result sets and the annotation result sets is generated.

[0110] S205, compare and detect the prediction result set and the annotation result set corresponding to the matching relationship, and generate the comparison and detection results of the target object report.

[0111] Specifically, the user terminal compares and detects the predicted result set and the labeled result set corresponding to the matching relationship, obtaining the differences between the two sets. These differences include error information, missing prediction information, and redundant prediction information in the structured prediction results. Based on these differences, a comparison and detection result of the target object report is generated. Please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is an example diagram illustrating the generation of comparative detection results provided in an embodiment of this application, such as... Figure 8 As shown, the figure includes the original report, multiple sets of predicted results, and multiple sets of annotated results. The predicted result sets include sets of visible organ descriptions, visible nodule descriptions, and examination conclusions. The annotated result sets include annotated sets of visible organ descriptions, annotated sets of visible nodule descriptions, and annotated sets of examination conclusions. Through comparative detection, the differences between the predicted result sets and the annotated result sets can be compared and output. Specifically, the comparative detection result for visible organ descriptions is "Location error; predicted: Location: Left nipple; annotated: Location: Right nipple." The comparative detection result for visible nodule descriptions is "Location error; predicted: Location: Right breast; annotated: Location: Left breast." The comparative detection result for examination conclusions is "Location error; predicted: Location: Left breast; annotated: Location: Breast; Location error; predicted: Location: Left breast; annotated: Location: Right breast; Lesion error; predicted: Lesion: Hypoechoic mass (space-occupying lesion); annotated: Lesion: Hyperechoic mass (space-occupying lesion)."

[0112] S206, Generate detection parameters for the target object report based on the comparison detection results; the detection parameters are used to measure the prediction effect of the structured prediction results.

[0113] In this embodiment of the invention, step S206 is described below. Figure 2 The specific description of step S104 in the illustrated embodiment will not be repeated here.

[0114] Please see Figure 9 , Figure 9 This is an example diagram illustrating the generation of comparative detection results provided in an embodiment of this application, such as... Figure 9 As shown in the figure, the correct prediction (TP), missing prediction information (FN), and redundant prediction information (FP) are given for multiple attributes. Based on the aforementioned TP, FN, and FP, the accuracy, recall, precision, and F1 score for each attribute are generated. The figure demonstrates that the method in this approach compares each set of predicted results with multiple sets of labeled results in the structured annotation results. This avoids the problem of inconsistent matching results caused by different attribute information order in the predicted and labeled results when comparing characters one by one, thus improving detection efficiency and ensuring the accuracy of the detection results.

[0115] In this embodiment, by acquiring a target object report and its corresponding structured annotation results (which include multiple annotation result sets), the target object report is further processed in a structured manner to generate a structured prediction result, which also includes multiple prediction result sets. Furthermore, the prediction result sets of the structured prediction results and the annotation result sets of the structured annotation results are compared and detected to generate a comparison and detection result for the target object report. Finally, detection parameters for the target object report are generated based on the comparison and detection results, and these parameters are used to measure the prediction effect of the structured prediction results. Using this method, a structured prediction result including multiple prediction result sets can be generated. By comparing each prediction result set with multiple annotation result sets in the structured annotation results, the problem of different matching results caused by different attribute information order in the prediction and annotation results when comparing characters one by one is avoided, thus improving detection efficiency and ensuring the accuracy of the detection results.

[0116] Please see Figure 10 , Figure 10This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of this application. The data processing apparatus may be a computer program (including program code) running on a computer device; for example, the data processing apparatus may be an application software. The apparatus may be used to execute corresponding steps in the methods provided in the embodiments of this application. Figure 10 As shown, the data processing device 1 in this application embodiment may include: a labeling result acquisition unit 11, a prediction result acquisition unit 12, a detection result generation unit 13, and a detection parameter generation unit 14.

[0117] The annotation result acquisition unit 11 is used to acquire the target object report and the structured annotation results corresponding to the target object report; the structured annotation results include multiple annotation result sets;

[0118] The prediction result acquisition unit 12 is used to perform structured processing on the target object report and generate a structured prediction result corresponding to the target object report; the structured prediction result includes multiple prediction result sets;

[0119] The detection result generation unit 13 is used to compare and detect the prediction result set of the structured prediction result and the annotation result set of the structured annotation result, and generate the comparison and detection result of the target object report.

[0120] The detection parameter generation unit 14 is used to generate detection parameters for the target object report based on the comparative detection results; the detection parameters are used to measure the prediction effect of the structured prediction results.

[0121] In one feasible implementation, the prediction result acquisition unit is used for:

[0122] Multiple nodes are generated based on the structure of the target object report, and a set of prediction results corresponding to each node is generated based on the attribute information corresponding to each node.

[0123] The set of prediction results corresponding to each node is determined as the structured prediction results corresponding to the target object report.

[0124] In one feasible implementation, the detection result generation unit is used for:

[0125] Obtain the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generate a similarity matrix;

[0126] The matching relationship between the predicted result set and the labeled result set is obtained based on the similarity matrix;

[0127] The predicted result set and the labeled result set corresponding to the matching relationship are compared and detected to generate the comparison and detection results of the target object report.

[0128] In one feasible implementation, the detection result generation unit is used for:

[0129] Obtain the target prediction result set from the structured prediction results and the target annotation result set from the structured annotation results; the target prediction result set is any one of the prediction result sets in the structured prediction results, and the target annotation result set is any one of the annotation result sets in the structured annotation results;

[0130] The attribute information in the target prediction result set is compared with the attribute information in the target annotation result set to generate a similarity score;

[0131] A similarity matrix for the target object report is generated based on the similarity between each set of predicted results and each set of labeled results.

[0132] In one feasible implementation, the detection result generation unit is used for:

[0133] Obtain the maximum similarity in the similarity matrix and record the index of the maximum similarity in the similarity matrix; each index corresponds to a set pair, and the set pair includes a set of prediction results and a set of annotation results;

[0134] The row and column of the maximum similarity in the similarity matrix are reset to negative values;

[0135] After all similarities in the similarity matrix are reset to negative values, an index sequence is generated based on the indices corresponding to at least two maximum similarities.

[0136] The matching relationship between the predicted result set and the labeled result set is generated based on the set corresponding to the index sequence.

[0137] Please see Figure 10 The data processing device 1 described in this application embodiment may further include: a set matching detection unit 15.

[0138] The set matching detection unit is used to determine that there are redundant prediction results in the structured prediction results reported by the target object if there is an unmatched set of prediction results in the structured prediction results.

[0139] If there is a set of unmatched annotation results in the structured annotation results, it is determined that the structured prediction results reported by the target object are missing.

[0140] The detection result generation unit is used for:

[0141] Based on the set of unmatched prediction results, the set of unmatched annotation results, and the set pairs corresponding to the index sequence, a matching relationship between the set of prediction results and the set of annotation results is generated.

[0142] In one feasible implementation, the detection parameter generation unit is used for:

[0143] Based on the comparative detection results, the prediction category of each prediction result set in the structured prediction results is statistically analyzed;

[0144] Based on the prediction category of each prediction result set, detection parameters for the target object report are generated; the detection parameters include accuracy, recall, precision, and F1 score.

[0145] In this embodiment, by acquiring a target object report and its corresponding structured annotation results (which include multiple annotation result sets), the target object report is further processed in a structured manner to generate a structured prediction result, which also includes multiple prediction result sets. Furthermore, the prediction result sets of the structured prediction results and the annotation result sets of the structured annotation results are compared and detected to generate a comparison and detection result for the target object report. Finally, detection parameters for the target object report are generated based on the comparison and detection results, and these parameters are used to measure the prediction effect of the structured prediction results. Using this method, a structured prediction result including multiple prediction result sets can be generated. By comparing each prediction result set with multiple annotation result sets in the structured annotation results, the problem of different matching results caused by different attribute information order in the prediction and annotation results when comparing characters one by one is avoided, thus improving detection efficiency and ensuring the accuracy of the detection results.

[0146] Please see Figure 11 , Figure 11 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 11As shown, the computer device 1000 may include: at least one processor 1001, such as a CPU, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen, and optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 11 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a data processing application program.

[0147] exist Figure 11 In the computer device 1000 shown, the network interface 1004 provides network communication functions, the user interface 1003 is mainly used to provide an input interface for the user, and the processor 1001 can be used to call the data processing application stored in the memory 1005 to achieve the above. Figures 2-9 The description of the data processing method in any corresponding embodiment will not be repeated here.

[0148] It should be understood that the computer device 1000 described in the embodiments of this application can execute the foregoing text. Figures 2-9 The description of the data processing method in any corresponding embodiment can also be performed as described above. Figure 10 The description of the data processing device in the corresponding embodiments will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated.

[0149] Furthermore, it should be noted that this application embodiment also provides a computer-readable storage medium, which stores a computer program executed by the aforementioned data processing device, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the aforementioned... Figures 2-9The description of the data processing method in any corresponding embodiment is already provided, and therefore will not be repeated here. Furthermore, the beneficial effects of using the same method will also not be repeated. For technical details not disclosed in the computer-readable storage medium embodiments related to this application, please refer to the description of the method embodiments of this application. As an example, program instructions may be deployed to execute on a single computing device, or on multiple computing devices located in one location, or on multiple computing devices distributed across multiple locations and interconnected via a communication network. These multiple computing devices distributed across multiple locations and interconnected via a communication network can constitute a blockchain system.

[0150] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The computer-readable storage medium can be an internal storage unit of a data processing apparatus or device provided in any of the foregoing embodiments, such as a hard disk or memory of an electronic device. The computer-readable storage medium can also be an external storage device of the electronic device, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device. The computer-readable storage medium can also include magnetic disks, optical disks, read-only memory (ROM), or random access memory, etc. Furthermore, the computer-readable storage medium can include both internal storage units and external storage devices of the electronic device. The computer-readable storage medium is used to store the computer program and other programs and quantities required by the electronic device. The computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0151] The terms "first," "second," etc., used in the claims, description, and drawings of this invention are used to distinguish different objects, not to describe a particular order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that comprises a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses. References to "embodiment" herein mean that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The presentation of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments. The term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0152] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Those skilled in the art can implement the described functions using different methods for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0153] In the various embodiments of this application, the functional units can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0154] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Obtain the target object report and the corresponding structured annotation results; The structured annotation results include multiple annotation result sets; Multiple nodes are generated based on the structure of the target object report, and a set of prediction results corresponding to each node is generated based on the attribute information corresponding to each node. The target object report includes one or more nodes corresponding to basic information, visible description, and inspection conclusion, respectively. The basic information, visible description, and inspection conclusion are divided according to the structure of the target object report. The set of prediction results corresponding to each node is determined as the structured prediction results corresponding to the target object report; The structured prediction results include multiple sets of prediction results; Obtain the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generate a similarity matrix; the similarity matrix includes the similarity between any set of labeled results and any set of predicted results. The matching relationship between the predicted result set and the labeled result set is obtained based on the similarity matrix; the matching relationship includes an index sequence, wherein a pair of indices in the index sequence is used to indicate a predicted result set and the labeled result set with the maximum similarity to the predicted result set; The prediction result set and the labeled result set corresponding to the matching relationship are compared and detected to generate the comparison and detection results of the target object report; the comparison and detection refers to comparing and detecting the prediction result set and the labeled result set belonging to the same pair of indexes; The detection parameters for generating the target object report are generated based on the comparative detection results; the detection parameters are used to measure the predictive effect of the structured prediction results.

2. The method according to claim 1, characterized in that, The step of obtaining the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured labeled results, and generating a similarity matrix, includes: Obtain the target prediction result set from the structured prediction results and the target annotation result set from the structured annotation results; the target prediction result set is any one of the prediction result sets in the structured prediction results, and the target annotation result set is any one of the annotation result sets in the structured annotation results; The attribute information in the target prediction result set is compared with the attribute information in the target annotation result set to generate a similarity score; A similarity matrix for the target object report is generated based on the similarity between each set of predicted results and each set of labeled results.

3. The method according to claim 1, characterized in that, The step of obtaining the matching relationship between the prediction result set and the annotation result set based on the similarity matrix includes: Obtain the maximum similarity in the similarity matrix and record the index of the maximum similarity in the similarity matrix; each index corresponds to a set pair, and the set pair includes a set of prediction results and a set of annotation results; The row and column of the maximum similarity in the similarity matrix are reset to negative values; After all similarities in the similarity matrix are reset to negative values, an index sequence is generated based on the indices corresponding to at least two maximum similarities. The matching relationship between the predicted result set and the labeled result set is generated based on the set corresponding to the index sequence.

4. The method according to claim 3, characterized in that, Also includes: If there is a set of unmatched prediction results in the structured prediction results, then it is determined that the structured prediction results reported by the target object contain redundant prediction results. If there is a set of unmatched annotation results in the structured annotation results, it is determined that the structured prediction results reported by the target object are missing. The step of generating a matching relationship between the predicted result set and the labeled result set based on the set corresponding to the index sequence includes: Based on the set of unmatched prediction results, the set of unmatched annotation results, and the set pairs corresponding to the index sequence, a matching relationship between the set of prediction results and the set of annotation results is generated.

5. The method according to claim 1, characterized in that, The detection parameters for generating the target object report based on the comparison detection results include: Based on the comparative detection results, the prediction category of each prediction result set in the structured prediction results is statistically analyzed; Based on the prediction category of each prediction result set, detection parameters for the target object report are generated; the detection parameters include accuracy, recall, precision, and F1 score.

6. A data processing apparatus, characterized in that, include: The annotation result acquisition unit is used to acquire the target object report and the structured annotation results corresponding to the target object report; The structured annotation results include multiple annotation result sets; The prediction result acquisition unit is used to generate multiple nodes according to the structure of the target object report, generate a prediction result set corresponding to each node according to the attribute information corresponding to each node, and determine the prediction result set corresponding to each node as the structured prediction result corresponding to the target object report. The structured prediction results include multiple prediction result sets; the target object report includes one or more nodes corresponding to basic information, visible description, and inspection conclusion, and the basic information, visible description, and inspection conclusion are divided according to the structure of the target object report; The detection result generation unit is used to obtain the similarity between each set of predicted results in the structured prediction results and each set of labeled results in the structured annotation results, and generate a similarity matrix; the similarity matrix includes the similarity between any set of labeled results and any set of predicted results. The detection result generation unit is further configured to obtain the matching relationship between the predicted result set and the labeled result set based on the similarity matrix; the matching relationship includes an index sequence, wherein a pair of indices in the index sequence is used to indicate a predicted result set and a labeled result set that has the highest similarity to the predicted result set; The detection result generation unit is also used to compare and detect the prediction result set and the labeled result set corresponding to the matching relationship, and generate the comparison and detection result of the target object report; the comparison and detection refers to comparing and detecting the prediction result set and the labeled result set belonging to the same pair of indices; The detection parameter generation unit is used to generate detection parameters for the target object report based on the comparative detection results; the detection parameters are used to measure the prediction effect of the structured prediction results.

7. A computer device, characterized in that, include: Processor, memory, and network interface; The processor is connected to the memory and the network interface, wherein the network interface is used to provide network communication functions, the memory is used to store program code, and the processor is used to call the program code to execute the method according to any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted to be loaded by a processor and to execute the method of any one of claims 1-5.

9. A computer program product or computer program comprising computer instructions stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions to perform the method of any one of claims 1-5.

Images