An algorithm component problem positioning method and device, and a storage medium

Abstract

The application discloses a problem positioning method and device of an algorithm component and a storage medium, relates to the technical field of artificial intelligence, and can improve the efficiency and accuracy of problem positioning of the algorithm component and reduce the cost of problem positioning. The method comprises the following steps: acquiring at least one abnormal sample and true value data corresponding to each abnormal sample, processing each abnormal sample by using a target algorithm component to obtain processing results of each functional module in the target algorithm component on each abnormal sample; for each functional module in the target algorithm component, obtaining an evaluation index of the functional module based on the processing results of each abnormal sample by the functional module and the labeled true value of each abnormal sample corresponding to the functional module; judging whether the functional module is a to-be-optimized functional module based on the evaluation index of the functional module; and when there is at least one to-be-optimized module in the target algorithm component, positioning problems of each to-be-optimized module to determine the problems of each to-be-optimized module.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a problem localization method, apparatus and storage medium for an algorithm component. Background Technology

[0002] With the increasing prevalence of AI (Artificial Intelligence) technology and its application across various industries, the application scenarios for AI are becoming more diverse. Correspondingly, the complexity of the corresponding AI algorithm components is also constantly rising. When these algorithm components are implemented in new scenarios, they may exhibit poor performance due to various reasons. Therefore, it is necessary to pinpoint the problems that arise in the algorithm components in order to further optimize them.

[0003] In related technologies, when the performance of an algorithm component is detected to be poor, it is necessary to have experienced algorithm engineers to intervene in the analysis. Therefore, the efficiency and accuracy of problem localization are low, and the labor cost is high. Summary of the Invention

[0004] This application provides a method, apparatus, and storage medium for locating problems in algorithm components, which can improve the efficiency and accuracy of locating problems in algorithm components and reduce the cost of locating problems in algorithm components.

[0005] In a first aspect, embodiments of this application provide a method for locating problems in an algorithm component. The method includes: acquiring at least one anomalous sample and ground truth data corresponding to each anomalous sample; the anomalous sample being a sample that produces an erroneous result after processing by a target algorithm component; the target algorithm component including multiple functional modules; and the ground truth data corresponding to the anomalous sample including the labeled ground truth values ​​of the anomalous samples corresponding to each functional module in the target algorithm component; processing each anomalous sample using the target algorithm component to obtain the processing results of each functional module in the target algorithm component for each anomalous sample; for each functional module in the target algorithm component, obtaining an evaluation index for the functional module based on the processing results of the functional module for each anomalous sample and the labeled ground truth values ​​of the anomalous samples corresponding to the functional module; determining whether the functional module is a functional module to be optimized based on the evaluation index of the functional module; and, when there is at least one functional module to be optimized in the target algorithm component, locating problems in each functional module to be optimized to determine the problems occurring in each functional module to be optimized.

[0006] The technical solution provided in this application provides at least the following beneficial effects: On the one hand, the problem localization of the target algorithm component does not rely on manual analysis, reducing the cost of problem localization and improving efficiency. On the other hand, when locating the module to be optimized in the target algorithm component, the localization is based on the evaluation index determined by the processing results of each functional module on each abnormal sample and the labeled true value of the abnormal sample corresponding to each functional module. Therefore, each functional module is evaluated during problem localization, and further, the problems of each module to be optimized are located, thus improving the accuracy of problem localization.

[0007] In some embodiments, the functional modules in the target algorithm component include a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. The above-mentioned processing of each abnormal sample by the target algorithm component to obtain the processing results of each abnormal sample by each functional module in the target algorithm component includes: for each first type of functional module, inputting each abnormal sample into the first type of functional module to obtain the processing results of the first type of functional module for each abnormal sample; for each second type of functional module, inputting the labeled true values ​​of each abnormal sample corresponding to the preceding module of the second type of functional module into the second type of functional module to obtain the processing results of the second type of functional module for each abnormal sample.

[0008] It should be understood that for any functional module, after inputting the correct input data, the actual processing result of that module on the correct input data can be obtained. This allows for further comparison between the actual processing result and the correct processing result to analyze the module's performance. If the functional module is of type I, since it has no preceding module, the correct input data for that module is the individual anomalous samples themselves. Conversely, if the functional module is of type II, since it has a preceding module, and the output data of that preceding module is incorrect, the output data of that preceding module cannot be used as the correct input data for each anomalous sample in the second type of functional module. Therefore, the labeled true values ​​of each anomalous sample corresponding to that preceding module are used as the correct input data for each anomalous sample in the second type of functional module. This allows for obtaining accurate actual processing results of each functional module on each anomalous sample, thereby improving the accuracy of problem localization.

[0009] In some embodiments, problem localization is performed on each module to be optimized to determine the problems that occur in each module, including: for each module to be optimized, obtaining the labeled ground truth values ​​of each training sample used when training the module to be optimized; based on the labeled ground truth values ​​of each abnormal sample corresponding to the module to be optimized, determining the first statistical information of each labeled ground truth attribute parameter; based on the labeled ground truth values ​​of each training sample corresponding to the module to be optimized, determining the second statistical information of each labeled ground truth attribute parameter; and analyzing the difference between the first statistical information and the second statistical information of each labeled ground truth attribute parameter to determine the problems that occur in the module to be optimized.

[0010] Based on this, the specific problematic annotation truth attribute parameters can be further determined in the module to be optimized, thereby achieving accurate problem localization in the module to be optimized.

[0011] In some embodiments, the method further includes: searching a database for a solution that matches the problem of the module to be optimized, wherein multiple solutions are pre-stored in the database; when a solution that matches the problem of the module to be optimized is found in the database, recommending the solution to the user of the target algorithm component; or, when no solution that matches the problem of the module to be optimized is found in the database, recommending a manual solution to the user of the target algorithm component.

[0012] Based on this, after identifying the problem in the target algorithm component, if the identified problem exists in the database, the system can proactively recommend solutions to the user based on the relevant content of pre-stored solutions, so that the user can further refer to the solution to optimize the algorithm component; or, if the identified problem does not exist in the database, the system can recommend a manual solution so that relevant technical personnel can intervene and handle the issue, thereby improving the reliability of locating algorithm component problems.

[0013] In some embodiments, the method further includes: recommending a manual solution to the user of the target algorithm component when the number of modules to be optimized in the target algorithm component is greater than a preset number.

[0014] It should be understood that if the number of modules to be optimized in the target algorithm component is greater than the preset number, it indicates that the problem in the target algorithm component is a complex problem, and the problem location may not be accurate enough. Therefore, a manual solution is recommended so that relevant technical personnel can intervene to improve the reliability of the algorithm component problem location.

[0015] Secondly, embodiments of this application provide a problem localization device for an algorithm component, comprising: an abnormal sample acquisition module, configured to acquire at least one abnormal sample and ground truth data corresponding to each abnormal sample, wherein the abnormal sample is a sample that produces an erroneous result after being processed by a target algorithm component, the target algorithm component includes multiple functional modules, and the ground truth data corresponding to the abnormal sample includes the labeled ground truth values ​​of the abnormal samples corresponding to each functional module in the target algorithm component; an abnormal sample processing module, configured to process each abnormal sample using the target algorithm component to obtain the processing results of each functional module in the target algorithm component for each abnormal sample; and a problem localization module, configured to, for each functional module in the target algorithm component, obtain an evaluation index of the functional module based on the processing results of the functional module for each abnormal sample and the labeled ground truth values ​​of the abnormal samples corresponding to the functional module; determine whether the functional module is a functional module to be optimized based on the evaluation index of the functional module; and when there is at least one functional module to be optimized in the target algorithm component, perform problem localization on each functional module to be optimized to determine the problem that occurs in each functional module to be optimized.

[0016] In some embodiments, the functional modules in the target algorithm component include a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. The above-mentioned abnormal sample processing module is specifically used to input each abnormal sample into each first type of functional module to obtain the processing result of each abnormal sample by the first type of functional module. For each second type of functional module, the labeled true values ​​of each abnormal sample corresponding to the preceding module of the second type of functional module are input into the second type of functional module to obtain the processing result of each abnormal sample by the second type of functional module.

[0017] In some embodiments, the problem localization module is specifically used to: obtain the labeled ground truth values ​​of each training sample used when training the module to be optimized; determine the first statistical information of each labeled ground truth attribute parameter based on the labeled ground truth values ​​of each abnormal sample corresponding to the module to be optimized; determine the second statistical information of each labeled ground truth attribute parameter based on the labeled ground truth values ​​of each training sample corresponding to the module to be optimized; and analyze the difference between the first statistical information and the second statistical information of each labeled ground truth attribute parameter to determine the problem in the module to be optimized.

[0018] In some embodiments, the above-described apparatus further includes a solution recommendation module; the solution recommendation module is used to search for solutions matching the problems of the module to be optimized from a database, wherein multiple solutions are pre-stored in the database; when a solution matching the problem of the module to be optimized is found from the database, the solution matching the problem of the module to be optimized is recommended to the user of the target algorithm component; or, when no solution matching the problem of the module to be optimized is found from the database, a manual solution is recommended to the user of the target algorithm component.

[0019] In some embodiments, the above-described solution recommendation module is further configured to recommend manual solutions to the user of the target algorithm component when the number of modules to be optimized in the target algorithm component is greater than a preset number.

[0020] Thirdly, embodiments of this application provide a problem localization device for an algorithm component, comprising: a memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions; wherein, when the processor executes the computer instructions, the problem localization device for the algorithm component performs a problem localization method for the algorithm component as described in the first aspect and any of its possible design schemes.

[0021] Fourthly, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on a computer, cause the computer to perform the methods provided in the first aspect and possible implementations.

[0022] Fifthly, embodiments of this application provide a computer program product containing computer instructions that, when executed on a computer, cause the computer to perform the methods provided in the first aspect and possible implementations described above.

[0023] It should be noted that the aforementioned computer instructions may be stored, in whole or in part, on a computer-readable storage medium. This computer-readable storage medium may be packaged together with the controller's processor, or it may be packaged separately from the controller's processor; this application does not impose any limitations on this.

[0024] For a detailed description of aspects two through five and their various implementations in this application, please refer to the detailed description in aspect one and its various implementations. The beneficial effects of aspects two through five and their various implementations can be found in the analysis of the beneficial effects of aspect one and its various implementations; they will not be repeated here. Attached Figure Description

[0025] Figure 1 A flowchart illustrating a problem localization method based on an algorithm component according to some embodiments. Figure 1 ;

[0026] Figure 2 This is a schematic diagram of the logical topology of a target algorithm component according to some embodiments;

[0027] Figure 3 A flowchart illustrating a problem localization method based on an algorithm component according to some embodiments. Figure 2 ;

[0028] Figure 4 A flowchart illustrating a problem localization method based on an algorithm component according to some embodiments. Figure 3 ;

[0029] Figure 5 This is a histogram of the aspect ratio of the detection bounding box of the ID card region corresponding to a training sample according to some embodiments;

[0030] Figure 6 This is a histogram of the aspect ratio of the detection bounding box of the ID card region corresponding to an abnormal sample according to some embodiments;

[0031] Figure 7 This is a schematic diagram of the structure of a problem localization device based on an algorithm component according to some embodiments. Figure 1 ;

[0032] Figure 8 This is a schematic diagram of the structure of a problem localization device based on an algorithm component according to some embodiments. Figure 2 . Detailed Implementation

[0033] The technical solutions of this application will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner. The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, unless otherwise stated, "a plurality of" means two or more.

[0035] As described in the background section, with the continuous popularization of AI (Artificial Intelligence) technology and its application in various industries, the application scenarios of AI technology are becoming increasingly diverse. When algorithm components are deployed in new scenarios, they may exhibit poor performance due to various reasons. Therefore, it is necessary to pinpoint the problems encountered by the algorithm components in order to further optimize them. In related technologies, when poor performance of an algorithm component is detected, experienced algorithm engineers are required to intervene in the analysis. Therefore, the efficiency and accuracy of problem localization are relatively low, and the labor costs are high.

[0036] To address this issue, this application provides a method for locating problems in algorithm components. The core idea of ​​this method is as follows: first, based on the evaluation metrics determined by the processing results of each functional module of the target algorithm component on each abnormal sample and the labeled ground truth values ​​of the abnormal samples corresponding to each functional module, the modules to be optimized in the target algorithm component are identified; then, the problems occurring in each module to be optimized are further located, thereby improving the accuracy of problem location for the algorithm component. Furthermore, the problem location for the target algorithm component does not rely on manual analysis, reducing the cost of problem location and improving efficiency.

[0037] It should be noted that the specific application scenarios of the problem localization method for the algorithm component in this application embodiment are not limited, and it can be applied to scenarios that can be handled in the field of AI technology. Furthermore, the problem localization method for the algorithm component provided in this application embodiment can be executed by a problem localization device for the algorithm component. For example, the problem localization device for the algorithm component can be a computer device. Another example is that the problem localization device for the algorithm component can be a server. For example, the problem localization device of the algorithm component can be any electronic chip with processing capabilities, such as a central processing unit (CPU), graphics processing unit (GPU), network processor (NP), digital signal processor (DSP), microprocessor, microcontroller, programmable logic device (PLD), or any combination thereof; or, for example, the problem localization device of the algorithm component can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, ultra-mobile personal computer (UMPC), netbook, as well as cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, etc. This application does not impose any special restrictions on the specific form of the problem localization device of the algorithm component.

[0038] To facilitate understanding, the problem localization method of the algorithm components provided in this application will be described in detail below with reference to the accompanying drawings.

[0039] Figure 1 This application presents a problem localization method for an algorithm component, applied in the field of artificial intelligence technology. The following example uses a server as the execution subject. Figure 1 The method shown will be explained in detail. For example... Figure 1 As shown, the problem localization method of this algorithm component includes the following steps S101 to S104:

[0040] S101. Obtain at least one abnormal sample and the ground truth data corresponding to each abnormal sample.

[0041] Among them, abnormal samples are samples that produce erroneous results after being processed by the target algorithm component. The target algorithm component includes multiple functional modules and is the algorithm component that is expected to locate the problem.

[0042] For example, Figure 2 The diagram illustrates the logical topological relationship of a target algorithm component used for ID card information recognition. (Refer to...) Figure 2 The algorithm component 100 includes multiple functional modules: a first detection module 101, a second detection module 102, a first recognition module 103, a second recognition module 104, a third recognition module 105, and a post-processing module 106.

[0043] The first detection module 101 receives image data as input and detects the ID card region within the image data, outputting the location information corresponding to the ID card region to the second detection module 102. The second detection module 102 segments the image region corresponding to the ID card region from the image data based on the location information, performs field detection on the image region corresponding to the ID card region, and inputs the location information of each field into the corresponding recognition module. For example, the location of the ID card number field is input to the first recognition module 103, the location of the name field is input to the second recognition module 104, and the location of the address field is input to the third recognition module 105. This allows the first recognition module 103 to segment the image region corresponding to the ID card number field from the image data based on the ID card number field location and recognize the ID card number content within the image region. Similarly, the second recognition module 104 recognizes the name content, and the third recognition module 105 recognizes the address content. The post-processing module 106 is used to receive the recognition content (i.e., the ID card number mentioned above) from the first recognition module 103, the recognition content (i.e., the name mentioned above) from the second recognition module 104, and the recognition content (i.e., the address mentioned above) from the third recognition module 105, and to verify the recognition content of the first recognition module 103, the second recognition module 104, and the third recognition module 105, and output the verified ID card recognition result.

[0044] In some examples, the anomalous samples corresponding to the target algorithm component 100 can be determined in the following way:

[0045] Optionally, if, during the verification process of the post-processing module 106, the proportion of samples corresponding to the image data that fail the verification exceeds a preset proportion, then each sample corresponding to the image data that fails the verification is considered an abnormal sample. Optionally, the circumstances of failure may include at least one of the following: the length of the ID card number does not meet the preset length (e.g., 18 or 15 digits), the font of the name on the ID card is not a preset font, or the address on the ID card does not meet the preset address verification conditions (e.g., the residential address on the ID card is not within the preset nationality range). It should be understood that the above circumstances of failure are merely examples, and this application does not impose specific limitations on them.

[0046] Optionally, anomaly checks are set for each functional module of the target algorithm component 100, and abnormal samples are determined based on the anomaly check results of each functional module. For example, taking the first detection module 101 as an example, since the first detection module 101 is used to detect the ID card area in the image data, its output may include: the image coordinates of the upper left corner of the ID card in the image data, the rotation angle of the ID card, and the aspect ratio of the ID card. Taking the aspect ratio of the ID card as an example, if the aspect ratio of the ID card output by multiple consecutive image samples after passing through the first detection module 101 does not meet the preset ratio, then each of the multiple image samples is regarded as an abnormal sample.

[0047] Optionally, the output data of each functional module in the multiple functional modules of the target algorithm component 100 is evaluated to obtain the evaluation index of each functional module. Then, the evaluation indexes of each functional module in the multiple functional modules of the target algorithm component 100 are combined to determine whether the output result of the target algorithm component 100 is abnormal, and the input image sample when the output result is abnormal is regarded as an abnormal sample. For example, the evaluation indexes of each functional module of the target algorithm component 100 are preset to values ​​between [0, 100%]. If, after a certain image sample is input into the target algorithm component 100, the average value of the evaluation indexes of the first detection module 101, the second detection module 102, the first recognition module 103, the second recognition module 104, the third recognition module 105, and the post-processing module 106 is lower than a first threshold, then the image sample is determined to be an abnormal sample. For example, if, after an image sample is input into the target algorithm component 100, the average of the evaluation metrics of the first detection module 101 and the second detection module 102 is below 95%, and the average of the evaluation metrics of the first recognition module 103, the second recognition module 104, and the third recognition module 105 is below 90, then the image sample is identified as an anomalous sample. It should be understood that the above method of determining whether a target algorithm component is anomalous based on the average of all functional modules or the average of some functional modules is merely an example and does not impose specific limitations.

[0048] In some embodiments, the ground truth data corresponding to the anomalous samples includes the labeled ground truth values ​​of the anomalous samples corresponding to each functional module in the target algorithm component.

[0049] For example, still using Figure 2Taking the target algorithm component 100 as an example, the ground truth data corresponding to the abnormal sample may include: the labeled ground truth corresponding to the first detection module 101, the labeled ground truth corresponding to the second detection module 102, the labeled ground truth corresponding to the first recognition module 103, the labeled ground truth corresponding to the second recognition module 104, the labeled ground truth corresponding to the third recognition module 105, and the labeled ground truth corresponding to the post-processing module 106. Taking the first detection module 101 as an example, the labeled ground truth corresponding to the abnormal sample may include: the coordinates of the upper left corner of the rectangular detection box of the ID card in the image, the coordinates of the lower left corner of the rectangular detection box of the ID card in the image, the coordinates of the upper right corner of the rectangular detection box of the ID card in the image, and the coordinates of the lower right corner of the rectangular detection box of the ID card in the image. Taking the first recognition module 103 as an example, the labeled ground truth may include: the real ID card number in the abnormal sample. Taking the post-processing module 106 as an example, the labeled ground truth corresponding to the abnormal sample may include: the address verification result of the ID card in the abnormal sample (e.g., whether the residential address on the ID card belongs to a preset nationality). It should be understood that the specific content of the ground truth data corresponding to the above-mentioned abnormal samples is only an example, and this application does not impose any specific limitations on it.

[0050] Furthermore, it should be noted that the topological relationship of the target algorithm components described above is merely an example. The target algorithm components only need to be able to achieve the functions that meet the user's business needs. This application does not impose specific restrictions on the functional modules included in the target algorithm components, the specific functions of each functional module, or the topological relationship between each functional module.

[0051] S102. Process each abnormal sample using the target algorithm component to obtain the processing results of each abnormal sample by each functional module in the target algorithm component.

[0052] In some embodiments, the functional modules in the target algorithm component include either a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. For example, referring to... Figure 2 , Figure 2 The first detection module 101 is the first functional module. The first detection module 101 does not have a preceding module, but directly uses image data as input data. Figure 2The second detection module 102, the first recognition module 103, the second recognition module 104, the third recognition module 105, and the post-processing module 106 are all second type functional modules; wherein, the preceding module of the second detection module 102 is the first detection module 101, the preceding modules of the first recognition module 103, the second recognition module 104, and the third recognition module 105 are all the second detection module 102, and the preceding modules of the post-processing module 106 include: the first recognition module 103, the second recognition module 104, and the third recognition module 105.

[0053] Furthermore, such as Figure 3 As shown, step S102 above can be specifically implemented as the following steps S1021 and S1022:

[0054] S1021. For each first type of functional module, each abnormal sample is input into the first type of functional module to obtain the processing result of each abnormal sample by the first type of functional module.

[0055] It should be understood that the processing result of the first type of functional module for each abnormal sample is the processing result of the first type of functional module when it receives the correct input data corresponding to each abnormal sample in the first type of functional module. Since the first type of functional module has no preceding module, the correct input data corresponding to the first type of functional module is the abnormal sample itself.

[0056] S1022. For each second type of functional module, input the labeled true values ​​of each abnormal sample corresponding to the preceding module of the second type of functional module into the second type of functional module to obtain the processing results of the second type of functional module for each abnormal sample.

[0057] It should be understood that the processing result of the second type of functional module for each abnormal sample is the processing result of the second type of functional module when it receives the correct input data corresponding to each abnormal sample in the second type of functional module. Since the second type of functional module has a preceding module, and the output data of the preceding module is incorrect, the output data of the preceding module cannot be used as the correct input data for each abnormal sample in the second type of functional module. Therefore, the labeled truth value of each abnormal sample corresponding to the preceding module is used as the correct input data for each abnormal sample in the second type of functional module.

[0058] It should be noted that step S1022 can be executed after step S1021, before step S1021, or simultaneously with step S1021. This application does not impose any specific restrictions on this.

[0059] also, Figure 3The illustrated embodiment can bring the following beneficial effects: For any functional module among the various functional modules, by inputting the correct input data into the functional module, the actual processing result of the functional module for each abnormal sample can be accurately obtained, thereby improving the accuracy of problem localization, and facilitating the subsequent comparison of the actual processing result and the correct processing result of the functional module for further analysis of the functional module's situation.

[0060] S103. For each functional module in the target algorithm component, based on the processing results of each abnormal sample by the functional module and the labeled true values ​​of each abnormal sample corresponding to the functional module, obtain the evaluation index of the functional module; based on the evaluation index of the functional module, determine whether the functional module is a functional module to be optimized.

[0061] In some embodiments, for any functional module among the various functional modules, the evaluation index of the functional module can be set according to the specific content of the functional module.

[0062] For example, if the function of this module is object detection, the evaluation metrics for this module may include recall and / or precision. Recall is the ratio of the number of correctly detected targets (e.g., the number of correctly detected bounding boxes) to the total number of true targets (e.g., the number of true bounding boxes); precision is the ratio of the number of correctly detected targets to the total number of all targets detected by this module (e.g., the total number of all detected bounding boxes). It should be understood that the recall and precision mentioned above are merely examples; for instance, the evaluation metric may also include accuracy, and this application does not impose specific limitations on this.

[0063] For example, if the function of this module is text recognition, the evaluation metrics for this module may include field recognition rate and / or edit distance recognition rate. Field recognition rate is the proportion of correctly recognized fields out of the total number of fields in this module; edit distance recognition rate, also known as edit distance accuracy, satisfies the following relationship: Edit distance recognition rate = (1 - number of edits / total number of characters in the sample set), where the number of edits is the number of editing operations required to correct the recognized fields. Editing operations include one or more of the following: inserting characters, deleting characters, or replacing characters.

[0064] In some embodiments, any functional module among the various functional modules may include one or more evaluation metrics. Optionally, if a functional module includes only one evaluation metric, such as accuracy, then if the accuracy of the functional module is lower than a second threshold, its performance is poor, and the functional module is set as a module to be optimized. Optionally, if a functional module includes multiple evaluation metrics, such as field recognition rate and edit distance recognition rate, then if the weighted average of the field recognition rate and edit distance recognition rate is lower than a third threshold, its performance is poor, and the functional module is set as a module to be optimized.

[0065] S104. When there is at least one module to be optimized in the target algorithm component, locate the problem of each module to be optimized and determine the problem of each module to be optimized.

[0066] In some embodiments, such as Figure 4 As shown, step S104 can be specifically implemented as steps S1041 to S1044:

[0067] S1041. For each module to be optimized, obtain the labeled true values ​​of each training sample used when training the module to be optimized.

[0068] S1042. Based on the labeled true values ​​of each abnormal sample corresponding to the module to be optimized, determine the first statistical information of each labeled true value attribute parameter.

[0069] In some examples, if the function of the module to be optimized is object detection, the first statistical information of the labeled truth attribute parameters of each abnormal sample corresponding to the module to be optimized includes: the aspect ratio histogram of the detection box corresponding to each abnormal sample, the rotation angle histogram of the detection box corresponding to each abnormal sample, and the area ratio histogram of the detection box corresponding to each abnormal sample, etc.

[0070] In other examples, if the function of the module to be optimized is text recognition, the first statistical information of the labeled truth attribute parameters of each abnormal sample corresponding to the module to be optimized includes: the frequency or proportion of characters appearing in the field corresponding to each abnormal sample, the frequency or proportion of each string appearing in the field corresponding to each abnormal sample, the color of characters in the field corresponding to each abnormal sample, and the font of characters in the field corresponding to each abnormal sample.

[0071] It should be understood that the above first statistical information is only an example to facilitate technical personnel's understanding of the solution. The specific content of the first statistical information may vary depending on the function of the module to be optimized. Furthermore, the first statistical information may include one or more labeled truth value attribute parameters, which can be changed according to the user's needs. This application does not impose any specific restrictions on this.

[0072] S1043. Based on the labeled true values ​​of each training sample corresponding to the module to be optimized, determine the second statistical information of each labeled true value attribute parameter.

[0073] In some examples, if the function of the module to be optimized is object detection, the second statistical information of the labeled true attribute parameters of each training sample corresponding to the module to be optimized includes: the aspect ratio histogram of the detection box corresponding to each training sample, the rotation angle histogram of the detection box corresponding to each training sample, and the area ratio histogram of the detection box corresponding to each training sample.

[0074] In other examples, if the function of the module to be optimized is text recognition, the second statistical information of the labeled true value attribute parameters of each training sample corresponding to the module to be optimized includes: the frequency or proportion of characters in the field corresponding to each training sample, the frequency or proportion of each string in the field corresponding to each training sample, the color of characters in the field corresponding to each training sample, and the font of characters in the field corresponding to each training sample.

[0075] It should be understood that the above-mentioned second statistical information is only an example, and the specific content of the first statistical information will vary depending on the function of the module to be optimized. Furthermore, the first statistical information may include one or more labeled truth value attribute parameters, which can be changed according to the user's needs. This application does not impose any specific restrictions on this.

[0076] S1044. Analyze the differences between the first and second statistical information of each labeled truth attribute parameter to identify the problems in the module to be optimized.

[0077] For example, still using Figure 2 Taking the logical topological relationship of the target algorithm components as an example. If... Figure 2 The first detection module 101 and the second recognition module 104 are both modules to be optimized, wherein:

[0078] The first statistical information of each labeled truth attribute parameter corresponding to the first detection module 101 includes, for example: Figure 5 The histograms shown represent the aspect ratios of the ID card region detection boxes for each training sample. The horizontal axis represents the aspect ratio of the ID card region detection boxes, and the vertical axis represents the proportion of training samples with different aspect ratios to the total number of samples. It can be seen that the aspect ratios of the ID card region detection boxes for each training sample fall between the first and second aspect ratios (the first aspect ratio is less than the second aspect ratio). The second statistical information for each labeled ground truth attribute parameter of the first detection module 101 includes, for example: Figure 6The histograms of aspect ratios of the ID card region detection boxes corresponding to each abnormal sample show that the aspect ratios of the ID card region detection boxes corresponding to each abnormal sample are concentrated between the third and fourth aspect ratios (the second aspect ratio is less than the third aspect ratio, and the third aspect ratio is less than the fourth aspect ratio). Therefore, the problem with the first detection module 101 can be identified as follows: there exists a detection box with an aspect ratio range that is not supported by the algorithm components.

[0079] The first statistical information of each labeled truth attribute parameter corresponding to the second recognition module 104 includes the font of the characters in the fields corresponding to each training sample, which is a Chinese font. The second statistical information of each labeled truth attribute parameter corresponding to the second recognition module 104 includes the font of the characters in the fields corresponding to each abnormal sample, which is an English font. At this time, the problem with the second recognition module 104 can be determined as: there is a font that the algorithm component does not support.

[0080] It should be noted that the problems of the modules to be optimized mentioned above are only examples. Different problems of the modules to be optimized can be determined based on different first and second statistical information. This application does not impose any specific restrictions on this.

[0081] It should be understood that Figure 1 The provided technical solution offers at least the following benefits: Firstly, it eliminates the need for manual analysis in locating problems within the target algorithm components, reducing costs and increasing efficiency. Secondly, when locating the modules to be optimized within the target algorithm components, the location is determined based on evaluation metrics established by the processing results of each functional module for each anomalous sample and the labeled ground truth values ​​of the anomalous samples corresponding to each functional module. Therefore, each functional module is evaluated during problem localization, and further, the problems in each module to be optimized are identified, thus improving the accuracy of problem localization.

[0082] In some embodiments, after performing step S104 above, the following steps are also performed: searching the database for a solution that matches the problem of the module to be optimized, wherein multiple solutions are pre-stored in the database; when a solution that matches the problem of the module to be optimized is found in the database, the solution that matches the problem of the module to be optimized is recommended to the user of the target algorithm component; or, when no solution that matches the problem of the module to be optimized is found in the database, a manual solution is recommended to the user of the target algorithm component.

[0083] For example, Table 1 shows the solution table corresponding to the embodiment in step S1044 above:

[0084] Table 1

[0085]

[0086] Referring to Table 1, it can be seen that, to address the issues of the first detection module and the second recognition module in this embodiment, the location information corresponding to the ID card area of ​​the abnormal sample and the name content of the ID card of the abnormal sample can be re-annotated before training the algorithm components. This method avoids the waste of manual labor costs caused by re-annotating and training all ground truths (such as ID card number content, ID card address content, etc.), thus improving the optimization efficiency of the algorithm components.

[0087] Based on this, after identifying the problem in the target algorithm component, if the identified problem exists in the database, the system can proactively recommend solutions to the user based on the relevant content of pre-stored solutions, so that the user can further refer to the solution to optimize the algorithm component and improve the optimization efficiency; or, if the identified problem does not exist in the database, the system can recommend manual solutions so that relevant technical personnel can intervene and improve the reliability of algorithm component problem identification.

[0088] In some other embodiments, after performing step S104 above, the following steps are also performed: when the number of modules to be optimized in the target algorithm component is greater than a preset number, a manual solution is recommended to the user of the target algorithm component.

[0089] It should be understood that if the number of modules to be optimized in the target algorithm component is greater than the preset number, it indicates that the problem in the target algorithm component is a complex problem, and the problem location may not be accurate enough. Therefore, a manual solution is recommended so that relevant technical personnel can intervene to improve the reliability of the algorithm component problem location.

[0090] As can be seen, the above mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, the embodiments of this application provide corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the modules and algorithm steps of the various examples described in the embodiments disclosed herein, the embodiments of this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0091] like Figure 7 As shown, this application embodiment provides a problem localization device for an algorithm component, used for executing... Figure 3The algorithm component shown is a problem localization method. The problem localization device 300 of the algorithm component includes: an anomaly sample acquisition module 301, an anomaly sample processing module 302, and a problem localization module 303.

[0092] An anomaly sample acquisition module 301 is used to acquire at least one anomaly sample and the corresponding ground truth data for each anomaly sample. An anomaly sample is a sample that produces an erroneous result after being processed by the target algorithm component. The target algorithm component includes multiple functional modules, and the ground truth data corresponding to the anomaly sample includes the labeled ground truth values ​​of the anomaly samples corresponding to each functional module in the target algorithm component. An anomaly sample processing module 302 is used to process each anomaly sample using the target algorithm component to obtain the processing results of each functional module in the target algorithm component for each anomaly sample. A problem localization module 303 is used to obtain the evaluation index of each functional module in the target algorithm component based on the processing results of each functional module for each anomaly sample and the labeled ground truth values ​​of each anomaly sample corresponding to the functional module. Based on the evaluation index of the functional module, it is determined whether the functional module is a functional module to be optimized. When there is at least one functional module to be optimized in the target algorithm component, the problem localization is performed on each functional module to be optimized to determine the problem that occurs in each functional module to be optimized.

[0093] In some embodiments, the functional modules in the target algorithm component include a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. The abnormal sample processing module 302 is specifically used to input each abnormal sample into each first type of functional module to obtain the processing result of each abnormal sample by the first type of functional module. For each second type of functional module, the labeled true values ​​of each abnormal sample corresponding to the preceding module of the second type of functional module are input into the second type of functional module to obtain the processing result of each abnormal sample by the second type of functional module.

[0094] In some embodiments, the problem localization module 303 is specifically used to: obtain the labeled ground truth values ​​of each training sample used when training the module to be optimized; determine the first statistical information of each labeled ground truth attribute parameter based on the labeled ground truth values ​​of each abnormal sample corresponding to the module to be optimized; determine the second statistical information of each labeled ground truth attribute parameter based on the labeled ground truth values ​​of each training sample corresponding to the module to be optimized; and analyze the difference between the first statistical information and the second statistical information of each labeled ground truth attribute parameter to determine the problem that occurs in the module to be optimized.

[0095] In some embodiments, the problem location device 300 of the above-mentioned algorithm component further includes a solution recommendation module 304; the solution recommendation module 304 is used to search for solutions matching the problem of the module to be optimized from the database, wherein multiple solutions are pre-stored in the database; when a solution matching the problem of the module to be optimized is found from the database, the solution matching the problem of the module to be optimized is recommended to the user of the target algorithm component; or, when no solution matching the problem of the module to be optimized is found from the database, a manual solution is recommended to the user of the target algorithm component.

[0096] In some embodiments, the above-described solution recommendation module 304 is further configured to recommend manual solutions to the user of the target algorithm component when the number of modules to be optimized in the target algorithm component is greater than a preset number.

[0097] It should be noted that, Figure 7 The module division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented either in hardware or as software functional modules.

[0098] In the case of implementing the functions of the integrated modules described above in hardware, this application embodiment provides another possible structural schematic diagram of the problem localization device for the algorithm components involved in the above embodiments. For example... Figure 8 As shown, the problem localization device 400 of the algorithm component includes: a processor 402 and a bus 404. Optionally, the problem localization device of the algorithm component may also include a memory 401; optionally, the problem localization device of the algorithm component may also include a communication interface 403.

[0099] Processor 402 may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 402 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 402 may also be a combination that implements computing functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0100] Communication interface 403 is used to connect to other devices via a communication network. This communication network can be Ethernet, wireless access network, wireless local area network (WLAN), etc.

[0101] The memory 401 may be a read-only memory (ROM) or other type of static storage device capable of storing static information and instructions, random access memory (RAM) or other type of dynamic storage device capable of storing information and instructions, or electrically erasable programmable read-only memory (EEPROM), disk storage medium or other magnetic storage device, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but is not limited thereto.

[0102] As one possible implementation, the memory 401 can exist independently of the processor 402. The memory 401 can be connected to the processor 402 via a bus 404 and is used to store instructions or program code. When the processor 402 calls and executes the instructions or program code stored in the memory 401, it can implement the problem localization method of the algorithm component provided in this application embodiment.

[0103] In another possible implementation, the memory 401 can also be integrated with the processor 402.

[0104] Bus 404 can be an extended industry standard architecture (EISA) bus, etc. Bus 404 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 8 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0105] Through the above description of the implementation methods, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the problem localization device of the algorithm component can be divided into different functional modules to complete all or part of the functions described above.

[0106] This application also provides a computer-readable storage medium. All or part of the processes in the above method embodiments can be executed by computer instructions instructing related hardware. The program can be stored in the aforementioned computer-readable storage medium, and when executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be any of the foregoing embodiments or memory. The aforementioned computer-readable storage medium can also be an external storage device of the aforementioned algorithm-based problem localization device, such as a pluggable hard drive, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the aforementioned algorithm-based problem localization device. Further, the aforementioned computer-readable storage medium can include both internal storage units of the aforementioned algorithm-based problem localization device and external storage devices. The aforementioned computer-readable storage medium is used to store the aforementioned computer program and other programs and data required by the aforementioned algorithm-based problem localization device. The aforementioned computer-readable storage medium can also be used to temporarily store data that has been output or will be output.

[0107] This application also provides a computer program product comprising a computer program that, when run on a computer, causes the computer to execute the problem localization method of any of the algorithm components provided in the above embodiments.

[0108] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0109] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0110] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A problem localization method for an algorithm component, characterized in that, include: At least one abnormal sample and the ground truth data corresponding to each abnormal sample are obtained. The abnormal sample is a sample that produces an erroneous result after being processed by the target algorithm component. The target algorithm component includes multiple functional modules. The ground truth data corresponding to the abnormal sample includes the labeled ground truth values ​​of the abnormal sample corresponding to each functional module in the target algorithm component. The target algorithm component is used to process each of the abnormal samples to obtain the processing results of each functional module in the target algorithm component for each of the abnormal samples. For each functional module in the target algorithm component, an evaluation index for the functional module is obtained based on the processing results of the functional module on each abnormal sample and the labeled ground truth value of each abnormal sample corresponding to the functional module; based on the evaluation index of the functional module, it is determined whether the functional module is a functional module to be optimized. When there is at least one module to be optimized in the target algorithm component, the problem of each module to be optimized is located and the problem of each module to be optimized is determined. The step of locating problems in each module to be optimized, and determining the problems in each module, includes: For each module to be optimized, obtain the labeled ground truth values ​​of each training sample used when training the module to be optimized; Based on the labeled ground truth values ​​of each abnormal sample corresponding to the module to be optimized, the first statistical information of each labeled ground truth attribute parameter is determined; Based on the labeled ground truth values ​​of each training sample corresponding to the module to be optimized, the second statistical information of each labeled ground truth attribute parameter is determined; By analyzing the differences between the first and second statistical information of each labeled truth attribute parameter, the problems of the module to be optimized can be identified.

2. The method according to claim 1, characterized in that, The functional modules in the target algorithm component include either a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. The step of processing each abnormal sample using the target algorithm component to obtain the processing results of each functional module in the target algorithm component for each abnormal sample includes: For each of the first type of functional modules, each abnormal sample is input into the first type of functional module to obtain the processing result of the first type of functional module for each abnormal sample; For each second type of functional module, the labeled true values ​​of each abnormal sample corresponding to the preceding module of the second type of functional module are input into the second type of functional module to obtain the processing results of the second type of functional module for each abnormal sample.

3. The method according to claim 1 or 2, characterized in that, The method further includes: Search the database for solutions that match the problems in the module to be optimized; multiple solutions are pre-stored in the database. When a solution matching the problem of the module to be optimized is found in the database, the solution matching the problem of the module to be optimized is recommended to users of the target algorithm component; or, If no matching solution is found in the database for the problem of the module to be optimized, a manual solution is recommended to the user of the target algorithm component.

4. The method according to claim 1 or 2, characterized in that, The method further includes: When the number of modules to be optimized in the target algorithm component exceeds a preset number, a manual solution is recommended to the user of the target algorithm component.

5. A problem localization device for an algorithm component, characterized in that, include: An abnormal sample acquisition module is used to acquire at least one abnormal sample and the ground truth data corresponding to each abnormal sample. The abnormal sample is a sample that produces an erroneous result after being processed by the target algorithm component. The target algorithm component includes multiple functional modules. The ground truth data corresponding to the abnormal sample includes the labeled ground truth values ​​of the abnormal sample corresponding to each functional module in the target algorithm component. An abnormal sample processing module is used to process each abnormal sample using the target algorithm component to obtain the processing results of each functional module in the target algorithm component for each abnormal sample. The problem localization module is used to, for each functional module in the target algorithm component, obtain an evaluation index for each functional module based on the processing results of each abnormal sample by the functional module and the labeled ground truth value of each abnormal sample corresponding to the functional module; determine whether the functional module is a functional module to be optimized based on the evaluation index of the functional module; and when there is at least one functional module to be optimized in the target algorithm component, perform problem localization on each functional module to be optimized to determine the problem of each functional module to be optimized. The step of locating problems in each module to be optimized, and determining the problems in each module, includes: For each module to be optimized, obtain the labeled ground truth values ​​of each training sample used when training the module to be optimized; Based on the labeled ground truth values ​​of each abnormal sample corresponding to the module to be optimized, the first statistical information of each labeled ground truth attribute parameter is determined; Based on the labeled ground truth values ​​of each training sample corresponding to the module to be optimized, the second statistical information of each labeled ground truth attribute parameter is determined; By analyzing the differences between the first and second statistical information of each labeled truth attribute parameter, the problems of the module to be optimized can be identified.

6. The apparatus according to claim 5, characterized in that, The functional modules in the target algorithm component include either a first type of functional module or a second type of functional module. The first type of functional module is a functional module without a preceding module, and the second type of functional module is a functional module with a preceding module. The device also includes a scheme recommendation module. The abnormal sample processing module is specifically used to input each abnormal sample into each first type of functional module to obtain the processing result of each abnormal sample by the first type of functional module; and to input the labeled truth value of each abnormal sample corresponding to the preceding module of the second type of functional module into the second type of functional module to obtain the processing result of each abnormal sample by the second type of functional module. The solution recommendation module is used to find solutions from the database that match the problems encountered by the module to be optimized. The database stores multiple solutions in advance. When a solution matching the problem of the module to be optimized is found in the database, the solution matching the problem of the module to be optimized is recommended to the users of the target algorithm component; Alternatively, if no solution matching the problem of the module to be optimized is found in the database, a manual solution is recommended to the user of the target algorithm component; The solution recommendation module is also used to recommend manual solutions to the user of the target algorithm component when the number of modules to be optimized in the target algorithm component is greater than a preset number.

7. A problem localization device for an algorithm component, characterized in that, include: A memory and a processor; the memory and the processor are coupled; the memory is used to store computer program code, the computer program code including computer instructions; Wherein, when the processor executes the computer instructions, the problem localization device of the algorithm component performs the problem localization method of the algorithm component as described in any one of claims 1 to 4.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes computer instructions; When the computer instructions are run on the problem location device of the algorithm component, the problem location device of the algorithm component performs the problem location method of the algorithm component as described in any one of claims 1 to 4.

Images