Model testing method and device, equipment and storage medium

Abstract

The embodiment of the invention provides a model testing method and device, equipment and a storage medium. The method comprises the steps that a plurality of local groups are created for a plurality of candidate defect types respectively; obtaining a plurality of target images, respectively inputting the plurality of target images into the defect detection model for detection to obtain a defect detection result of each target image, and for each target image, obtaining a defect detection result of a target defect type corresponding to the target image based on the first confidence interval and the target confidence of the target defect type corresponding to the target image; determining a first target group in each local group corresponding to the target defect type, and dividing the target image into the first target group; and for each candidate defect type, determining a first detection group in each local group corresponding to the candidate defect type, and determining a test result of the target model based on a relationship between the number of the target images corresponding to each first detection group and the total number of the target images. According to the embodiment of the invention, the stability and robustness of the model in a production field can be improved.

Description

Technical Field

[0001] The present application relates to, but is not limited to, the field of industrial testing technology, and in particular to a model testing method, apparatus, device, and storage medium. Background Art

[0002] Currently, the testing of defect detection models before deployment to the production site is usually the responsibility of the model's algorithm personnel. However, algorithm personnel are usually more focused on algorithm performance and ignore the complexity and diverse needs of the production environment. This may lead to incomplete testing and low stability and robustness of the defect detection model. Summary of the Invention

[0003] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0004] The embodiments of the present application provide a model testing method, apparatus, device, and storage medium, which can improve the stability and robustness of the model at the production site.

[0005] To achieve the above-mentioned purpose, a first aspect of an embodiment of the present application proposes a model testing method, comprising: creating multiple local groups for multiple candidate defect types, wherein each of the local groups is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type are different; acquiring multiple target images, inputting the multiple target images into a defect detection model for detection, and obtaining defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types; for each The target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, determines a first target group in each of the local groups corresponding to the target defect type, and divides the target image into the first target group; for each of the candidate defect types, determines a first detection group in each of the local groups corresponding to the candidate defect type, and determines the test result of the target model based on the relationship between the number of target images corresponding to each of the first detection groups and the total number of target images, wherein the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.

[0006] In some embodiments, the test result of the target model is determined based on the relationship between the number of target images corresponding to each of the first detection groups and the total number of target images, including: for each of the candidate defect types, determining a second detection group in each of the local groups corresponding to the candidate defect type, wherein the upper limit of the first confidence interval corresponding to the second detection group is less than a preset second confidence threshold, and the second confidence threshold is less than the first confidence threshold; determining a first effective number based on the difference between the total number of target images and the number of target images in each of the second detection groups; for each of the candidate defect types, determining a first target detection rate of the candidate defect type based on the ratio between the number of target images in each of the first detection groups corresponding to the candidate defect type and the first effective number; comparing the first target detection rate of each of the candidate defect types with the corresponding reference threshold, and determining the test result of the target model based on the comparison result.

[0007] In some embodiments, the target image carries a label bounding box of the defect bounding box, and the first target detection rate of each of the candidate defect types is compared with the corresponding reference threshold, and the test result of the target model is determined based on the comparison result, including: for each of the target images, based on the intersection-and-union ratio between the defect bounding box and the corresponding label bounding box, determining whether the defect detection result of the target image belongs to a correct prediction result or an incorrect prediction result; determining the accuracy rate based on the ratio between the number of correct prediction results and the number of defect detection results; comparing the first target detection rate and the accuracy rate of each of the candidate defect types with the corresponding reference threshold, and determining the test result of the target model based on the comparison result.

[0008] In some embodiments, the first detection groups are each configured with a reference weight, and the reference weight is positively correlated with the lower limit of the first confidence interval configured for the first detection group. For each of the candidate defect types, based on the ratio between the number of target images in each of the first detection groups corresponding to the candidate defect type and the first effective number, the first target detection rate of the candidate defect type is determined, including: for each of the candidate defect types, based on the reference weight, weighted summing the number of target images in each of the first detection groups corresponding to the candidate defect type to obtain the reference detection number corresponding to the candidate defect type; for each of the candidate defect types, based on the ratio between the reference detection number corresponding to the candidate defect type and the first effective number, determining the first target detection rate of the candidate defect type.

[0009] In some embodiments, the test result of the target model is determined based on the relationship between the number of the target images corresponding to each of the first detection groups and the total number of the target images, including: creating multiple global groups, wherein each of the global groups is configured with a second confidence interval, and the second confidence intervals configured for any two of the global groups do not overlap; for each of the target images, based on the second confidence interval and the target confidence of the target defect type corresponding to the target image, determining a second target group in each of the global groups, and dividing the target image into the second target group; determining a third detection group in each of the global groups, wherein the second confidence interval corresponding to the third detection group The lower limit of the confidence interval is greater than or equal to a preset third confidence threshold; a fourth detection group is determined in each of the global groups, wherein the upper limit of the second confidence interval corresponding to the fourth detection group is less than the preset fourth confidence threshold, and the fourth confidence threshold is less than the third confidence threshold; a second effective number is determined based on the difference between the total number of target images and the number of target images corresponding to the fourth detection group; a second target detection rate is determined based on the ratio between the number of target images in the third detection group and the second effective number; the first target detection rate and the second target detection rate are respectively compared with the corresponding reference thresholds, and the test result of the target model is determined based on the comparison results.

[0010] In some embodiments, the second target detection rate and the first target detection rate of each candidate defect type are respectively compared with corresponding reference thresholds, and the test result of the target model is determined based on the comparison results, including: determining the reference detection rate based on the ratio between the second valid number and the total number of the target images; comparing the first target detection rate, the second target detection rate and the reference detection rate with corresponding reference thresholds, and determining the test result of the target model based on the comparison results.

[0011] In some embodiments, for each of the target images, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, a first target group is determined in each of the local groups corresponding to the target defect type, including: for each of the target images, taking the minimum value of the target confidence of the same target defect type in the target image as the reference confidence; based on the first confidence interval and the reference confidence, determining the first target group in each of the local groups corresponding to the target defect type.

[0012] To achieve the above-mentioned purpose, the second aspect of an embodiment of the present application proposes a model testing device, comprising: an initialization module, used to create multiple local groups for multiple candidate defect types, wherein each of the local groups is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type do not overlap; a detection module, used to acquire multiple target images, input the multiple target images into a defect detection model for detection, and obtain defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types; a grouping module A block is used to determine, for each of the target images, a first target group in each of the local groups corresponding to the target defect type based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, and divide the target image into the first target group; a testing module is used to determine, for each of the candidate defect types, a first detection group in each of the local groups corresponding to the candidate defect type, and determine the test result of the target model based on the relationship between the number of the target images corresponding to each of the first detection groups and the total number of the target images, wherein the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.

[0013] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the model testing method described in the first aspect.

[0014] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is executed by a processor, it implements the model testing method described in the first aspect above.

[0015] The embodiments of the present application include at least the following beneficial effects: by creating multiple local groups for multiple candidate defect types, and each local group is configured with a first confidence interval, the first confidence intervals configured for any two local groups corresponding to the same candidate defect type are different, and then multiple target images are obtained, and the multiple target images are respectively input into the defect detection model for detection to obtain defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types, and then for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, a first target group is determined in each local group corresponding to the target defect type, and the target image is divided into the first target group, so that each local group can reflect the number of target images of different target detection types in different first confidence intervals, thereby reflecting the defect detection results. The detection accuracy of the test model for different target detection types is measured, and then for each candidate defect type, the first detection group is determined in each local grouping corresponding to the candidate defect type, and the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to the preset first confidence threshold. The first detection group can reflect the number of target images corresponding to the more accurate target detection type in the defect detection result. Therefore, the first detection group can more effectively reflect the detection accuracy of the defect detection model for different target detection types, and based on the relationship between the number of target images corresponding to each first detection group and the total number of target images, it can better reflect the detection ability of the defect detection model for each target detection type in the actual production site, thereby grasping the actual production environment's demand for the defect detection model, and testing the detection ability of the defect detection model in multiple dimensions, so that the test results of the target model are more comprehensive, which can improve the stability and robustness of the defect detection model in the production site.

[0016] Other features and advantages of the present application will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present application. The purposes and other advantages of the present application can be achieved and obtained through the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings are used to provide a further understanding of the technical solution of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the technical solution of the present application and do not constitute a limitation on the technical solution of the present application.

[0018] Figure 1 An optional flow chart of the model testing method provided in the embodiment of the present application;

[0019] Figure 2 A schematic diagram of an optional process for determining a first target detection rate provided in an embodiment of the present application;

[0020] Figure 3 A schematic diagram of another optional flow chart for determining accuracy provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of another optional flow chart for determining a first target detection rate provided in an embodiment of the present application;

[0022] Figure 5 A schematic diagram of an optional process for determining a second target detection rate provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of an optional process for determining a reference detection rate provided in an embodiment of the present application;

[0024] Figure 7 A schematic diagram of an optional process for confidence grouping provided in an embodiment of the present application;

[0025] Figure 8 A schematic diagram of an optional architecture of the model testing method provided in an embodiment of the present disclosure;

[0026] Figure 9 A schematic diagram of an optional structure of a model testing device provided in an embodiment of the present application;

[0027] Figure 10 A schematic diagram of an optional hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0029] It should be noted that, in each specific embodiment of the present application, when it comes to the need to perform relevant processing based on data related to the characteristics of the target object such as target object attribute information or attribute information set, the permission or consent of the target object will be obtained first, and the collection, use and processing of these data will comply with relevant laws, regulations and standards. Among them, the target object can be a user. In addition, when the embodiment of the present application needs to obtain target object attribute information, the target object's separate permission or separate consent will be obtained by means of a pop-up window or jumping to a confirmation page. After clearly obtaining the target object's separate permission or separate consent, the necessary target object-related data for enabling the normal operation of the embodiment of the present application will be obtained.

[0030] In the description of this application, "several" means one or more, "more" means more than two, "greater than", "less than", "exceed", etc. are understood to exclude the number, and "above", "below", "within", etc. are understood to include the number.

[0031] It should be noted that although the device schematics illustrate functional module divisions and the flowcharts illustrate logical sequences, in certain circumstances, the steps shown or described may be performed in a sequence that differs from the module divisions in the device or the sequence in the flowcharts. The terms "first," "second," and the like in the specification, claims, or accompanying drawings are used to distinguish similar items and are not necessarily used to describe a specific sequence or precedence.

[0032] To facilitate understanding of the technical solutions provided in the embodiments of the present application, some key terms used in the embodiments of the present application are explained here:

[0033] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0034] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0035] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0036] Before a deep learning model is deployed to a production site for defect detection, a series of tests are required to ensure that the model's performance and stability can meet actual production needs.

[0037] Currently, deep learning models are usually tested manually. However, since manual testing may ignore the needs of the actual production environment and the testing efficiency is low, it is impossible to conduct in-depth testing on deep learning models. As a result, deep learning models that have successfully passed the test may not show high stability and robustness in the production site.

[0038] To address the problem of poor performance of deep learning models in production sites, the present application provides a model testing method, apparatus, device and storage medium, the method comprising: creating multiple local groups for multiple candidate defect types, wherein each local group is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type are different; obtaining multiple target images, inputting the multiple target images into a defect detection model for detection, and obtaining defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types; for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, determining a first target group in each local group corresponding to the target defect type, and dividing the target image into the first target group; for each candidate defect type, determining a first detection group in each local group corresponding to the candidate defect type, and determining a test result of the target model based on the relationship between the number of target images corresponding to each first detection group and the total number of target images, wherein the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.According to the solution provided in the embodiment of the present application, multiple local groups are created for multiple candidate defect types, and each local group is configured with a first confidence interval. The first confidence intervals configured for any two local groups corresponding to the same candidate defect type are different. Then, multiple target images are obtained, and the multiple target images are input into the defect detection model for detection to obtain defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types. Then, for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, a first target group is determined in each local group corresponding to the target defect type, and the target image is divided into the first target group, so that each local group can reflect the number of target images of different target detection types in different first confidence intervals, thereby reflecting the defect detection. The model's detection accuracy for different target detection types, then for each candidate defect type, determines the first detection group in each local grouping corresponding to the candidate defect type, and the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to the preset first confidence threshold, the first detection group can reflect the number of target images corresponding to the target detection type with more accurate defect detection results. Therefore, the first detection group can more effectively reflect the detection accuracy of the defect detection model for different target detection types, and based on the relationship between the number of target images corresponding to each first detection group and the total number of target images, it can better reflect the detection ability of the defect detection model for each target detection type in the actual production site, thereby grasping the actual production environment's demand for the defect detection model, and testing the detection ability of the defect detection model in multiple dimensions, so that the test results of the target model are more comprehensive, which can improve the stability and robustness of the defect detection model in the production site.

[0039] The model testing method, apparatus, device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the model testing method in the embodiments of the present application is described.

[0040] The model testing method provided in the embodiment of the present application relates to the field of computer technology. The model testing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or as a server cluster or distributed system composed of multiple physical servers, or as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the model testing method, etc., but is not limited to the above forms.

[0041] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0042] The embodiments of the present application are further described below with reference to the accompanying drawings.

[0043] like Figure 1 As shown, Figure 1 An optional flow chart of a model testing method provided in an embodiment of the present application is provided. The model testing method can be executed by a server, or by a terminal, or by a server in conjunction with a terminal. The model testing method includes but is not limited to the following steps S110 to S140:

[0044] Step S110 : creating multiple local groups for each of the multiple candidate defect types.

[0045] Each local group is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type do not overlap.

[0046] The first confidence intervals corresponding to the local groups of the candidate defect types may be the same.

[0047] In one possible implementation, for candidate defect types whose confidence levels are more than half distributed in a lower first confidence interval, the lower limit of each first confidence interval can be adjusted lower; for candidate defect types whose confidence levels are more than half distributed in a higher first confidence interval, the lower limit of each first confidence interval can be adjusted higher.

[0048] In step S120 , a plurality of target images are acquired, and the plurality of target images are respectively input into a defect detection model for detection to obtain defect detection results for each target image.

[0049] Among them, the defect detection results include at least one defect bounding box and the target defect type of the defect bounding box. The target defect type is one of multiple candidate defect types. For example, the target image is an image of the aluminum shell surface, and the candidate defect types are pits, damage, bumps, minor pits and minor damage.

[0050] The defect detection results may also include the bounding box confidence level corresponding to the defect bounding box. The defect detection model may be a deep learning model that can sequentially output defect detection results for each target image or simultaneously infer each target image to obtain defect detection results for each target image. This is not limited in the present embodiment.

[0051] Among them, the defect bounding box is used to indicate the location of the defect in the target image. A coordinate system can be established on the target image, the position of the center point of the defect bounding box is indicated by the coordinate point, and the size of the target image is indicated by the length and width.

[0052] Among them, before inputting the defect detection model, the sizes of each target image are different, that is, the length, width and number of channels of each target image are different. You can choose to cut each target image first, and then uniformly scale the target image to the preset inference size of the defect detection model, or directly scale the target image to the inference size, depending on the preprocessing strategy of model training. Then input the scaled target image into the defect detection model. For example, the target image needs to be scaled and completed in two steps to be consistent with the inference size. Assume that the inference size of the defect detection model is 640px wide and 640px high, and the size of the target image is 960px wide and 480px high. First, divide the width of the inference size by the width of the target image to get a=640px÷960px, and divide the height of the inference size by the height of the target image to get b=640px÷480px. Select the smaller value of a and b, here is a, and then multiply the size of the target image by a to get the scaled target size, that is, 640px wide and 320px high. Scale the target image to the target size, and then fill the part of the target image size that is less than the inference size with black pixels to make it consistent with the inference size, that is, 640px wide and 640px high.

[0053] Step S130 : for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, determine a first target group in each local group corresponding to the target defect type, and divide the target image into the first target group.

[0054] When the number of defect bounding boxes corresponding to a candidate defect type in the target image is 0, the target image is divided into a first target group corresponding to a first confidence interval with the lowest lower limit.

[0055] Step S140: For each candidate defect type, determine a first detection group in each local group corresponding to the candidate defect type, and determine the test result of the target model based on the relationship between the number of target images corresponding to each first detection group and the total number of target images.

[0056] The lower limit of the first confidence interval corresponding to the first detected group is greater than or equal to a preset first confidence threshold.

[0057] The number of first detection groups corresponding to each candidate defect type may be one or more. When the number of first detection groups is more than one, the first confidence intervals corresponding to each first detection group may be continuous.

[0058] Among them, the first confidence intervals corresponding to each local grouping can be continuous and non-overlapping. For each candidate defect type, the confidence corresponding to the first detection group is higher than that of other local groupings. For example, 6 local groups are created for each candidate defect type, corresponding to 6 first confidence intervals, and the lower limits of each first confidence interval are 0, 0.5, 0.6, 0.7, 0.8 and 0.9 respectively. The lower limit of the first confidence interval corresponding to the latter local grouping is the upper limit of the first confidence interval corresponding to the previous local grouping, and the upper limit of the first confidence interval corresponding to the last local grouping is 1. The first confidence threshold can be 0.7.

[0059] Based on this, multiple local groups are created for multiple candidate defect types, and each local group is configured with a first confidence interval. The first confidence intervals configured for any two local groups corresponding to the same candidate defect type are different. Then, multiple target images are obtained, and the multiple target images are input into the defect detection model for detection to obtain defect detection results for each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types. Then, for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, the first target group is determined in each local group corresponding to the target defect type, and the target image is divided into the first target group, so that each local group can reflect the number of target images of different target detection types in different first confidence intervals, thereby reflecting the defect detection model for different The detection accuracy of the same target detection type, then for each candidate defect type, determine the first detection group in each local grouping corresponding to the candidate defect type, and the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to the preset first confidence threshold, the first detection group can reflect the number of target images corresponding to the target detection type with more accurate defect detection results. Therefore, the first detection group can more effectively reflect the detection accuracy of the defect detection model for different target detection types, and based on the relationship between the number of target images corresponding to each first detection group and the total number of target images, it can better reflect the detection ability of the defect detection model for each target detection type in the actual production site, thereby grasping the actual production environment's demand for the defect detection model, and testing the detection ability of the defect detection model in multiple dimensions, so that the test results of the target model are more comprehensive, which can improve the stability and robustness of the defect detection model in the production site.

[0060] In addition, refer to Figure 2In one embodiment, determining the test result of the target model based on the relationship between the number of target images corresponding to each first detection group and the total number of target images includes but is not limited to the following steps:

[0061] Step S210 : For each candidate defect type, determine a second detected group in each local group corresponding to the candidate defect type.

[0062] Step S220 : determining a first effective number based on a difference between the total number of target images and the number of target images in each second detected group.

[0063] Step S230 : For each candidate defect type, determine a first target detection rate of the candidate defect type based on a ratio between the number of target images in each first detection group corresponding to the candidate defect type and the first valid number.

[0064] In step S240 , the first target detection rate of each candidate defect type is compared with the corresponding reference threshold, and a test result of the target model is determined based on the comparison result.

[0065] The upper limit of the first confidence interval corresponding to the second detected group is smaller than a preset second confidence threshold, and the second confidence threshold is smaller than the first confidence threshold.

[0066] Among them, the first confidence interval whose upper limit is less than the second confidence threshold is determined as an invalid confidence interval. For each candidate defect type, the target confidence of the target image corresponding to the invalid confidence interval is too low, indicating that the defect detection model has missed the detection or the target image does not have a defect bounding box corresponding to the candidate defect type; the first confidence interval whose lower limit is greater than or equal to the second confidence threshold is determined as a valid confidence interval, and the first valid number is the sum of the number of target images corresponding to the valid confidence intervals.

[0067] The first target detection rate of each candidate defect type can be expressed by a formula. The formula of the first target detection rate is as follows:

[0068]

[0069] For each candidate defect type, D1 is the first target detection rate of the candidate defect type, n1 is the sum of the number of target images in each first detection group of the candidate defect type, and M1 is the first effective number corresponding to the candidate defect type.

[0070] The first target detection rate is used to indicate the proportion of target images with higher target confidence for the target defect type. Therefore, the closer the first target detection rate is to 1, the higher the detection accuracy of the defect detection model.

[0071] The reference threshold corresponding to the first detection rate may be obtained through multiple field experiments and may be obtained through testing with data from a test set, and is not limited in the present embodiment.

[0072] Among them, the test results of the target model include the first test results and the second test results. The first test result is used to indicate that the test of the defect detection model has passed and the defect detection model can be put online. The second test result is used to indicate that the test of the defect detection model has failed and it will not be put online. The defect detection model needs to be optimized. For example, the first target detection rate is 99.88%, and the reference threshold corresponding to the first target detection rate is 96%. Since the first target detection rate is greater than the corresponding reference threshold, the test of the defect detection model has passed and the defect detection model can be put online.

[0073] Based on this, for each candidate defect type, a second detection group is determined in each local group corresponding to the candidate defect type. Since the upper limit of the first confidence interval corresponding to the second detection group is less than a preset second confidence threshold, and the second confidence threshold is less than the first confidence threshold, the target image corresponding to the second detection group is invalid for determining the test result. Then, based on the difference between the total number of target images and the number of target images in each second detection group, a first valid number is determined, and the target images corresponding to the second detection group are excluded from the test result determination process, thereby improving the reliability of the test result. For each candidate defect type, based on the ratio between the number of target images in each first detection group corresponding to the candidate defect type and the first valid number, a first target detection rate of the candidate defect type is determined. Then, the first target detection rate of each candidate defect type is compared with the corresponding reference threshold. Based on the comparison result, the test result of the target model is determined. This allows the detection accuracy of the defect detection model for each candidate defect type to be determined. The defect detection model can be tested in multiple dimensions to determine the detection capability of the defect detection model. This can improve the accuracy of the model test results and improve the stability and robustness of the defect detection model in the production site.

[0074] In addition, refer to Figure 3 In one embodiment, the target image carries a label bounding box of a defect bounding box, and the first target detection rate of each candidate defect type is compared with the corresponding reference threshold. The test result of the target model is determined based on the comparison result, including but not limited to the following steps:

[0075] In step S310 , for each target image, based on the intersection-over-union ratio between the defect bounding box and the corresponding label bounding box, it is determined whether the defect detection result of the target image is a correct prediction result or an incorrect prediction result.

[0076] In step S320 , the accuracy rate is determined based on the ratio between the number of correct prediction results and the number of defect detection results.

[0077] In step S330 , the first target detection rate and accuracy of each candidate defect type are compared with the corresponding reference thresholds, and the test result of the target model is determined based on the comparison results.

[0078] Among them, the label bounding box is obtained through manual annotation. Like the defect bounding box, it can describe the position of the defect in the target image through the center point coordinates and the size of the defect in the target image through the length and width.

[0079] It should be noted that the intersection-in-union (IoU) is used to indicate the accuracy of the defect bounding box of the defect detection result. When the defect bounding box does not overlap with the label bounding box, the IoU is 0. When the defect bounding box overlaps with the label bounding box, the IoU is calculated as follows: assuming that the area corresponding to the overlapping part of the defect bounding box and the label bounding box is area A, the difference between the total area of ​​the defect bounding box and area A is area B, the difference between the total area of ​​the label bounding box and area A is area C, and the sum of area A, area B and area C is the total interaction area. The quotient of area A and the total interaction area is determined as the intersection-in-union (IoU) ratio.

[0080] Among them, the target image also carries a label defect type of the target defect type. When the target defect type is the same as the label defect type and the intersection-and-union ratio is greater than or equal to the reference intersection-and-union ratio threshold, the defect detection result is a correct prediction result; when the target defect type is the same as the label defect type and the intersection-and-union ratio is less than the reference intersection-and-union ratio threshold, the defect detection result is an incorrect prediction result; when the target defect type is different from the label defect type, the defect detection result is an incorrect prediction result. For example, when the reference intersection-and-union ratio threshold is 0.5, when the target defect type is the same as the label defect type and the intersection-and-union ratio is 0.7, the defect detection result is a correct prediction result.

[0081] The sum of the number of defect detection results of all target images and the number of non-overlapping label bounding boxes of all target images is determined as the number of bounding boxes, and the quotient of the number of correct prediction results of all target images and the number of bounding boxes is determined as the accuracy rate.

[0082] The reference threshold corresponding to the first target detection rate and the reference threshold corresponding to the accuracy rate of each candidate defect type can be determined through multiple experiments, and the embodiments of the present disclosure are not limited thereto.

[0083] Based on this, accuracy is another indicator for determining the test results, which can increase the diversity of test indicators. In addition, since the accuracy is determined based on the intersection-over-union ratio between the defect bounding box and the corresponding label bounding box, the accuracy has a higher certainty and can increase the accuracy of the test results based on the first target detection rate.

[0084] In addition, refer to Figure 4 In one embodiment, each first detection group is configured with a reference weight, and the reference weight is positively correlated with the lower limit of the first confidence interval configured for the first detection group. For each candidate defect type, based on the ratio between the number of target images in each first detection group corresponding to the candidate defect type and the first valid number, a first target detection rate of the candidate defect type is determined, including but not limited to the following steps:

[0085] In step S410 , for each candidate defect type, based on a reference weight, weighted sum is performed on the number of target images in each first detection group corresponding to the candidate defect type to obtain a reference detection number corresponding to the candidate defect type.

[0086] Step S420 : For each candidate defect type, a first target detection rate of the candidate defect type is determined based on a ratio between a reference detection quantity corresponding to the candidate defect type and a first effective quantity.

[0087] Among them, the product of the number of target images in the first detection group and the reference weight of the first detection group is determined as the weighted reference number of the first detection group, and the sum of the weighted reference numbers of each first detection group is determined as the reference detection number of the first detection group. For example, the number of first detection groups is 3, and the corresponding reference weights are 0.2, 0.3 and 0.5 respectively. The corresponding total number of target images is 7, 77 and 2507 respectively. Therefore, the reference detection number is 0.2×7+0.3×77+0.5×2507=1278. Further, assuming that the first effective number is 2594, the first target detection rate is 1278÷2594=49.26%.

[0088] Based on this, for each candidate defect type, based on the reference weight, the number of target images in each first detection group corresponding to the candidate defect type is weighted and summed to obtain the reference detection number corresponding to the candidate defect type. Then, for each candidate defect type, based on the ratio between the reference detection number corresponding to the candidate defect type and the first effective number, the first target detection rate of the candidate defect type is determined. This can differentiate the influence of defect detection results corresponding to different target confidence levels on the test results, avoid affecting the overall judgment due to low-quality detection results, and improve the accuracy and robustness of the test results.

[0089] In addition, refer to Figure 5 In one embodiment, determining the test result of the target model based on the relationship between the number of target images corresponding to each first detection group and the total number of target images includes but is not limited to the following steps:

[0090] Step S510: Create multiple global groups.

[0091] Step S520 : For each target image, based on the second confidence interval and the target confidence of the target defect type corresponding to the target image, determine a second target group in each global group, and divide the target image into the second target group.

[0092] Step S530: Determine a third detected group in each global group.

[0093] Step S540: Determine a fourth detected group in each global group.

[0094] Step S550 : determining a second valid number based on a difference between the total number of target images and the number of target images corresponding to the fourth detected group.

[0095] Step S560: Determine a second target detection rate based on a ratio between the number of target images in the third detection group and the second effective number.

[0096] In step S570 , the first target detection rate and the second target detection rate are respectively compared with corresponding reference thresholds, and a test result of the target model is determined based on the comparison results.

[0097] Each global group is configured with a second confidence interval, and the second confidence intervals configured for any two global groups do not overlap.

[0098] The lower limit of the second confidence interval corresponding to the third detection group is greater than or equal to a preset third confidence threshold.

[0099] The upper limit of the second confidence interval corresponding to the fourth detected group is smaller than a preset fourth confidence threshold, and the fourth confidence threshold is smaller than the third confidence threshold.

[0100] Among them, the upper limit of the second confidence interval can be the same as the upper limit of the first confidence interval, the lower limit of the second confidence interval can be the same as the lower limit of the first confidence interval, the third confidence threshold can be the same as the first confidence threshold, and the fourth confidence threshold can be the same as the second confidence threshold.

[0101] When the number of defect bounding boxes in the target image is 0, the target image is divided into the second target group corresponding to the second confidence interval with the lowest lower limit.

[0102] It can be understood that the process of determining the second target detection rate is consistent with the process of determining the first target detection rate, that is, the global grouping corresponds to the local grouping of a single candidate defect type, the second target grouping corresponds to the first target grouping, the third detection grouping corresponds to the first detection grouping of a single candidate defect type, and the fourth detection grouping corresponds to the second detection grouping of a single candidate defect type.

[0103] The second target detection rate can be expressed by a formula, and the formula for the second target detection rate is as follows:

[0104]

[0105] Wherein, D2 is the second target detection rate, n2 is the sum of the number of target images in each third detection group, and M2 is the second effective number.

[0106] Based on this, the defect detection results of all target images are grouped as a whole, so that the determined second target detection rate can reflect the overall detection capability of the defect detection model. Combined with the first target detection rate that can reflect the local detection capability of the defect detection model for each candidate defect type, the detection capability of the defect detection model in the actual production site can be analyzed at the overall and local levels, making the test results of the target model more accurate, and improving the stability and robustness of the defect detection model that has passed the test in the production site.

[0107] In addition, refer to Figure 6 In one embodiment, the second target detection rate and the first target detection rate of each candidate defect type are respectively compared with the corresponding reference threshold value, and the test result of the target model is determined based on the comparison results, including but not limited to the following steps:

[0108] Step S610: determining a reference detection rate based on a ratio between the second effective number and the total number of target images.

[0109] In step S620 , the first target detection rate, the second target detection rate, and the reference detection rate are respectively compared with corresponding reference thresholds, and a test result of the target model is determined based on the comparison results.

[0110] The second effective number is the number of target images that can be used to determine the test result of the model, and the ratio between the second effective number and the total number of target images is determined as the reference detection rate.

[0111] The first target detection rate of each candidate defect type can be expressed by a formula. The formula of the first target detection rate is as follows:

[0112]

[0113] Where P is the reference detection rate, M2 is the second effective number, and M is the total number of target images.

[0114] Based on this, the reference detection rate reflects the proportion of target images that can be used to determine the test results of the model. The reference detection rate is positively correlated with the probability of the model passing the test. Therefore, the reference detection rate is an important indicator for determining the test results of the model. Combined with the first target detection rate and the second target detection rate, it can improve the accuracy of the model's test results and improve the stability and robustness of the defect detection model that has passed the test in the production site.

[0115] In addition, refer to Figure 7 In one embodiment, for each target image, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, determining a first target group in each local group corresponding to the target defect type includes but is not limited to the following steps:

[0116] Step S710 : For each target image, the minimum value of the target confidence of the same target defect type in the target image is used as a reference confidence.

[0117] Step S720 : determining a first target group in each local group corresponding to the target defect type based on the first confidence interval and the reference confidence.

[0118] The reference confidence level is compared with each first confidence interval corresponding to the target defect type to determine the target confidence interval, and the local group corresponding to the target confidence interval is determined as the first target group.

[0119] It should be noted that the second target group may be determined in each local group corresponding to the target defect type based on the second confidence interval and the minimum value of the reference confidence of each target defect type.

[0120] For example, the target image is input into the defect detection model for detection, and 6 defect detection results are obtained, corresponding to 3 target defect types, defect 1, defect 2, and defect 3, which are defect 1A, defect 1B, defect 2A, defect 2B, defect 3A, and defect 3B respectively. The target confidence corresponding to defect 1A is 0.67, the target confidence corresponding to defect 1B is 0.71, the target confidence corresponding to defect 2A is 0.75, the target confidence corresponding to defect 2B is 0.81, the target confidence corresponding to defect 3A is 0.55, the target confidence corresponding to defect 3B is 0.98, and the reference confidence corresponding to defect 1 is 0. The reference confidence level for defect 1 is 0.67, the reference confidence level for defect 2 is 0.75, and the reference confidence level for defect 3 is 0.55. Moreover, the lower limits of the first confidence intervals are 0, 0.5, 0.6, 0.7, 0.8, and 0.9, respectively. Therefore, the lower limit of the first confidence interval for defect 1 is 0.6 and the upper limit is 0.7, the lower limit of the first confidence interval for defect 2 is 0.7 and the upper limit is 0.8, and the lower limit of the first confidence interval for defect 1 is 0.5 and the upper limit is 0.6. The local groups corresponding to the first confidence intervals are determined as the first target groups corresponding to the target defect type.

[0121] Based on this, the reference confidence is the minimum target confidence of the same target defect type in the target image, which can reflect the worst detection results of each target defect type in the target image, so that the determined first detection rate, second detection rate and reference detection rate are determined by the defect detection model under the worst case scenario. In fact, the defect detection model that successfully passes the test has better performance than the test results. Even if an unexpected situation occurs at the production site, the performance of the defect detection model will not differ too much from the test results, which can improve the stability and robustness of the defect detection model at the production site.

[0122] For example, the result analysis report is shown in Table 1 below:

[0123] Table 1

[0124]

[0125]

[0126] The complete process of the model testing method is described in detail below.

[0127] Reference Figure 8 , Figure 8 A schematic diagram of an optional architecture of the model testing method provided in an embodiment of the present disclosure.

[0128] First, multiple target images for testing the defect detection model and the defect detection model to be tested are loaded into the model inference module of the test system;

[0129] Then, the inference parameters of the defect detection model are set. The specific inference parameters are the cut image size, each target confidence interval, and each candidate defect type. Referring to Table 1, the target confidence interval corresponding to "<0.5" is [0, 0.5), the target confidence interval corresponding to "≥0.5" is [0.5, 0.6), the target confidence interval corresponding to "≥0.6" is [0.6, 0.7), the target confidence interval corresponding to "≥0.7" is [0.7, 0.8), the target confidence interval corresponding to "≥0.8" is [0.8, 0.9), and the target confidence interval corresponding to "≥0.9" is [0.9, 1). The candidate defect types correspond to Table 1, and the candidate defect types are pits, damage, gouges, minor damage, and minor damage.

[0130] Each target image is then cropped and scaled to the inference size or directly scaled to the inference size. The preprocessed target images are then input into the defect detection model for detection to obtain defect detection results for each target image. The defect detection results include at least one defect bounding box and a target defect type of the defect bounding box. The target defect type is one of multiple candidate defect types. The defect detection model filters out defect bounding boxes corresponding to target defect types whose target confidence lies in the interval [0, 0.5) in the defect detection results of each target image.

[0131] Then the test system performs inference on the image to obtain the result analysis report, defect detection result map and annotation file, and inputs the result analysis report, defect detection result map and annotation file into the model evaluation module. Among them, the result analysis report corresponds to Table 1, the defect detection result map is the defect detection result output by the defect detection model, and the annotation file is used to pre-annotate the target image;

[0132] Then, the result analysis report is analyzed to obtain the reference detection rate and effective detection rate in Table 1, where the effective detection rate includes the first target detection rate and the second target detection rate. For example, the first target detection rate corresponding to the candidate defect type "pit" is 99.88%, the first target detection rate corresponding to the candidate defect type "damage" is 99.92%, the first target detection rate corresponding to the candidate defect type "knock" is 99.73%, the first target detection rate corresponding to the candidate defect type "minor injury" is 99.61%, the first target detection rate corresponding to the candidate defect type "minor damage" is 99.14%, the second target detection rate is 99.75%, and the reference detection rate is 99.963%;

[0133] Finally, the defect detection result image is compared with the defect detection label carried by the target image to determine the accuracy of the defect detection model. The defect detection label includes the label bounding box and the label defect type. When the first condition is met, the test of the defect detection model passes and the defect detection model can be put online; when the first condition is not met, the test of the defect detection model fails and is not put online. The defect detection model needs to be optimized. Among them, the first condition is that the first target detection rate, the second target detection rate, the reference detection rate and the accuracy rate are all greater than the corresponding reference threshold.

[0134] In addition, the target image can be pre-annotated through the annotation file. Before the defect detection model is tested, each target image is input into the defect detection model for detection to obtain a pre-annotated image. The defect pre-annotation result is drawn on the pre-annotated image. The defect pre-annotation result includes the defect bounding box and the target defect type corresponding to the defect bounding box. The annotator can significantly improve the annotation efficiency of the target image by modifying the defect pre-annotation result on the pre-annotated image instead of annotating the target image entirely manually, which is conducive to improving the efficiency of preparing test data.

[0135] In addition, reference Figure 9 , the present application also provides a model testing device 900, comprising:

[0136] Initialization module 910 is configured to create multiple local groups for multiple candidate defect types, wherein each local group is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type do not overlap;

[0137] A detection module 920 is configured to acquire multiple target images, input each of the multiple target images into a defect detection model for detection, and obtain a defect detection result for each target image, wherein the defect detection result includes at least one defect bounding box and a target defect type of the defect bounding box, where the target defect type is one of multiple candidate defect types;

[0138] A grouping module 930 is configured to determine, for each target image, a first target group in each local group corresponding to the target defect type based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, and divide the target image into the first target group;

[0139] Testing module 940 is used to determine, for each candidate defect type, a first detection group in each local group corresponding to the candidate defect type, and determine a test result of the target model based on the relationship between the number of target images corresponding to each first detection group and the total number of target images, wherein the lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.

[0140] It can be understood that the specific implementation of the model testing device 900 is basically the same as the specific embodiment of the above-mentioned model testing method, and will not be repeated here.

[0141] In addition, refer to Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0142] The processor 1001 can be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application.

[0143] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called by the processor 1001 to execute the model testing method of the embodiments of this application.

[0144] Input / output interface 1003, used to implement information input and output;

[0145] Communication interface 1004, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0146] Bus 1005 , which transmits information between various components of the device (e.g., processor 1001 , memory 1002 , input / output interface 1003 , and communication interface 1004 );

[0147] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0148] An embodiment of the present application also provides a storage medium, which is a computer-readable storage medium used for computer-readable storage. The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the above-mentioned model testing method.

[0149] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely arranged relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0150] The embodiments described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided by the embodiments of this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0151] It will be understood by those skilled in the art that Figures 1 to 10 The technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figures, or a combination of certain steps, or different steps.

[0152] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, i.e., they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0153] Those skilled in the art will appreciate that all or some of the steps in the methods, systems, and functional modules / units in the devices disclosed above may be implemented as software, firmware, hardware, or appropriate combinations thereof.

[0154] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0155] It should be understood that in this application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0156] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the above-mentioned units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0157] The units described above as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0158] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0159] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes multiple instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: various media that can store programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0160] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but are not intended to limit the scope of the present invention. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and essence of the present invention should be within the scope of the present invention.

Claims

1. A model testing method, characterized in that: include: Creating a plurality of local groups for a plurality of candidate defect types respectively, wherein each of the local groups is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type do not overlap; Acquire multiple target images, input the multiple target images into the defect detection model for detection, and obtain defect detection results of each target image, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the multiple candidate defect types; For each of the target images, based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, determine a first target group in each of the local groups corresponding to the target defect type, and divide the target image into the first target group; For each of the candidate defect types, a first detection group is determined in each of the local groups corresponding to the candidate defect type, and based on the relationship between the number of target images corresponding to each of the first detection groups and the total number of target images, a test result of the target model is determined, wherein a lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.

2. The model testing method according to claim 1, characterized in that: The determining the test result of the target model based on the relationship between the number of the target images corresponding to each of the first detection groups and the total number of the target images includes: For each of the candidate defect types, determining a second detection group in each of the local groups corresponding to the candidate defect type, wherein an upper limit of the first confidence interval corresponding to the second detection group is less than a preset second confidence threshold, and the second confidence threshold is less than the first confidence threshold; determining a first effective number based on a difference between the total number of the target images and the number of the target images in each of the second detected groups; For each of the candidate defect types, determining a first target detection rate of the candidate defect type based on a ratio between the number of the target images in each of the first detection groups corresponding to the candidate defect type and the first valid number; The first target detection rate of each of the candidate defect types is compared with a corresponding reference threshold, and a test result of the target model is determined based on the comparison result.

3. The model testing method according to claim 2, characterized in that: The target image carries a label bounding box of the defect bounding box, and the first target detection rate of each of the candidate defect types is compared with a corresponding reference threshold, and a test result of the target model is determined based on the comparison result, including: For each of the target images, determining whether the defect detection result of the target image is a correct prediction result or an incorrect prediction result based on an intersection-over-union ratio between the defect bounding box and the corresponding label bounding box; determining an accuracy rate based on a ratio between the number of the correct prediction results and the number of the defect detection results; The first target detection rate and the accuracy rate of each of the candidate defect types are respectively compared with corresponding reference thresholds, and a test result of the target model is determined based on the comparison results.

4. The model testing method according to claim 2, characterized in that: The first detection groups are each configured with a reference weight, the reference weight is positively correlated with a lower limit of the first confidence interval configured for the first detection group, and for each of the candidate defect types, based on a ratio between the number of the target images in each of the first detection groups corresponding to the candidate defect type and the first valid number, determining a first target detection rate of the candidate defect type, including: For each of the candidate defect types, based on the reference weight, weighted summing is performed on the number of the target images in each of the first detection groups corresponding to the candidate defect type to obtain a reference detection number corresponding to the candidate defect type; For each of the candidate defect types, a first target detection rate of the candidate defect type is determined based on a ratio between the reference detection quantity corresponding to the candidate defect type and the first effective quantity.

5. The model testing method according to claim 2, characterized in that: The determining the test result of the target model based on the relationship between the number of the target images corresponding to each of the first detection groups and the total number of the target images includes: Creating a plurality of global groups, wherein each of the global groups is configured with a second confidence interval, and the second confidence intervals configured for any two of the global groups do not overlap; For each of the target images, based on the second confidence interval and the target confidence of the target defect type corresponding to the target image, determine a second target group in each of the global groups, and divide the target image into the second target group; Determine a third detected group in each of the global groups, wherein a lower limit of the second confidence interval corresponding to the third detected group is greater than or equal to a preset third confidence threshold; Determine a fourth detected group in each of the global groups, wherein an upper limit of the second confidence interval corresponding to the fourth detected group is less than a preset fourth confidence threshold, and the fourth confidence threshold is less than the third confidence threshold; determining a second effective number based on a difference between the total number of the target images and the number of the target images corresponding to the fourth detected group; determining a second target detection rate based on a ratio between the number of the target images in the third detection group and the second effective number; The first target detection rate and the second target detection rate are respectively compared with corresponding reference thresholds, and a test result of the target model is determined based on the comparison results.

6. The model testing method according to claim 5, characterized in that: The step of comparing the second target detection rate and the first target detection rate of each of the candidate defect types with corresponding reference thresholds, and determining the test result of the target model based on the comparison result, includes: determining a reference detection rate based on a ratio between the second effective number and the total number of the target images; The first target detection rate, the second target detection rate and the reference detection rate are respectively compared with corresponding reference thresholds, and a test result of the target model is determined based on the comparison results.

7. The model testing method according to claim 1, characterized in that: The step of determining, for each of the target images, a first target group in each of the local groups corresponding to the target defect type based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, comprises: For each of the target images, taking the minimum value of the target confidence of the same target defect type in the target image as the reference confidence; Based on the first confidence interval and the reference confidence, a first target group is determined in each of the local groups corresponding to the target defect type.

8. A model testing device, characterized in that: include: An initialization module, used to create a plurality of local groups for a plurality of candidate defect types, respectively, wherein each of the local groups is configured with a first confidence interval, and the first confidence intervals configured for any two local groups corresponding to the same candidate defect type do not overlap; a detection module, used to acquire a plurality of target images, input the plurality of target images into a defect detection model for detection, and obtain defect detection results of the target images, wherein the defect detection results include at least one defect bounding box and a target defect type of the defect bounding box, and the target defect type is one of the plurality of candidate defect types; A grouping module, configured to determine, for each of the target images, a first target group in each of the local groups corresponding to the target defect type based on the first confidence interval and the target confidence of the target defect type corresponding to the target image, and divide the target image into the first target group; A testing module is used to determine, for each of the candidate defect types, a first detection group in each of the local groups corresponding to the candidate defect type, and determine a test result of the target model based on the relationship between the number of target images corresponding to each of the first detection groups and the total number of target images, wherein a lower limit of the first confidence interval corresponding to the first detection group is greater than or equal to a preset first confidence threshold.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the model testing method according to any one of claims 1 to 7 when executing the computer program.

10. A storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the model testing method according to any one of claims 1 to 7 is implemented.

Images