Model defect detection method, computer device and readable storage medium

Abstract

The present application discloses a model defect detection method, a computer device and a readable storage medium, and relates to the field of model testing technology. The method disclosed in the present application includes: obtaining a synthetic image data set, wherein the data in the synthetic image data set is image data with annotations; verifying the model to be detected based on the synthetic image data set, and determining the first defect sample data in the synthetic image data set based on the verification result; obtaining a real image data set, wherein the data in the real image data set is unlabeled image data; obtaining the second defect sample data based on the first defect sample data and the real image data set; and detecting whether the model to be detected has model defects based on the second defect sample data. The present application achieves the purpose of quickly detecting model defects at very low annotation costs, and effectively improves the model defect detection coverage and model safety verification efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of model testing, and in particular to a model defect detection method, a computer device and a readable storage medium. Background Art

[0002] In recent years, with the rapid development of deep learning technology, models based on deep learning, such as object detection models, have become an important research hotspot in the field of computer vision. In practical applications, after the model is trained, it is usually necessary to evaluate and test the performance of the model to determine the success of the trained model and the defects of the model.

[0003] In the prior art, the following methods are usually used to detect model defects: (1) Using the uncertainty of the model to discover model defects, and then using the model of the same task to measure the uncertainty. Since the correlation between the size of the uncertainty and whether it is a model defect is related to the training data of the model itself, the model defects obtained are often inaccurate; (2) Using multiple models to verify each other to discover model defects. No manual labeling is required, but it will be affected by the distribution of the model training data itself, and sometimes systematic deviations will occur, and it cannot be guaranteed that all defects will be discovered; (3) Using a real database to refresh the library to obtain model defects. Since a large amount of real data is required, and this data needs to be accurately labeled, it will consume a lot of computing resources.

[0004] Therefore, existing model defect detection methods have the problems of incomplete model defect mining and high data annotation costs. Summary of the invention

[0005] In order to overcome the above-mentioned defects, the model defect detection method, computer equipment and readable storage medium provided by the present invention can solve the problems in the prior art that model defect detection by annotating training data requires a lot of time and cost, the efficiency of model defect detection is low, and model defect detection through model verification without data annotation has incomplete model defect mining.

[0006] In a first aspect, the present invention provides a model defect detection method, the method comprising:

[0007] Acquire a synthetic image data set, wherein the data in the synthetic image data set is image data with annotations;

[0008] Verifying the model to be detected based on the synthetic image data set, and determining first defect sample data in the synthetic image data set based on the verification result;

[0009] Acquire a real image data set, where data in the real image data set is unlabeled image data;

[0010] Obtaining second defect sample data based on the first defect sample data and the real image data set;

[0011] Based on the second defect sample data, the model to be inspected is detected whether there is a model defect.

[0012] In some implementations, acquiring the synthetic image dataset includes:

[0013] A scene simulation image is generated by using simulation software, and data generalization processing is performed on the scene simulation image to obtain the synthetic image data set.

[0014] In some implementations, the verifying the model to be detected based on the synthetic image data set and determining the first defect sample data in the synthetic image data set based on the verification result includes:

[0015] The model to be detected is verified based on the synthetic image data set to obtain a model evaluation index, and the first defect sample data in the synthetic image data set is determined according to the model evaluation index.

[0016] Further, verifying the model to be detected based on the synthetic image data set to obtain a model evaluation index, and determining the first defect sample data in the synthetic image data set according to the model evaluation index includes:

[0017] Using the image data in the synthetic image data set as input of the model to be detected, and performing target prediction on the image data based on the model to be detected;

[0018] A model evaluation index is determined based on a target prediction result output by the model, and image data in the synthetic image data set corresponding to when the model evaluation index is lower than a preset threshold is selected as the first defect sample data.

[0019] Furthermore, the annotation information carried by the image data in the synthetic image data set includes at least target coordinates and scene labels.

[0020] In some implementations, obtaining second defect sample data based on the first defect sample data and the real image data set includes:

[0021] Based on the first defect sample data, sample data to be labeled is obtained from the real image data set, and the sample data to be labeled is labeled to obtain the second defect sample data.

[0022] Furthermore, the acquiring the sample data to be labeled from the real image data set based on the first defect sample data includes:

[0023] A similar graph retrieval technology is used to select image data similar to the first defect sample data from the real image data set as the sample data to be labeled.

[0024] In some implementations, detecting whether the model to be detected has a model defect based on the second defect sample data includes:

[0025] The second defect sample data is used as the input of the model to be detected, and a target prediction is performed on the second defect sample data based on the model to be detected. A model evaluation index is determined based on the target prediction result output by the model. When the model evaluation index is lower than a preset threshold, it is determined that the model to be detected has a model defect.

[0026] In a second aspect, the present invention provides a computer device comprising a processor and a memory, wherein the memory is suitable for storing multiple program codes, and the program codes are suitable for being loaded and run by the processor to execute the model defect detection method described in any one of the technical solutions of the above-mentioned model defect detection method.

[0027] In a third aspect, the present invention provides a computer-readable storage medium storing a plurality of program codes, wherein the program codes are suitable for being loaded and run by a processor to execute the model defect detection method described in any one of the technical solutions of the above-mentioned model defect detection method.

[0028] The above one or more technical solutions of the present invention have at least one or more of the following beneficial effects: the present application utilizes the data in the synthetic image data set as annotated image data, verifies the model to be detected based on the synthetic image data set to determine the first defect sample data in the synthetic image data set, then obtains the second defect sample data based on the real image data set and the first defect sample data, and finally detects whether the model to be detected has model defects based on the second defect sample data. The present application only needs to annotate a small amount of data obtained from the real image data set to obtain the second defect sample data, thereby achieving the purpose of quickly detecting model defects at very low annotation costs. In addition, based on the characteristic that the data in the synthetic image data set can be randomly generalized, the present application enhances the coverage of the image data in the synthetic image data set for the real scene images, which can effectively improve the model defect detection coverage, and thus effectively improve the model safety verification efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The disclosure of the present invention will become more easily understood with reference to the accompanying drawings. It is easy for those skilled in the art to understand that these drawings are only for illustrative purposes and are not intended to limit the scope of protection of the present invention. In addition, similar numbers in the figures are used to represent similar components, among which:

[0030] Figure 1 It is a schematic flow chart of the main steps of a model defect detection method according to an embodiment of the present application;

[0031] Figure 2 It is a schematic diagram of the main structure block diagram of a model defect detection device according to an embodiment of the present application;

[0032] Figure 3 It is a main structural diagram of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] Some embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0034] In the description of the present invention, "module" and "processor" may include hardware, software or a combination of the two. A module may include hardware circuits, various suitable sensors, communication ports, and memories, and may also include software parts, such as program codes, or a combination of software and hardware. The processor may be a central processing unit, a microprocessor, an image processor, a digital signal processor, or any other suitable processor. The processor has data and / or signal processing functions. The processor may be implemented in software, hardware, or a combination of the two. Non-temporary computer-readable storage media include any suitable medium that can store program codes, such as a magnetic disk, a hard disk, an optical disk, a flash memory, a read-only memory, a random access memory, and the like. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The term "at least one A or B" or "at least one of A and B" has a similar meaning to "A and / or B", and may include only A, only B, or A and B. The singular terms "one" and "the" may also include plural forms.

[0035] Model defects: Model defects in this application refer to the problems of missed detection and false detection in the model. They are usually the terms used by the security team to discover under what specific conditions the model has deficiencies and provide them to the technical team for improvement.

[0036] See attached Figure 1 , Figure 1 1 is a flowchart of the main implementation steps of a model defect detection method provided by an embodiment of the present application. As shown in the figure, the method mainly includes the following steps S101 to S105:

[0037] Step S101: obtaining a synthetic image data set;

[0038] The data in the synthetic image data set described in this application is image data with annotations.

[0039] Specifically, in this embodiment, a large amount of image data can be generated by using existing simulation software to obtain the synthetic image data set. It is understandable that the existing simulation software can directly generate image data with annotations. Furthermore, the characteristic of arbitrary generalization of synthetic data can be utilized to obtain generalized data through data generalization processing to form the synthetic image data set.

[0040] A specific implementation of step S101 may be: using simulation software to generate a scene simulation image, wherein the scene simulation image is image data with annotations, and performing data generalization processing on the scene simulation image to obtain the synthetic image data set. Exemplarily, the simulation software may be software such as VEStudio and Labelbox. The data generalization processing may specifically be generalization processing based on the attributes of the scene and based on the attributes of the target in the scene, wherein the attributes of the scene may include the light intensity and light color in the scene, and the attributes of the target may include the size and texture of the target. For example, the illumination may be changed by the light source component in the simulation software, such as controlling the generalization of the illumination in the scene by controlling the intensity, color, range and other attributes of the light source, and the generalization of the target in the scene may also be achieved by changing the size of the target in the scene and the texture, color, and mapping of the material.

[0041] Step S102: verifying the model to be detected based on the synthetic image data set, and determining first defect sample data in the synthetic image data set based on the verification result;

[0042] Specifically in this embodiment, this step may be to verify the model to be detected based on the synthetic image data set to obtain a model evaluation index, and determine the first defect sample data in the synthetic image data set according to the model evaluation index.

[0043] Further, the first defect sample data in the synthetic image data set may be specifically determined by the following steps S1021 and S1022:

[0044] Step S1021, using the image data in the synthetic image data set as input of the model to be detected, and performing target prediction on the image data based on the model to be detected;

[0045] Step S1022, determining a model evaluation index based on the target prediction result output by the model, and selecting image data in the synthetic image data set corresponding to when the model evaluation index is lower than a preset threshold as the first defect sample data.

[0046] It should be noted that the annotation information carried by the image data in the synthetic image data set in this embodiment includes at least target coordinates and scene labels. Exemplarily, the target coordinates can be represented by 7 coordinate parameters (x, y, z, w, h, l, yaw), where x, y, z represent the position of the target in three-dimensional space, w represents the posture of the target, h represents the height of the target, l represents latitude, which is a parameter in the geographic coordinate system, and yaw represents the yaw angle; the scene label can be a label representing a vehicle, pedestrian, building, etc.

[0047] Step S103: obtaining a real image data set, where data in the real image data set is unlabeled image data;

[0048] Specifically in this embodiment, the real image data set may be formed by acquiring image data in a real scene using a sensor.

[0049] Step S104: obtaining second defect sample data based on the first defect sample data and the real image data set;

[0050] Specifically in this embodiment, this step may be to obtain the sample data to be labeled from the real image data set based on the first defect sample data, and then label the sample data to be labeled to obtain the second defect sample data. It should be understood that the acquisition of the sample data to be labeled from the real image data set based on the first defect sample data may be to screen the image data that is consistent with the first defect sample data or has the highest similarity from the real image data set through feature comparison of image data as the sample data to be labeled. Since the first defect sample data is annotated image data, the labeling of the sample data to be labeled may be specifically to label the sample data to be labeled according to the labeling information of the first defect sample data to obtain the second defect sample data. For example, if the scene label of the first defect sample data is vehicle, labeling the sample data to be labeled includes labeling the scene label of the sample data to be labeled as vehicle. Exemplarily, in this embodiment, the sample data to be labeled may be labeled using a three-dimensional bbox labeling tool.

[0051] A specific implementation of step S104 may be: using similar graph retrieval technology to select image data similar to the first defect sample data from the real image data set as the sample data to be labeled, and labeling the sample data to be labeled according to the labeling information carried by the first defect sample data to obtain the second defect sample data. It should be understood that the similar graph retrieval technology may be an existing technology, such as image similarity retrieval based on a hash algorithm or image similarity retrieval based on a machine learning model.

[0052] Step S105: Detecting whether the model to be detected has model defects based on the second defect sample data.

[0053] Specifically in this embodiment, this step may be to use the second defect sample data as the input of the model to be detected, perform target prediction on the second defect sample data based on the model to be detected, determine the model evaluation index based on the target prediction result output by the model, and when the model evaluation index is lower than a preset threshold, determine that the model to be detected has a model defect. Exemplarily, the model to be detected may be a target detection model, and the model evaluation index may be an accuracy rate and / or a recall rate. It should be understood that the preset threshold may be the value of the accuracy rate and / or the recall rate of the model set according to the actual application needs, that is, when the accuracy rate and / or the recall rate of the model calculated according to the target prediction result output by the model is lower than the preset threshold, it indicates that the target detection model may have false detection or missed detection of some targets, which is risky for the security application scenario of the model. Therefore, the model defect detection method proposed in this application only needs to obtain the second defect sample data based on very little data annotation cost and be used for comprehensive mining of model defects, which can effectively improve the efficiency of model defect detection.

[0054] See attached Figure 2 , Figure 2 1 is a schematic diagram of the main components of a model defect detection device provided by an embodiment of the present application. The device shown in the figure mainly includes a first acquisition module 201, a model verification module 202, a second acquisition module 203, a data screening module 204 and a defect detection module 205, wherein:

[0055] The first acquisition module 201 is used to acquire a synthetic image data set, where the data in the synthetic image data set is image data with annotations;

[0056] The model verification module 202 is used to verify the model to be detected based on the synthetic image data set, and determine the first defect sample data in the synthetic image data set based on the verification result;

[0057] In one embodiment, the model verification module 202 is specifically used to verify the model to be detected based on the synthetic image data set to obtain a model evaluation index, and determine the first defect sample data in the synthetic image data set according to the model evaluation index;

[0058] The second acquisition module 203 is used to acquire a real image data set, where the data in the real image data set is unlabeled image data;

[0059] The data screening module 204 is used to obtain second defect sample data based on the first defect sample data determined by the model verification module 202 and the real image data set obtained by the second acquisition module 203;

[0060] In one implementation, the data screening module 204 is specifically configured to screen and obtain sample data to be labeled from the real image data set based on the first defect sample data, and to label the sample data to be labeled to obtain second defect sample data;

[0061] The defect detection module 205 is used to detect whether the model to be detected has a model defect based on the second defect sample data obtained by the data screening module 204.

[0062] Further, it should be understood that since the setting of each module is only for illustrating the functional units of the device of the present invention, the physical devices corresponding to these modules may be the processor itself, or a part of the software in the processor, a part of the hardware, or a part of the combination of software and hardware. Therefore, the number of each module in the figure is only schematic.

[0063] Those skilled in the art will appreciate that the modules in the device can be adaptively split or merged. Such splitting or merging of specific modules will not cause the technical solution to deviate from the principle of the present invention, and therefore, the technical solutions after splitting or merging will fall within the protection scope of the present invention.

[0064] It is understood by those skilled in the art that the present invention implements all or part of the processes in the method of the above embodiment, and can also be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable storage medium may include: any entity or device, medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory, random access memory, electric carrier signal, telecommunication signal and software distribution medium, etc. that can carry the computer program code.

[0065] Furthermore, the present invention also provides a computer device. Figure 3 , Figure 3It is a schematic diagram of the main structure of a computer device according to an embodiment of the present application. In an embodiment of a computer device according to the present application, the computer device includes a processor 301 and a memory 302. The memory 302 can be configured to store a program for executing the model defect detection method of the above method embodiment, and the processor 301 can be configured to execute the program in the memory, which includes but is not limited to the program for executing the model defect detection method of the above method embodiment. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer device can be a control device device formed by various electronic devices.

[0066] In some possible implementations of the present application, the computer device may include multiple processors 301 and multiple memories 302. The program for executing the model defect detection method of the above method embodiment may be divided into multiple subprograms, and each subprogram may be loaded and run by the processor 301 to execute different steps of the model defect detection method of the above method embodiment. Specifically, each subprogram may be stored in different memories 302, and each processor 301 may be configured to execute the program in one or more memories 302 to jointly implement the model defect detection method of the above method embodiment, that is, each processor 301 executes different steps of the model defect detection method of the above method embodiment to jointly implement the model defect detection method of the above method embodiment.

[0067] Furthermore, the present invention also provides a computer-readable storage medium. In a computer-readable storage medium embodiment according to the present invention, the computer-readable storage medium can be configured to store a program for executing the model defect detection method of the above-mentioned method embodiment, and the program can be loaded and run by the processor to implement the above-mentioned model defect detection method. For ease of explanation, only the parts related to the embodiment of the present invention are shown. For specific technical details not disclosed, please refer to the method part of the embodiment of the present invention. The computer-readable storage medium can be a storage device formed by various electronic devices. Optionally, the computer-readable storage medium in the embodiment of the present invention is a non-temporary computer-readable storage medium.

[0068] The user personal information processed by this application will vary depending on the specific product / service scenario, and must be based on the specific scenario in which the user uses the product / service. It may involve the user's account information, device information and other relevant information. This application will treat the user's personal information and its processing with a high degree of diligence. This application attaches great importance to the security of user personal information, and has taken reasonable and feasible security protection measures that meet industry standards to protect user information and prevent personal information from being unauthorized access, public disclosure, use, modification, damage or loss.

[0069] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

Claims

1. A model defect detection method, characterized in that: The method comprises: Acquire a synthetic image data set, wherein the data in the synthetic image data set is image data with annotations; Verifying the model to be detected based on the synthetic image data set, and determining first defect sample data in the synthetic image data set based on the verification result; Acquire a real image data set, where data in the real image data set is unlabeled image data; Obtaining second defect sample data based on the first defect sample data and the real image data set; Detecting whether the model to be detected has a model defect based on the second defect sample data; The step of verifying the model to be detected based on the synthetic image data set and determining the first defect sample data in the synthetic image data set based on the verification result comprises: using the image data in the synthetic image data set as the input of the model to be detected and performing target prediction on the image data based on the model to be detected; determining a model evaluation index based on the target prediction result output by the model, and selecting the image data in the synthetic image data set corresponding to the time when the model evaluation index is lower than a preset threshold as the first defect sample data; The obtaining of second defect sample data based on the first defect sample data and the real image data set includes: obtaining to-be-annotated sample data from the real image data set based on the first defect sample data, and annotating the to-be-annotated sample data to obtain the second defect sample data.

2. The method according to claim 1, characterized in that The obtaining of the synthetic image data set comprises: A scene simulation image is generated by using simulation software, and data generalization processing is performed on the scene simulation image to obtain the synthetic image data set.

3. The method according to claim 1, characterized in that: The annotation information carried by the image data in the synthetic image data set includes at least target coordinates and scene labels.

4. The method according to claim 1, characterized in that: The acquiring the sample data to be labeled from the real image data set based on the first defect sample data includes: A similar graph retrieval technology is used to select image data similar to the first defect sample data from the real image data set as the sample data to be labeled.

5. The method according to claim 1, characterized in that The detecting whether the model to be detected has a model defect based on the second defect sample data includes: The second defect sample data is used as the input of the model to be detected, target prediction is performed on the second defect sample data based on the model to be detected, and a model evaluation index is determined based on the target prediction result output by the model. When the model evaluation index is lower than a preset threshold, it is determined that the model to be detected has a model defect.

6. A computer device comprising a processor and a memory, wherein the memory is suitable for storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by the processor to execute the model defect detection method according to any one of claims 1 to 5.

7. A computer-readable storage medium storing a plurality of program codes, characterized in that: The program code is suitable for being loaded and run by a processor to execute the model defect detection method according to any one of claims 1 to 5.

Images