Training system and method, testing system and method, and data screening system and method

Abstract

The invention provides a training system and method, a test system and method, a data screening system and method, and a computer-readable recording medium having a program stored therein, including determining whether an object detection model completes training set learning and determining whether the object detection model is over-trained by a verification set, and outputting a training model as a main model before the object detection model is overtrained, and how to iterate the main model to conform to a test result of a test set and continuously maintain an online test level. And a standardized method for achieving data marking consistency and training data selection is provided.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network training methods, and more particularly to the technical field of object detection model training methods. Background Art

[0002] General users (especially those who only have domain knowledge but lack AI knowledge and skills) still often encounter many problems when using LowCode / No Code artificial intelligence platform tools to conduct on-site practical artificial intelligence model training and development, especially when facing more complex detection problems such as defect detection. It is uncertain whether the content of the collected training images is sufficient; there may be some images in the image that fall into the fuzzy judgment range, which will cause inaccurate model judgment. If you want to use object detection model calculation technology to achieve better defect recognition capabilities, a standard object detection model has very high requirements for the quantity and quality of training images: in terms of quantity, the model may require a lot of training images, but the object detection image marking method is very time-consuming, and defect images are usually difficult to collect in a factory production environment; in terms of quality, it is difficult to achieve labeling consistency due to the gradual variation of defects, resulting in the inability to improve model accuracy. In addition, in order to improve the model's capabilities, it is necessary to find areas where the original data is not perfectly labeled, and find data similar to the misjudged images to add to the training, and continue to iterate and optimize the model's performance. This often takes a lot of effort and time from the user, and requires the user to have a certain degree of data analysis capabilities. Therefore, a complete set of systematic standard execution processes is needed, or even an application that can automatically execute the process. Summary of the invention

[0003] In view of this, some embodiments of the present invention provide a training system, a training method, a testing system, a testing method, a data screening system, a data screening method, and a computer-readable recording medium with a stored program to improve the problems of the prior art.

[0004] Some embodiments of the present invention provide a training system, comprising: at least one processing unit and an object detection model; the at least one processing unit is configured to perform: (a) before the object detection model completes learning of the training set, repeatedly perform: training the object detection model based on the training set with a current number of times, wherein the training set comprises a plurality of training images, each training image comprises at least one labeled outer frame, and a labeled area and a non-labeled area formed by the at least one labeled outer frame of each training image are used to train the object detection model; obtaining a first false positive set in the training set based on the trained object detection model and the training set; modifying the training set based on the first false positive set; and increasing the current number of times; and (b) obtaining a false positive value of the object detection model and a second false positive set in the verification set based on the trained object detection model and each of a plurality of verification images in the verification set; modifying the verification set based on the second false positive set; in response to the false positive value satisfying a condition, increasing the current number of times and performing steps (a) to (b); and in response to the false positive value not satisfying the aforementioned condition, outputting the previous training model as the main model.

[0005] Some embodiments of the present invention provide a training method, which is suitable for training an object detection model and is executed by at least one processing unit. The training method includes the aforementioned (a) before the object detection model completes learning of the training set, repeatedly executing: training the object detection model based on the training set with a current number of times, wherein the training set includes multiple training images, each of the training images includes at least one labeled outer frame, and the labeled area and the non-labeled area formed by the at least one labeled outer frame of each training image are used to train the object detection model; obtaining a first false positive set in the training set based on the trained object detection model and the training set; modifying the training set based on the first false positive set; and increasing the current number of times; and (b) obtaining a false positive value of the object detection model and a second false positive set in the verification set based on the trained object detection model and each of the multiple verification images in the verification set; modifying the verification set based on the second false positive set; in response to the false positive value satisfying a condition, increasing the current number of times and executing steps (a) to (b); and in response to the false positive value not satisfying the condition, outputting the previous training model as the main model.

[0006] Some embodiments of the present invention provide a testing system, comprising: at least one processing unit, configured to execute; obtaining a test error value of a main model and a test error set in a test set based on a main model and a test set; in response to the error value satisfying a test condition, outputting the main model as an offline model; and in response to the test error value not satisfying the test condition, outputting the main model and the test error set, wherein the test set comprises a plurality of test images, the test images comprising a plurality of first test images belonging to a first category and a plurality of second test images belonging to a second category, wherein an image belonging to the first category indicates that the image is determined to contain at least one defective image region belonging to one of a plurality of standard defect categories, and an image belonging to the second category indicates that the image is determined not to contain an image region (image region) belonging to one of the standard defect categories.

[0007] Some embodiments of the present invention provide a testing method, which is executed by at least one processing unit, and includes: obtaining a test error value of a main model and a test error set in a test set based on a main model and a test set; in response to the error value satisfying a test condition, outputting the main model as an offline model; and in response to the test error value not satisfying the test condition, outputting the main model and the test error set, wherein the test set includes a plurality of test images, the test images including a plurality of first test images belonging to a first category and a plurality of second test images belonging to a second category, wherein an image belonging to the first category indicates that the image is determined to include at least one defective image block belonging to one of a plurality of standard defect categories, and an image belonging to the second category indicates that the image is determined not to include an image block belonging to one of the standard defect categories.

[0008] Some embodiments of the present invention provide a data screening system, comprising: at least one processing unit, configured to execute; receiving a main model, a data pool, a training set and a test false positive set; selecting a plurality of test false positive images corresponding to at least one unresolved condition from the test false positive set; displaying the test false positive images, and annotating each test false positive image with a frame based on an annotation input, and performing an unresolved condition annotation process on a plurality of annotated areas of each test false positive image obtained when performing the frame annotation so that the annotated areas of each test false positive image have an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the at least one unresolved condition; based on the test false positive images , training the screening model until the screening model completes the screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes at least one unresolved condition; obtaining a false positive set in the training set based on the screening model and the training set, and in response to the false positive set being non-empty, displaying and processing the false positive set based on a first input to obtain a processed training set; and obtaining an acceptable data set in the data pool based on the screening model and the data pool, and in response to the acceptable data set being non-empty, for at least one acceptable image in the acceptable data set, displaying and processing the currently displayed acceptable image based on a second input; and integrating the processed training set and the acceptable data set as a new training set.

[0009] Some embodiments of the present invention provide a data screening method, which is executed by at least one processing unit, and includes: receiving a main model, a data pool, a training set, and a test misjudgment set; selecting a plurality of test misjudgment images corresponding to at least one unresolved condition from the test misjudgment set; displaying the test misjudgment images, and annotating each test misjudgment image with a frame based on an annotation input, and performing an unresolved condition annotation process on a plurality of annotated areas of each test misjudgment image obtained when performing the frame annotation so that the annotated areas of each test misjudgment image have an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the at least one unresolved condition; based on the test misjudgment images, training The method comprises the steps of: training a screening model until the screening model completes screening training, wherein the structure of the screening model is the same as that of the main model, and at least one detection category of the screening model includes at least one unresolved condition; obtaining a false positive set in the training set based on the screening model and the training set, and in response to the false positive set being non-empty, displaying and processing the false positive set based on a first input to obtain a processed training set; and obtaining an acceptable data set in the data pool based on the screening model and the data pool, and in response to the acceptable data set being non-empty, displaying and processing a currently displayed acceptable image based on a second input for at least one acceptable image in the acceptable data set; and integrating the processed training set and the acceptable data set as a new training set.

[0010] The present invention provides a computer-readable medium with a stored program, which can complete the aforementioned training method, testing method, data screening method and online model training method after at least one processor loads the program and executes it.

[0011] Based on the above, some embodiments of the present invention provide a training system, a training method, a testing system, a testing method, a data screening system, a data screening method, and a computer-readable recording medium with a stored program. By modifying the training set, domain knowledge can be continuously integrated to improve the quality of the training set. By continuously judging whether the object detection model has completed the training set learning, it can be ensured that the model has passed the underfitting stage. In addition, by judging whether the object detection model is overfitting with a validation set, the object detection model in the best training state can be obtained before the object detection model is overtrained as the main model. By finding an acceptable data set, the training set can be automatically analyzed to find the cause of misjudgment in the test set. And by using the program algorithm flow, combined with domain knowledge, the existing training set annotation problems can be automatically corrected and new missing data can be introduced to complete the data set required for the next training. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention.

[0013] Figure 2 FIG. 4 is a schematic diagram of image category classification according to an embodiment of the present invention.

[0014] Figure 3 It is a schematic diagram of an outer frame marking according to an embodiment of the present invention.

[0015] Figure 4 FIG. 4 is a schematic diagram of undetermined defect categories according to an embodiment of the present invention.

[0016] Figure 5 It is a non-classified schematic diagram drawn according to an embodiment of the present invention.

[0017] Figure 6 FIG. 4 is a schematic diagram of processing a first misjudgment set according to an embodiment of the present invention.

[0018] Figure 7 FIG. 4 is a schematic diagram of processing a first misjudgment set according to an embodiment of the present invention.

[0019] Figure 8 FIG. 4 is a schematic diagram of processing a first misjudgment set according to an embodiment of the present invention.

[0020] Fig. 9 FIG. 4 is a schematic diagram of processing a first misjudgment set according to an embodiment of the present invention.

[0021] Fig.10 FIG. 4 is a schematic diagram of processing a first misjudgment set according to an embodiment of the present invention.

[0022] Fig.11 It is a schematic diagram of a process of marking a situation to be resolved according to an embodiment of the present invention.

[0023] Fig.12 It is a schematic diagram of a process of marking a situation to be resolved according to an embodiment of the present invention.

[0024] Fig.13 FIG. 4 is a schematic diagram of a predicted missing set cleaning operation according to an embodiment of the present invention.

[0025] Fig.14 FIG. 4 is a schematic diagram of a predicted missing set cleaning operation according to an embodiment of the present invention.

[0026] Fig.15 FIG. 4 is a schematic diagram of a prediction misjudgment set cleaning operation according to an embodiment of the present invention.

[0027] Fig.16 FIG. 4 is a schematic diagram of screening acceptable data according to an embodiment of the present invention.

[0028] Fig.17 FIG. 4 is a schematic diagram of screening acceptable data according to an embodiment of the present invention.

[0029] Fig.18 FIG. 4 is a schematic diagram of a process of labeling acceptable image defect categories according to an embodiment of the present invention.

[0030] Fig.19 FIG. 4 is a schematic diagram of a process of labeling acceptable image defect categories according to an embodiment of the present invention.

[0031] Fig. 20 is a flow chart of a training method according to an embodiment of the present invention.

[0032] Fig.21 is a flow chart of a training method according to an embodiment of the present invention.

[0033] Fig. 22 is a flow chart of a training method according to an embodiment of the present invention.

[0034] Fig.23 is a flow chart of a training method according to an embodiment of the present invention.

[0035] Fig.24 is a flow chart of a training method according to an embodiment of the present invention.

[0036] Fig.25is a test flow chart according to an embodiment of the present invention.

[0037] Fig.26 is a flow chart of a sub-process according to an embodiment of the present invention.

[0038] Fig. 27 It is a flow chart of an online model training method according to some embodiments of the present invention.

[0039] Fig.28 It is a flowchart of an online sub-process according to some embodiments of the present invention.

[0040] Fig.29 is a flow chart of a training method according to an embodiment of the present invention.

[0041] Fig.30 is a flow chart of an online training method according to an embodiment of the present invention.

[0042] 100: Electronic equipment

[0043] 101: Processing Unit

[0044] 102: Internal memory

[0045] 103: Non-volatile memory

[0046] 104: User Interface

[0047] class A, class B, class C, class D, class E, class F: annotation

[0048] 201,202,203,401,402,403,501,502,503,601,602,701,702,801,802,901,902,1001,1002,1003: Image

[0049] 2011,2012,2013,2021,2022,2023,2031,2032,2033,301,4011,4013,5011,6011,7011,8011,9011,10011,10012,10013,11011,12011,14011,14012,14013,15011,15011',15012,15013,16011,16021,17011,17021,17031,18011: Image blocks

[0050] 302,303,4012,4014,5012,5013,6012,7012,8012,9012,9013,100111,100121,100131,110111,120111,130111,130121,130131,140111,140112,140121,140131,150111,150113,150121,150131,160121,160221,170121,170221,170321,180111,180121,190111: Frame

[0051] 13011,13012,13013,150112,150122,150132,16012,16022,17012,17022,17032,18012,19011,19012: Area

[0052] 1101,1102,1201,1202: Test misjudgment images

[0053] 1301,1302,1402,1403,1501,1502: Filter training images

[0054] 1401: Training image

[0055] 1601, 1602: First status image

[0056] 1701, 1702, 1703: Second situation image

[0057] 1801,1802,1803,1901,1902: Acceptable images

[0058] S2001~S2007,S2101~S2104,S2201~S2204,S2301~S2305,S2401~S2405,S2501~S2504,S2601~S2604,S2701~S2707,S2801~S2804: Steps DETAILED DESCRIPTION

[0059] The aforementioned and other technical contents, features and effects of the present invention will be clearly presented in the following detailed description of the embodiments with reference to the diagrams. The thickness or size of each element in the diagram is expressed in an exaggerated, omitted or approximate manner for the understanding and reading of those skilled in the art, and the size of each element is not completely its actual size, and is not used to limit the limiting conditions that the present invention can be implemented, so it has no technical substantive significance. Any structural modification, change in proportional relationship or adjustment of size, without affecting the effects that the present invention can produce and the purposes that can be achieved, should still fall within the scope of the technical contents disclosed in the present invention. The same reference numerals will be used to represent the same or similar elements in all diagrams.

[0060] Figure 1 is a block diagram of an electronic device according to an embodiment of the present invention. Figure 1 , the electronic device 100 includes a processing unit 101, an internal memory 102, a non-volatile memory 103 and a user interface 104. The internal memory 102 and the non-volatile memory 103 are used to store programs, and the programs may include program codes, and the program codes include computer operation instructions. The internal memory 102 and the non-volatile memory 103 provide instructions and data to the processing unit 101. The processing unit 101 reads the corresponding computer program from the non-volatile memory 103 into the internal memory 102 and then runs it. The processing unit 101 may include a single processor or a plurality of processors of different types. The user interface 104 is configured to display images or information that the processing unit 101 wants to present to the user, and to receive signals generated by user operations.

[0061] In some embodiments of the present invention, the training system implements Figure 1 The electronic device 100 is shown. A non-volatile memory 103 stores an object detection model to be trained (e.g., an instance segmentation model). The object detection model includes multiple parameters. The multiple parameters of the object detection model may be initially randomly set values, or values ​​obtained by the object detection model through some training process.

[0062] After training, the aforementioned object detection model can receive images and output the categories of all objects appearing in the image and mark the locations of the objects. In some embodiments of the present invention, the aforementioned object detection model is an instance segmentation model. The instance segmentation model classifies objects in the image into different categories at the pixel level. Instance segmentation models are, for example, DeepMask, SharpMask, InstanceFCN, FCIS, Mask R-CNN, Mask Scoring R-CNN, YOLACT, etc., which are not limited by the present invention. In some embodiments of the present invention, the aforementioned object detection model is a YOLO model. Of course, the aforementioned object detection model can be other models that can output the categories of all objects appearing in the image and mark the locations of the objects after training, which are not limited by the present invention. The following will mainly illustrate the object detection model as an instance segmentation model.

[0063] In some embodiments of the present invention, the training system includes a neural network module, which stores the object detection model. In some embodiments, the neural network module is a neural-network processing unit (NPU), which is interconnected with the processing unit 101, the internal memory 102, the non-volatile memory 103 and the user interface 104 in the electronic device 100.

[0064] The following is a detailed description of the training methods of some embodiments of the present invention and how the modules of the electronic device 100 work together with each other with reference to the drawings.

[0065] Fig. 20 FIG. 1 is a flow chart of a training method according to an embodiment of the present invention. Fig. 20In the illustrated embodiment, the training method includes executing steps S2001 to S2007 by the processing unit 101. In step S2001, the processing unit 101 trains the object detection model based on a training set with a current number of times. The current number of times refers to the number of the current training cycle (epoch). A training cycle means that all data in the training set are used to train the neural network once. Among them, the training set includes a plurality of training images, each of which includes at least one defective image block belonging to one of a plurality of defect categories, and these defective image blocks are all marked with an outer frame, and the marked area formed by the outer frame of each defective image block includes the defective image block itself, and the unmarked area has no defects. The purpose of training the object detection model is to hope that the trained object detection model can identify the image blocks and non-defective parts in the input image that belong to the aforementioned defect category, and the image blocks identified as belonging to the aforementioned defect category are marked with an outer frame so that the marked area formed by each outer frame includes the respective defective image blocks, and the non-marked area that is not framed will not be mistakenly identified. It is worth noting that in one embodiment of the present invention, when marking the outer frame of the defective image block, a closed outer frame line is drawn with a line. At this time, the closed outer frame line is the outer frame of the defective image block, and the area included in the closed outer frame line is the marked area formed by the outer frame. In some embodiments of the present invention, when marking an outer frame for a defective image block, a block is used (for example, a color block is used to cover the defective image block on the user interface 104) to cover an area including the defective image block. At this time, the edge of the area including the defective image block covered by the block is the outer frame of the defective image block, and the area including the defective image block covered by the block is the marked area formed by the aforementioned outer frame. It is worth noting that any method that can clearly mark the range of the defective image block can be applied to the present invention, and the present invention is not limited to the aforementioned two embodiments.

[0066] In step S2002, the processing unit 101 obtains a first misjudgment set in the training set based on the trained object detection model and the aforementioned training set. That is, the processing unit 101 inputs all the data in the training set to the object detection model for identification, calculates the number of misjudgments, and sets the training images misjudged by the object detection model in the training set as the first misjudgment set.

[0067] The processing unit 101 processes the first misjudgment set. In some embodiments of the present invention, the processing unit 101 processes the first misjudgment set based on the following steps: the processing unit 101 determines whether the first misjudgment set is non-empty. If the first misjudgment set is non-empty, that is, the first misjudgment set is not an empty set, the processing unit 101 displays all misjudged training images in the first misjudgment set, and modifies the first misjudgment set based on the received first input. Since the first misjudgment set is included in the training set, if the first misjudgment set is corrected based on the first input, it is equivalent to the training set being corrected. In other words, the processing unit 101 modifies the training set based on the first misjudgment set.

[0068] In step S2003, the processing unit 101 determines whether the object detection model has completed the training set learning based on the trained object detection model and the aforementioned training set. In response to the object detection model completing the training set learning, step S2005 is executed. In response to the object detection model not completing the training set learning, step S2004 is first executed to increase the current number, and then the training is continued in step S2001. It is worth noting that if the first misjudgment set has been corrected based on the first input in the aforementioned steps, when returning to step S2001 to continue training, the object detection model is retrained with the corrected training set. It is also worth noting that in step S2004, the current number can be increased by adding a preset value to the current number as the new current number, or the current value can be added to other values ​​according to other circumstances to be the new current number.

[0069] In step S2005, the processing unit 101 obtains the misjudgment value of the object detection model and the second misjudgment set in the verification set based on the trained object detection model and each of the multiple verification images in the verification set, wherein the misjudgment value is used to evaluate whether to retrain the object detection model. That is, in this step, the trained object detection model is verified with the image verification set of each of the multiple verification images in the verification set, and the second misjudgment set of the object detection model misjudgment is obtained. The processing unit 101 also processes the second misjudgment set. In some embodiments of the present invention, the processing unit 101 modifies the second misjudgment set based on the following steps: If the second misjudgment set is not empty, the processing unit 101 displays the second misjudgment set and processes the second misjudgment set based on the received second input. In step S2006, it is determined whether the misjudgment value satisfies a condition. In response to the aforementioned misjudgment value satisfying the aforementioned conditions, the object detection model of this time is stored as the "previous training model" (wherein the previous time refers to the previous time of the next time) and the misjudgment value of this time, and step S2004 is first executed to increase the current number, and then step S2001 is returned to continue training. In response to the misjudgment value not satisfying the aforementioned conditions, step S2007 is executed, and the previous training model is output as the main model in step S2007.

[0070] It is worth noting that, since the second misjudgment set is included in the validation set, if the second misjudgment set is corrected based on the second input, it is equivalent to the validation set being corrected. In other words, the processing unit 101 modifies the validation set based on the second misjudgment set. And after the process returns to step S2001 and continues training, when it is executed to S2005 again, the processing unit 101 verifies the trained object detection model with the corrected validation set.

[0071] In some embodiments of the present invention, the aforementioned condition is that the misjudgment value is less than or equal to the previous misjudgment value. Under the condition that the misjudgment value is less than or equal to the previous misjudgment value, it means that training the object detection model with the training set again can obtain an object detection model with a smaller misjudgment value. If the misjudgment value of this time is greater than the previous misjudgment value, the stored previous object detection model is output.

[0072] In some embodiments of the present invention, the aforementioned conditions are: the misjudgment value is not equal to zero and the misjudgment value of this time is less than or equal to the misjudgment value of the previous time. When the processing unit 101 actually determines whether the misjudgment value satisfies the aforementioned conditions, the processing unit 101 can use the following judgment logic to determine whether the misjudgment value satisfies the aforementioned conditions: the processing unit 101 first determines whether the misjudgment value is equal to zero. In response to the misjudgment value not being equal to zero, the processing unit 101 continues to determine whether the misjudgment value is less than or equal to the previous misjudgment value. In response to the misjudgment value being less than or equal to the previous misjudgment value, the misjudgment value is determined to satisfy the aforementioned conditions; in response to the misjudgment value being equal to zero, the misjudgment value is directly determined to not satisfy the aforementioned conditions (because zero is already the minimum value). In other words, when the misjudgment value is not equal to zero, whether to continue training the model is determined by whether the misjudgment value becomes larger; but when the misjudgment value is equal to zero, the training will be stopped directly and the current model will be output, instead of training to let the misjudgment value rise before taking the previous model, thereby shortening unnecessary model training time.

[0073] In the aforementioned embodiment, by modifying the training set through the first input processing, the domain knowledge of the executor can be continuously integrated to improve the quality of the training set. By continuously judging whether the object detection model has completed the training set learning, it can be ensured that the model has passed the underfitting stage. In addition, by judging whether the object detection model is overfitting with the validation set, the object detection model with the best training state can be obtained before the object detection model is overtrained to serve as the main model.

[0074] It is worth noting that the aforementioned training system and training method are applicable to various object detection application scenarios, for example, training of instance segmentation models for medical image detection, or training of instance segmentation models for defect detection.

[0075] The “whether the training set learning is completed”, the “first misjudgment set” and the “misjudgment” referred to in the aforementioned step S2003 are defined differently depending on different applications. Fig.21 is a flow chart of a training method according to an embodiment of the present invention. Fig.21 ,exist Fig.21 In the illustrated embodiment, "false positive" is defined as the trained object detection model missing at least one defective image block of the training image, that is, at least one defective image block of the training image is not detected by the trained object detection model. It is worth noting that, since the trained object detection model will simultaneously output the defect category to which the detected defective image block belongs, the aforementioned at least one defective image block is not detected by the trained object detection model, including the situation where the defective category is output incorrectly. In this embodiment, the aforementioned step S2003 includes steps S2101 to S2104. In step S2101, the processing unit 101 inputs each training image into the trained object detection model, and calculates the number of missing images in the missing set that are not detected by the trained object detection model in at least one defective image block of all the training images. In step S2102, the processing unit 101 determines whether the aforementioned number of missing images is zero. In step S2103, in response to the number of misses being zero, the processing unit 101 determines that the object detection model has completed the training set learning, and sets the first false positive set to be an empty set. In step S2104, in response to the number of misses being non-zero, the processing unit 101 determines that the object detection model has not completed the training set learning, and sets at least one false positive training image having a member of the miss set among the training images as the first false positive set.

[0076] It is worth noting that, in some embodiments of the present invention, the aforementioned defect categories include standard defect categories and special defect categories, wherein the standard defect categories are what types of "to be detected" image blocks that the executor hopes the model can identify, and the special defect categories are used in the subsequent disclosed systems and processes (the standard defect categories and special defect categories will be further explained in the following embodiments). In this embodiment, when calculating the number of misses included in the miss set of at least one defective image block of all the training images that has not been detected by the trained object detection model, the calculation can be performed only for the standard defect category, while the special defect category is ignored.

[0077] In some embodiments of the present invention, a defective image block is "detected" by a trained object detection model if the intersection over union (IOU) between the outer frame output by the object detection model and the outer frame marked by the defective image block is greater than a preset IOU, and the confidence level of the defective image block output by the object detection model is greater than a preset confidence level. In this case, the defective image block is considered to be "detected" by the trained object detection model. Otherwise, the defective image block is considered not to be "detected". In some embodiments of the present invention, the preset IOU is 0.6, and the preset confidence level is 0.8.

[0078] Fig. 22 is a flow chart of a training method according to an embodiment of the present invention. Fig. 22 ,exist Fig. 22 In the illustrated embodiment, "false positives" are defined as misses by the trained object detection model and misses in the training image that are not labeled as belonging to one of the standard defect categories but are detected by the trained object detection model. In this embodiment, the aforementioned step S2003 includes steps S2201 to S2204. In step S2201, the processing unit 101 inputs each training image into the trained object detection model, and calculates the number of misses included in the number of at least one defect image block in all the training images that are not detected by the trained object detection model, and calculates the number of errors included in the number of errors included in the number of at least one defect image block in all the training images that are not labeled as belonging to one of the standard defect categories but are detected by the trained object detection model. In step S2202, the processing unit 101 determines whether the sum of the number of misses and the number of errors is zero. In step S2203, in response to the sum of the number of misses and the number of errors being zero, the processing unit 101 determines that the object detection model has completed the training set learning, and sets the first false positive set to be an empty set. In step S2204, in response to the sum of the number of misses and the number of errors being non-zero, the object detection model is determined to have not completed the training set learning, and at least one false positive training image having a member of the miss set or a member of the error set in the training image is set as the first false positive set.

[0079] It is worth noting that in the above embodiment, the object detection model is judged to have completed the training set learning only when the number of misses is zero or the sum of the number of misses and the number of errors is zero. The above condition can be relaxed to the number of misses being a preset small number or the sum of the number of misses and the number of errors being a preset small number, and the present invention is not limited thereto.

[0080] Fig.23 FIG. 1 is a flow chart of a training method according to an embodiment of the present invention. Fig.23In the illustrated embodiment, the aforementioned step S2002 includes steps S2301 to S2305 to process the first misjudgment set. In step S2301, the processing unit 101 confirms whether the first misjudgment set is non-empty. In step S2302, if the first misjudgment set is non-empty, the processing unit 101 executes step S2303. If the first misjudgment set is not non-empty (i.e., an empty set), it enters step S2305. In step S2305, the processing unit 101 does not process the first misjudgment set. In step S2303, the processing unit 101 displays at least one misjudgment set image in the first misjudgment set on the user interface 104. A modification signal is received from the user interface 104 for each misjudged image. The aforementioned modification signal may be a signal for skipping modification, and the signal for skipping modification indicates that the displayed misjudged image does not need to be corrected. The implementation method of the modification signal will be further disclosed in subsequent embodiments. In step S2304, the processing unit 101 uses at least one modification signal corresponding to at least one false positive set image received by the user interface 104 as a first input, and modifies the first false positive set based on the first input. It is worth noting that in the aforementioned step S2005, the processing unit 101 may also use Fig.23 The illustrated steps process the second set of false positives.

[0081] The following diagrams illustrate the preparation process of the aforementioned training set and validation set in some embodiments.

[0082] Figure 2 is a schematic diagram of image classification according to an embodiment of the present invention. Figure 2 ,exist Figure 2 In the illustrated embodiments, in some embodiments of the present invention, the executive personnel collect all available image data (such as photos) as much as possible as a data pool. The morphological distribution of the data image blocks in the data pool is arbitrary, but the data content in the data pool should be enriched as much as possible. The data in the data pool will serve as the data source for the training set and the verification set in the aforementioned training system and training method. It is worth noting that during the execution of the training system and the training method, if the executive personnel collect new image data, they can also add it to the data pool at any time. The executive personnel obtain the test set by full inspection or sampling based on the actual production data of the production line. The morphological distribution of the data content of the test set is the data distribution pattern representing the actual production information of the current production line, which is used as the data for each generation of model performance test in the test process disclosed below. In the test process, it is a fixed amount of data. The test set and the aforementioned data pool are two independent sets.

[0083] Generally, model training will prepare three sets of data: "training set", "validation set" and "test set", which is the same as the practice of this embodiment; however, general model training usually randomly divides all collected data into these three sets of data in a certain ratio (for example, 6:2:2). The concept of this practice is actually to train a "general standard" model (the degree of generality depends on the collected data), but we define two types of data: "data pool" and "test set" at the beginning, where the meaning of the data in the "data pool" is similar to the "all collected data" in general model training, and the meaning of the "test set" is to simulate the "data distribution pattern during the actual operation of the production line". Therefore, the "training set" and "validation set" data of this embodiment are both from the data pool, and the purpose is to train a model that is "approximate to the general standard", but then the "test set" data will be used to calibrate this model, so that it can be corrected in the direction of "applicable to the actual data distribution of the production line" under the premise of having "approximate to the general standard". In this way, the generalization ability of the "general standard" model can be maintained at the same time, and it can also be corrected in the direction of "specific data distribution pattern", so as to achieve the purpose of production line application more quickly.

[0084] The images collected by the operator may contain background information and some traces, which are called image regions of the image. For example, Figure 2The image 201 shown has image blocks 2011, 2012 and 2013, image blocks 2021, 2022 and 2023, and image blocks 2031, 2032 and 2033. The image blocks of the image can be divided into three types by the executive personnel according to the "degree" of their presentation: "to be detected", "not to be detected" and "uncertain whether to be detected". The executive personnel hope that this model can identify which types of "to be detected" image blocks are called standard defect categories. The standard defect categories are defined by the executive personnel themselves. The number of categories, names, presentation differences between categories, etc. are all defined by the user according to their needs (the standard defect categories will be further explained in the following instructions). The executive personnel then divide all photos into three data categories: the first category, the second category and the third category based on the judgment criteria. The images in the first category are determined to contain image blocks belonging to one of the multiple standard defect categories, and the aforementioned image blocks belonging to one of the multiple standard defect categories are called defective image blocks. For example, image block 2011 of image 201 (presented by solid lines in the figure, it is worth noting that the solid line presentation is only for illustration and is not used to limit the actual presentation method of the defective image blocks of the present invention) is a defective image block. The images in the second category are determined not to contain image blocks belonging to one of the standard defect categories, nor to contain image blocks that cannot be determined whether they belong to one of the defect categories (that is, the aforementioned "uncertain whether to detect" image blocks, hereinafter referred to as uncertain image blocks), but contain acceptable image blocks (that is, the aforementioned "undetectable" image blocks), the aforementioned acceptable image blocks are also called actual OK image blocks, and the second category can also be called the actual OK category. For example, image block 2013 of image 201, image block 2023 of image 202, and image block 2031, image block 2032, and image block 2033 of image 203 are actually OK image blocks (presented by dotted lines in the figure, it is worth noting that the dotted lines are only for illustration and are not used to limit the actual presentation method of image blocks acceptable to the present invention). The third category does not include image blocks belonging to one of the multiple standard defect categories but includes image blocks that cannot be determined whether they belong to one of the defect categories. For example, image block 2012 of image 201, image block 2021, and image block 2022 of image 202 are uncertain image blocks (presented by dotted lines in the figure, it is worth noting that the dotted lines are only for illustration and are not used to limit the actual presentation method of uncertain image blocks of the present invention). In one embodiment, an image may not include any image blocks of the aforementioned images, in which case these image blocks certainly belong to the second category (actual OK category).In other embodiments, the image blocks of the first category and the image blocks of the second category are marked first, and the remaining unmarked image blocks belong to the image blocks of the third category. In actual operation, a label is added to the image so that the processing unit 101 can distinguish whether the image belongs to the first category, the second category or the third category.

[0085] Since image 201 is determined to contain an image block 2011 belonging to one of the multiple standard defect categories, image 201 belongs to the first category. Since image 203 is determined not to contain an image block belonging to one of the standard defect categories and does not contain an uncertain image block, image 203 belongs to the second category. Since image 202 does not contain an image block belonging to one of the multiple standard defect categories but contains image blocks (image blocks 2021 and image blocks 2022) that cannot be determined whether they belong to one of the defect categories, image 202 belongs to the third category. The set of images belonging to the first category in the data pool is called the first category set, the set of images belonging to the second category in the data pool is called the second category set, and the set of images belonging to the third category in the data pool is called the third category set. For the convenience of explanation, the images in the first category set are called the first images, and the images in the second category set are called the second images. In the test set, all data are divided into the first image, the second image, and the third image based on this concept, but when actually testing the model, in this embodiment, we will only take the first image and the second image for testing and misjudgment analysis.

[0086] In some embodiments of the present invention, the aforementioned standard defect categories are user-defined "to be detected" defect categories: for example, foreign matter categories, scratch categories, and other user-defined standard defect categories. However, based on process operation requirements, we additionally define special defect categories: undetermined defect categories and no categories. Special defect categories are used in the subsequent disclosed systems and processes, and special defect categories will be further described in the following embodiments.

[0087] It is worth noting that the "to be detected" defect categories described in the aforementioned embodiment include foreign matter categories and scratch categories, which is only a possible implementation method. The user-defined standard defect categories can be of many different types, and the number is not limited. Multiple types of standard defect categories can be defined according to user needs.

[0088] It is worth noting that the defective image block / actual OK image block represents data that can be correctly classified as the first category / second category by the labeler according to the current judgment standard, while the uncertain image block represents data that may be difficult for the executor to correctly classify as the first category / second category according to the current judgment standard (for example, the image block has a shallow mark). Therefore, the aforementioned training method will focus on the actual OK image block / defective image block data for object detection model training and verification to avoid the impact of "inconsistent judgment standards" on the data annotation quality, which in turn leads to limitations in the continuous optimization process of the object detection model. For the data of the uncertain image block, if the executor can find a new judgment standard in the future to ensure the consistency of the judgment of the current uncertain image block, then this data can be moved from the uncertain image block to the actual OK image block / defective image block, and the object detection model training and verification can be re-executed.

[0089] Next, it is explained how to select the aforementioned initial training set and validation set from the data pool in this embodiment. For each standard defect category (the aforementioned foreign matter category, scratch category), the executive personnel judge the possible presentation patterns based on the domain knowledge they have, and further define these defect presentation forms under each standard defect category (for example, the foreign matter category includes point-shaped foreign matter, hair-shaped foreign matter, etc., and the scratch category includes point-shaped punctures, linear slip marks, etc.), and finally, from the first category set of the data pool, an equal number of photos of each defect form are selected with the concept of average distribution (for example, 10 photos of each defect form are selected for each point-shaped foreign matter, hair-shaped foreign matter, point-shaped punctures, linear slip marks, etc.), forming the aforementioned training set.

[0090] Next, the executor annotates the frames of the images selected from the first category set in the data pool as the training set. Figure 3 is a schematic diagram of a frame marking according to an embodiment of the present invention. Figure 3 , the executor performs different frame annotations on the images selected from the first category set of the data pool as the training set according to the type of object detection model. For example, when the object detection model is an instance segmentation model, Figure 3 The recorded image block 301 is marked with an outer frame 302. When the object detection model is a YOLO model, Figure 3 The image block 301 is marked with a frame 303. The frame marking method depends on the type of object detection model, and the present invention is not limited thereto. When marking the frame, the defect category to which the image block belongs is also specified. For example, if the image block 301 is classified as a foreign object, the image block 301 is also specified as a foreign object category when marking the frame, for subsequent training of the object detection model. Figure 3As shown, the image block 301 is marked as class A, where class A represents the category of foreign matter. Figure 3 Draw class A next to image block 301, but Figure 3 The illustration is for illustration only. In practice, the processing unit 101 will store the category of the image block 301 in an appropriate data structure, and display class A or other recognizable symbols on the user interface when appropriate to indicate that the category of the image block 301 is a foreign object category. Since the image of the training set may have multiple annotated frames, the area contained in each annotated frame will be used to train the object detection model.

[0091] The validation set is composed of an appropriate number (eg, 100) of images randomly selected from the first category set and the second category set of the data pool.

[0092] Finally, the training set will include multiple training images, the verification set will include multiple verification images, the training images are selected from the first category set in the data pool, the verification images are selected from the first category set in the data pool and the second category set in the data pool, the first category set includes multiple first images, the second category set includes multiple second images, and at least one defective image block of the training images is marked with a frame (it is worth noting that, as mentioned above, the executor first selects the training images from the first category set in the data pool, and then marks the training images with frames, so the images of the first category set in the data pool that are not selected are not marked with frames). Fig. 20 During the verification step of step S2005, the object detection model can be verified by using the label of the image itself that can be identified as belonging to the first category or the second category, so only the training set needs to perform the outer frame annotation, and the verification set does not need it.

[0093] When training the object detection model, it is necessary to initialize an object detection model structure first. The object detection model structure will define the object detection model's ability to detect "how many types" of defect categories. Although the training set initially only contains standard defect categories such as foreign matter and scratches, it can also add the definition of special defect categories (undetermined defect categories and no categories) to increase the ability of the object detection model to learn these two special defect categories in the future. When the executor uses the annotation component (which can be implemented in software or hardware) to perform frame annotation, it is necessary to define all the "defect category names" that the executor wants to detect (which also defines the ability of the model to detect the number of defect types). In this embodiment, in addition to the defect categories defined by the executor himself, two special categories will be automatically "pre-defined", namely the two special defect categories mentioned above: undetermined defect categories and no categories, in order to be used in the subsequent public system and process. The meaning of the aforementioned undetermined defect category refers to the image block that the trained object detection model considers to be a defective image block, but the executor considers it to be an uncertain image block based on the judgment criteria. A training image in a training set may contain several image blocks marked as defective image blocks and several image blocks marked as uncertain image blocks. When an image block of a training image is considered to be a defective image block by the trained object detection model, but this image block is considered to be an uncertain image block by the executor, then this image block will be assigned a special category of undetermined defect category.

[0094] It is also worth noting that, based on the above definition, the undetermined defect category is a category of "passive annotation". That is, at the beginning, the executor will not actively annotate the image blocks that he thinks are uncertain. Instead, during the object detection model training process, the object detection model detects that the area belongs to a certain "standard defect category", but the executor does not think it belongs to a certain "standard defect category" to be detected during the re-judgment, and the executor cannot determine that it is an image block of the image that can be ignored and "not detected", so the detection area of ​​this object detection model is modified to a special defect category annotation of "undetermined defect category".

[0095] In some embodiments of the present invention, the object detection model structure only includes standard defect categories.

[0096] Figure 4 is a schematic diagram of undetermined defect categories according to an embodiment of the present invention. Figure 4, image block 4011 in image 401 is initially considered as a defective image block belonging to the foreign body category (so it is marked with an outer frame 4012 and is labeled as class A), and image block 4013 in image 401 is initially not considered as an image block "to be detected" (so it is not labeled) (in general operation, the executor can relatively easily determine whether the defect belongs to "to be detected" or "not to be detected", but it is not easy to determine whether it is "uncertain whether to be detected". Therefore, the executor is usually not required to determine whether it is a defect of "uncertain whether to be detected", but when the personnel cannot determine whether it is "to be detected" or "not to be detected", it will be classified as "uncertain whether to be detected"). However, the trained object detection model considers image block 4013 to be a defective image block belonging to the foreign body category, and annotates it with an outer frame and is marked as class A and displayed on the user interface 104 (as shown in image 402). The executor re-determines that image block 4013 is an uncertain image block, so the processing unit 101 designates image block 4013 as an undetermined defect category based on the executor's input (so it is marked with an outer frame 4014 and marked as class D), and the processing unit 101 stores the changed image 401 (as shown in image 403) (with the addition of the annotation that image block 4013 belongs to the undetermined defect category class D). It is worth noting that the outer frame 4014 of image block 4013 is also stored together. In the above description, images 401 to 403 are actually the same image, but in order to indicate that the annotations of their image blocks have been changed, they are represented by images 401 to 403 respectively. In the following description, the same representation will be used.

[0097] The aforementioned unclassified has multiple functions. Figure 5 is a schematic diagram of a non-classified structure according to an embodiment of the present invention. Figure 5, the image block 5011 in the image 501 is initially considered to be an actual OK image block that does not belong to the foreign body category (so it is not labeled). However, the trained object detection model considers the image block 5011 to be a defective image block that belongs to the foreign body category, and displays it on the user interface 104 with an outer frame 5012 (as shown in the image 502). The executor re-judged that the image block 5011 is an actual OK image block, so the processing unit 101, based on the input of the executor, labels the outer frame 5013 to specify the area and specifies the area surrounded by the outer frame 5013 as unclassified (labeled as class C), and the processing unit 101 deletes the outer frame 5012 of the image block 5011, and the processing unit 101 stores the changed image 501 (as shown in the image 503). In order to avoid automatically ignoring unlabeled images when executing model training, in this embodiment, a photo that has the feature that requires the object detection model to learn "that is a defect that does not need to be detected" is forcibly added to the training set for the model to learn using the aforementioned method. The aforementioned undetermined defect category and other functions without category will be further explained in the following embodiments.

[0098] Figures 6 to 10 is a schematic diagram of processing the first misjudgment set according to an embodiment of the present invention. Figures 6 to 10 , Fig. 20 as well as Fig.23 In this embodiment, when executing step S2303, the processing unit 101 displays all the false positive set images in the first false positive set on the user interface 104. In some embodiments, the processing unit 101 displays on the user interface 104 Figure 6 The recorded image 601 includes an image block 6011 originally designated as a scratch category. The image block 6011 is marked with an outer frame 6012 and is marked as class B, indicating that it is designated as a scratch category. The image block 6011 is considered to belong to the foreign object category by the trained object detection model. After re-judgment by the executor, it is considered that the original designated defect category is incorrect and the foreign object category is correct. Therefore, the executor specifies the category of the image block 6011 as the foreign object category through the user interface 104. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal. The processing unit 101 modifies the category of the image block 6011 to the foreign object category (therefore re-marking it as class A) with the modification signal received by the user interface 104, and stores the modified image 601 (as shown in image 602).

[0099] In some embodiments, the processing unit 101 displays on the user interface 104 Figure 7The recorded image 701 includes an image block 7011 originally designated as a scratch category. The image block 7011 is marked with an outer frame 7012 and is marked as class B, indicating that it is designated as a scratch category. The image block 7011 is considered to belong to the foreign body category by the trained object detection model, and is considered to be an image block of an uncertain image block after re-judgment by the executor. Therefore, the executor specifies the category of the image block 7011 as a special category of an undetermined defect category through the user interface 104. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal. The processing unit 101 modifies the category of the image block 7011 to the undetermined defect category (marked as class D) using the modification signal received by the user interface 104, and stores the modified image 701 (as shown in image 702).

[0100] In some embodiments, the processing unit 101 displays on the user interface 104 Figure 8 The recorded image 801 includes an image block 8011 originally designated as a scratch category. The image block 8011 is marked with an outer frame 8012 and is marked as class B, indicating that it is designated as a scratch category. The image block 8011 is considered to belong to the foreign body category by the trained object detection model, and is considered to be an actual OK image block after re-judgment by the executor. Therefore, the executor removes the scratch category (class B) and the outer frame 8012 originally designated for the image block 7011 through the user interface 104. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal. The processing unit 101 removes the scratch category (class B) and the outer frame 8012 originally designated for the image block 8011 using the modification signal received by the user interface 104, and stores the changed image 801 (as shown in image 802).

[0101] Figures 6 to 8 It refers to the situation where there was an annotation originally, and the model also detected it (but the detection category is different) or did not detect it, so it appeared in the first misjudgment set. After rejudgment, the personnel decided to change the original category annotation (which can be a standard defect category, uncertain image block, actual OK image block, etc.). The corresponding annotation modification process diagram.

[0102] In some embodiments, the processing unit 101 displays on the user interface 104 Fig. 9The recorded image 901 includes an image block 9011. The image block 9011 was originally designated as a scratch category (labeled as class B) and has an outer frame 9012. The trained object detection model believes that the image block 9011 belongs to the scratch category but the outer frame should be an outer frame 9013. Since the intersection and union ratio of outer frame 9012 and outer frame 9013 is less than 0.5 (which can be set to other values ​​according to requirements), the processing unit 101 is displayed on the user interface 104. After the execution personnel re-judgment, it is believed that the outer frame of the image block 9011 should be the outer frame 9013. Therefore, the execution personnel changes the outer frame of the image block 9011 to the outer frame 9013 through the user interface 104. The signal generated by the execution personnel operating the user interface 104 is the aforementioned modification signal. The processing unit 101 modifies the outer frame of the image block 9011 with the modification signal received by the user interface 104, and stores the changed image 901 (as shown in image 902).

[0103] In some embodiments, Fig.10 The recorded image 1001 includes image block 10011, image block 10012 and image block 10013. Image block 10011 was originally determined to be a foreign body category (so it was marked with an outer frame 100111 and labeled as class A). Image block 10012 and image block 10013 were not considered to belong to any of the defect categories, so they were not assigned any defect category (so they were not labeled). Image block 10011 is considered by the trained object detection model to belong to the foreign object category, image block 10012 is considered by the trained object detection model to belong to the foreign object category and is marked with an outer frame 100121 and is marked as class A by the trained object detection model, and image block 10013 is considered by the trained object detection model to belong to the foreign object category and is marked with an outer frame 100131 and is marked as class A by the trained object detection model, and the results of the above object detection model identification are displayed on the user interface 104 (as shown in image 1002). After the execution personnel re-judgment, it is considered that image block 10012 does belong to the foreign object category, so the execution personnel designate the category of image block 10012 as the foreign object category through the user interface 104 and mark it with an outer frame 100121. After review by the executor, it is considered that the image block 10013 is an uncertain image block, so the executor specifies the category of the image block 10013 as a special category of an undetermined defect category (so it is marked as class D) through the user interface 104 and marks the outer frame 100131. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal, and the processing unit 101 modifies the image 1002 with the modification signal received by the user interface 104, and stores the modified image 1002 (as shown in image 1003).

[0104] Fig.10 This indicates that there was no mark originally, but a mark is added later. The original image only marked image block 10011 (image block 10012 and image block 10013 may be image blocks that were missed or previously identified as uncertain image blocks, so they are not marked), but the object model detected two "misjudged" places, image block 10012 and image block 10013. After re-judgment, the executor determined that image block 10012 is a defect to be detected (so the standard category mark class A is added), and after re-judgment, the executor determined that image block 10013 is an uncertain image block (so the class D mark is added).

[0105] The "false positive value" and "second false positive set" described in the aforementioned step S2005 are defined in different ways according to different applications. In some embodiments of the present invention, the leak set is composed of images in the verification image that belong to the first category set but are not detected by the trained object detection model to contain defect image blocks belonging to the standard defect categories, and the overkill set is composed of images in the verification image that belong to the second category set but are detected by the trained object detection model to contain defect image blocks belonging to the standard defect categories. The false positive value is the sum of the number of leaks and the number of false positives, wherein the number of leaks is the number included in the leak set, and the number of false positives is the number included in the false positive set. Fig.24 is a flow chart of a training method according to an embodiment of the present invention. Fig.24 ,exist Fig.24 In the illustrated embodiment, the aforementioned step S2005 includes steps S2401 - S2405 .

[0106] In step S2401, the processing unit 101 inputs each verification image in the verification set into the trained object detection model, and calculates the number of missing images included in the missing set of these verification images, and calculates the number of misjudgments included in the misjudgment set of these verification images. In step S2402, the processing unit 101 sets the misjudgment value to the sum of the number of missing images and the number of misjudgments. In step S2403, the processing unit 101 determines whether the sum of the number of missing images and the number of misjudgments is zero. If the sum of the number of missing images and the number of misjudgments is zero, step S2404 is executed, and if the sum of the number of missing images and the number of misjudgments is not zero, step S2405 is executed. In step S2404, in response to the sum of the number of missing images and the number of misjudgments being zero, the processing unit 101 sets the second misjudgment set to an empty set. In step S2405, in response to the sum of the number of missing images and the number of false positives being non-zero, the processing unit 101 sets at least one false positive verification image in the verification images that belongs to the missing set or the false positive set as a second false positive set.

[0107] The definition of the aforementioned "false positive value" and "second false positive set" only requires knowing whether each verification image in the verification set belongs to the first category set or the second category set, so there is no need to label the verification images in the verification set.

[0108] Please refer to Fig. 20 In some embodiments of the present invention, the aforementioned step S2005 includes the following steps to process the second false positive set. In this step, the processing unit 101 confirms whether the second false positive set is non-empty. If the second false positive set is non-empty, the processing unit 101 executes step S2005. If the second false positive set is not non-empty (i.e., an empty set), the processing unit 101 does not process the second false positive set. In step S2005, the processing unit 101 displays the false positive verification images in the second false positive set on the user interface 104. In addition, the processing unit 101 uses at least one modification signal corresponding to at least one false positive set image received by the user interface 104 as a second input, and modifies the second false positive set based on the second input.

[0109] In some embodiments of the present invention, the processing unit 101 sequentially displays the misjudged verification images in the second misjudged set on the user interface 104. For the misjudged images belonging to the missing set, the executor re-judges the defective image block of the misjudged image that was previously classified into the first category set; if the executor re-judges that the misjudged verification image does belong to the first category set, the executor inputs a correct signal as a modification signal, and the processing unit 101 does not change the misjudged verification image after receiving the correct signal. If the executor re-judges that the misjudged verification image belongs to the third category set, the executor inputs a removal signal as a modification signal, and the processing unit 101 removes the misjudged verification image from the verification set after receiving the removal signal (for example, the processing unit 101 can achieve the removal effect by directly modifying its classification label to the third category set without directly deleting or moving it from the file level). If the executor re-judges that the misjudged verification image belongs to the second category set, the executor inputs a reclassification signal as a modification signal, and the processing unit 101 classifies the misjudged image into the second category set (for example, its classification label can be modified) after receiving the reclassification signal. It is worth reiterating that, as mentioned above, in actual operation, if the personnel's re-judgment result does not belong to the original category set, no matter whether it is re-judged into the second or third category set, the "category attribute" of the photo can be directly changed to the re-judgment result. Because when calculating the number of missed or wrongly judged images, only images from the "first category set" and "second category set" are taken, so if the image is marked as belonging to the third category set, it will naturally be excluded when performing verification next time.

[0110] For the re-judgment of the validation set (the image belongs to the first category, but the object detection model does not detect the defective area), the analysis method of the executor is to first find the image block that belongs to one of the multiple defective categories in the image. If it can be found, it means there is no problem (no need to change the category of the image). If it cannot be found, then see if the uncertain image block can be found. If it can be found (it belongs to the uncertain image block after re-judgment), then remove the photo from the validation set. If it cannot be found, then change it to the second category (it belongs to the second category after re-judgment). If the category of these images that originally belonged to the first category in the validation set is modified during the re-judgment process, because the validation set is randomly taken from the data pool, correcting the classification of the image in the validation set is equivalent to correcting the classification of the image in the data pool.

[0111] For the misjudged verification image belonging to the misjudged set, the executor re-judged at least one defective image block detected by the trained object detection model of the misjudged image; if the executor re-judged that at least one defective image block detected by the trained object detection model of the misjudged verification image is actually an OK image block, the executor inputs a correct signal as a modification signal, and the processing unit 101 does not change the misjudged verification image after receiving the correct signal. If the executor re-judged that the misjudged verification image belongs to the third category set, the executor inputs a removal signal as a modification signal, and the processing unit 101 removes the misjudged verification image from the verification set after receiving the removal signal (for example, the processing unit 101 can achieve the removal effect by directly modifying its classification label to the third category set without directly deleting or moving it from the file level). If the executor re-judges the misjudged verification image and finds that at least one defective image block detected by the trained object detection model is a defective image block according to the judgment standard, the executor inputs a motion signal as a modification signal. After receiving the motion signal, the processing unit 101 classifies the misjudged verification image into the first category set (for example, its classification label can be modified), and sets the misjudged verification image to belong to the first category set. It is worth explaining again that, as mentioned above, in actual operation, if the personnel re-judgment result does not belong to the original category set, whether it is re-judged into the first or third category set, the "category attribute" of the photo can be directly changed to the re-judgment result. Because when calculating the number of missed or wrongly judged images, only images are taken from the "first category set" and the "second category set", so if the image is marked as belonging to the third category set, it will naturally be excluded when the verification is performed next time.

[0112] For the re-judgment of the validation set (the photo belongs to the second category, but the object detection model has detected a defective area), the analysis method of the executor is to re-judge the defective area found by the object detection model. If the executor confirms that at least one defective area found by the object detection model belongs to an image block in one of the multiple standard defect categories, then the image is reclassified as the first category; if the executor confirms that at least one defective area found by the object detection model belongs to an uncertain image block (the others are actually OK image blocks), then the image is reclassified as the third category and removed from the validation set; if all defective areas found by the object detection model belong to actual OK image blocks, then the image does not need to be modified in category. Similarly, if the categories of these images that originally belonged to the second category in the validation set are modified during the re-judgment process, the corresponding images in the data pool are also modified in category synchronously.

[0113] Fig.25 is a test flow chart according to an embodiment of the present invention. Fig.25 ,exist Fig.25 In the illustrated embodiment, after the processing unit 101 obtains the main model, the processing unit 101 then performs steps S2501 to S2504 based on the main model and the test set to confirm whether to output the aforementioned main model as an offline model. In step S2501, the processing unit 101 obtains at least one test false positive value of the main model and a test false positive set in the test set based on the main model and the test set. In step S2502, the processing unit 101 determines whether at least one test false positive value satisfies the test condition, and in response to the test false positive value satisfying the aforementioned test condition, performs step S2503, and in response to at least one test false positive value not satisfying the aforementioned test condition, performs step S2504. In step S2503, the processing unit 101 outputs the aforementioned main model as an offline model. In step S2504, a sub-process is performed based on the test false positive set. The test set includes a plurality of test images, and the test images include a plurality of first test images belonging to the first category and a plurality of second test images belonging to the second category. It is worth noting that the data image block distribution of the test set represents the data distribution pattern of the actual production information of the current production line, and is independent of the aforementioned data pool. The aforementioned test conditions are to achieve an acceptable error rate for the production line.

[0114] The "at least one test misjudgment value" and the "test misjudgment set" described in the aforementioned step S2501 are defined in different ways according to different applications. In some embodiments of the present invention, the "at least one test misjudgment value" and the "test misjudgment set" are defined in the same way as the aforementioned "misjudgment value" and the "second misjudgment set", the test misjudgment value is the sum of the number of missing and the number of wrong judgments, and at least one test misjudgment image in the test image that belongs to the missing set or the wrong judgment set is set as the test misjudgment set. In some embodiments of the present invention, the processing unit 101 inputs each test image in the test set into the main model, and calculates the number of missing images included in the missing set in these test images, and calculates the number of wrong judgments included in the wrong judgment set in these test images. Then the processing unit 101 sets the test misjudgment value to the sum of the number of missing images and the number of wrong judgments. The processing unit 101 determines whether the sum of the number of missing images and the number of wrong judgments is zero. If the sum of the number of misses and the number of misjudgments is zero, then in response to the sum of the number of misses and the number of misjudgments being zero, the test misjudgment set is set to an empty set; if the sum of the number of misses and the number of misjudgments is not zero, then in response to the sum of the number of misses and the number of misjudgments being non-zero, at least one misjudgment test image in the test image that belongs to the miss set or the misjudgment set is set to the test misjudgment set.

[0115] In some embodiments of the present invention, the aforementioned test condition is that the test misjudgment value is less than a predetermined value.

[0116] In practical applications, acceptable "missing number" and "misjudgment number" can also be defined separately. The missing number represents the risk that defective products may flow out of this detection system, so the requirements are generally stricter (the value can be smaller); while the misjudgment number represents the production loss that this test system may mistakenly kill good products. Generally, as long as the requirements of the factory's production cost control are met, the requirements are generally relatively loose (the value can be larger). In some embodiments of the present invention, the aforementioned "at least one test misjudgment value" includes two test misjudgment values. For the convenience of explanation, the aforementioned two test misjudgment values ​​are respectively referred to as the first test misjudgment value and the second test misjudgment value. The first test misjudgment value is the missing number, and the second test misjudgment value is the misjudgment number, and at least one test misjudgment image in the test image that belongs to the missing set or the misjudgment set is set to the test misjudgment set. In addition, the aforementioned test conditions are that the first test misjudgment value is less than the predetermined missing number value and the second test misjudgment value is less than the predetermined misjudgment number value. Among them, the predetermined misjudgment number value can be greater than the predetermined missing number value.

[0117] In some embodiments of the present invention, the aforementioned steps include the following first step and second step: in response to the test false positive set being non-empty, the processing unit 101 processes the test false positive set based on the test verification input. In the first step, the processing unit 101 displays the false negative test images in the test false positive set on the user interface 104. In the second step, the processing unit 101 uses at least one modification signal corresponding to at least one false positive test image received by the user interface 104 as the test verification input, and processes the test false positive set based on the test verification input.

[0118] In some embodiments of the present invention, the processing unit 101 sequentially displays the false positive test images in the test false positive set on the user interface 104. For the false positive test images belonging to the missing set, the executor re-judges the defective image block of the false positive test image that was previously classified into the first category set; if the executor re-judges that the false positive test image does belong to the first category set, the executor inputs a correct signal as a modification signal, and the processing unit 101 does not change the false positive test image after receiving the correct signal. If the executor re-judges that the false positive test image does belong to the third category set, the executor inputs a removal signal as a modification signal, and the processing unit 101 removes the false positive test image from the test set after receiving the removal signal (for example, the processing unit 101 can achieve the removal effect by directly modifying its classification label to the third category set without directly deleting or moving it from the file level). If the executor re-judges that the misjudged image of the error omission test belongs to the second category set, the executor inputs a reclassification signal as a modification signal. After receiving the reclassification signal, the processing unit 101 classifies the misjudged image of the error omission test into the second category set (for example, its classification label can be modified). It is worth noting that the classification of the aforementioned misjudged image of the error omission test into the second category set does not require direct deletion or movement from the archive level, but only requires changing its classification (for example, its category label can be modified). As mentioned above, in actual operation, if the result of the personnel's re-judgment does not belong to the original category set, regardless of whether it is re-judged into the second or third category set, the "category attribute" of the photo is directly changed to the re-judgment result. Because when calculating the number of missing or misjudged images, only images are taken from the "first category set" and the "second category set", so if the image is marked as belonging to the third category set, it will naturally be excluded when the test is executed next time.

[0119] For the false positive test image belonging to the false positive set, the executor re-judges the at least one defective image block detected by the trained object detection model for the false positive test image; if the executor re-judges the at least one defective image block detected by the trained object detection model for the false positive test image to be an actual OK image block, the executor inputs a correct signal as a modification signal, and the processing unit 101 does not change the false positive test image after receiving the correct signal. If the executor re-judges the false positive test image to belong to the third category set, the executor inputs a removal signal as a modification signal, and the processing unit 101 removes the false positive test image from the test set after receiving the removal signal (for example, the processing unit 101 can achieve the removal effect by directly modifying its classification label to the third category set without directly deleting or moving it from the file level). If the executor re-judges the at least one defective image block detected by the trained object detection model of the false omission test image and finds that one of the defective image blocks is a defective image block according to the judgment standard, the executor inputs a reclassification signal as a modification signal. After receiving the reclassification signal, the processing unit 101 classifies the false omission test image into the first category set (for example, its classification label can be modified). It is worth noting that classifying the false omission test image into the first category set does not need to be directly deleted from the file level, but only needs to change its classification (for example, its classification label can be modified). As mentioned above, in actual operation, if the personnel re-judgement result does not belong to the original category set, whether it is re-judged into the first or third category set, the "category attribute" of the photo is directly changed to the re-judgement result. Because when calculating the number of missing or wrong judgments, only images are taken from the "first category set" and the "second category set", so if the image is marked as belonging to the third category set, it will naturally be excluded when the test is executed next time.

[0120] Fig.26 is a flow chart of a sub-process according to an embodiment of the present invention. Fig.26 ,exist Fig.26 In the illustrated embodiment, the aforementioned sub-process includes steps S2601 to S2604. In step S2601, the processing unit 101 selects a plurality of test misjudgment images corresponding to at least one unresolved condition from the test misjudgment set; the processing unit 101 displays the test misjudgment images, and performs a frame annotation on each test misjudgment image based on the annotation input, and performs a unresolved condition annotation process for a plurality of annotated areas of each test misjudgment image obtained when performing the frame annotation so that the annotated areas of each test misjudgment image have an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the aforementioned at least one unresolved condition.

[0121] In some embodiments of the present invention, the aforementioned at least one unresolved situation includes a loss situation and a misjudgment situation, wherein the definitions of the loss situation and the misjudgment situation are respectively the same as the definitions of the aforementioned loss set and the misjudgment set, and are not repeated herein. Fig.11 It is a schematic diagram of a process of marking a situation to be resolved according to an embodiment of the present invention. Fig.12 is a schematic diagram of a process of marking a status to be resolved according to an embodiment of the present invention. Fig.26 , Fig.11 as well as Fig.12 , the following Fig.11 as well as Fig.12 To illustrate the execution process of step S2601 in this embodiment. For the test misjudged image corresponding to the missing condition, such as Fig.11 As shown, the processing unit 101 displays a test misjudgment image 1101 corresponding to a missing condition, and the test misjudgment image 1101 includes an image block 11011 determined as a defective image block by the executor according to the determination criteria. The executor performs an outer frame annotation on the image block through the user interface 104 to annotate the outer frame 110111, and specifies that the unresolved condition of the image block 11011 is annotated as missing (annotated as class E to indicate missing). The signal generated by the executor operating the user interface 104 is the aforementioned annotation input. Based on the aforementioned annotation input, the processing unit 101 annotates the image block 11011 with the outer frame 110111, and sets the unresolved condition of the image block 11011 to be annotated as missing, and stores the changed test misjudgment image 1101 (as shown in the test misjudgment image 1102).

[0122] For the test misjudgment images corresponding to the misjudgment conditions, such as Fig.12 As shown, the processing unit 101 displays a test misjudgment image 1201 corresponding to the misjudgment condition. The test misjudgment image 1201 includes an image block 12011 that is considered by the main model to belong to the foreign body category (so it is displayed as class A to indicate the foreign body category) and is marked with an outer frame 120111 and displayed on the user interface 104 (as shown in the test misjudgment image 1201). The executor specifies that the unresolved condition of the image block 12011 is marked as a misjudgment (marked as class F to indicate a misjudgment) through the user interface 104. The signal generated by the executor operating the user interface 104 is the aforementioned marking input. Based on the aforementioned marking input, the processing unit 101 stores the outer frame 120111 of the image block 12011, sets the unresolved condition of the image block 12011 to be marked as a misjudgment, and stores the changed test misjudgment image 1201 (as shown in the test misjudgment image 1202).

[0123] Please refer to Fig.26In step S2602, the processing unit 101 trains a screening model based on the aforementioned outer frame annotation and the test misjudged image with the unresolved condition annotation until the screening model completes the screening training, wherein the structure of the screening model is the same as that of the main model, but at least one detection category of the screening model includes the aforementioned at least one unresolved condition. That is, the main model is defined as having the detection capability of detecting the aforementioned multiple defect categories (including standard defect categories and special defect categories), and the screening model is defined as having the detection capability of detecting the aforementioned at least one unresolved condition category, and the difference between the screening model and the main model is only the defined detection category. In some embodiments of the present invention, the aforementioned "training a screening model until the screening model completes the screening training" means that "the screening model can correctly detect all the annotated unresolved conditions", that is, the "missing number = 0 (the number of missing sets included = 0)" of the unresolved conditions of the test misjudged image used for training is used as the judgment basis. In some embodiments of the present invention, the aforementioned "training a screening model until the screening model completes the screening training" means that "the screening model can correctly detect all labeled pending conditions" and "will not capture unlabeled areas", that is, the screening model uses "the number of missing conditions = 0 and the number of wrong judgments = 0 (the number of missing sets = 0 and the number of wrong judgment sets = 0)" for the pending conditions of the test misjudged images used for training as the basis for judgment.

[0124] In step S2603, the processing unit 101 obtains a third false positive set in the training set based on the screening model and the training set. In response to the third false positive set being non-empty, the processing unit 101 displays the third false positive set and processes the third false positive set based on the received third input. Since the third false positive set is in the training set, processing the third false positive set is equivalent to processing the training set, so the processed training set can be obtained after processing the third false positive set.

[0125] The "third false positive set" described in the aforementioned step S2603 is defined differently depending on different applications. In some embodiments of the present invention, the aforementioned at least one condition to be resolved includes a missing condition and a false positive condition, and the "third false positive set" is defined as a screening training image in which the screening model predicts from the training set that it has an area belonging to a missing condition or an area belonging to a false positive condition. The set of screening training images in which the aforementioned screening model predicts from the training set that it has an area belonging to a missing condition is called a predicted missing set, and the set of screening training images in which the aforementioned screening model predicts from the training set that it has an area belonging to a false positive condition is called a predicted false positive set. In some embodiments of the present invention, the processing unit 101 inputs each training image in the training set into the screening model, and calculates the predicted missing number of the number of images included in the predicted missing set in these training images, and calculates the predicted false positive number of the number of images included in the predicted false positive set in these training images. The processing unit 101 determines whether the sum of the predicted missing number and the predicted false positive number is zero. If the sum of the predicted missing number and the predicted misjudgment number is zero, then in response to the sum of the predicted missing number and the predicted misjudgment number being zero, the third misjudgment set is set to an empty set; if the sum of the predicted missing number and the predicted misjudgment number is not zero, then in response to the sum of the predicted missing number and the predicted misjudgment number being non-zero, at least one screened training image in the training image that belongs to the predicted missing set or the predicted misjudgment set is set to the third misjudgment set.

[0126] In some embodiments of the present invention, the aforementioned steps include the following first step and second step: in response to the third false positive set being non-empty, the processing unit 101 processes the third false positive set based on the screening cleanup input. In the first step, the processing unit 101 displays the screening training images in the third false positive set on the user interface 104. In the second step, the processing unit 101 uses at least one modification signal corresponding to at least one screening training image received by the user interface 104 as the screening cleanup input, and processes the third false positive set based on the screening cleanup input.

[0127] In some embodiments of the present invention, the processing unit 101 sequentially displays the screening training images in the third misjudgment set on the user interface 104. For the screening training images belonging to the predicted missing set, the executor checks whether the area predicted by the screening model to have a missing condition on the screening training image overlaps with the image block originally marked as a defective image block of the screening training image. If so, the executor does not need to process it. If not, the following cleaning operation is performed: (only the "image block that is not originally marked as a defective image block" but "the area predicted by the screening model to have a missing condition" needs to be processed): (1) The area predicted by the screening model to have a missing condition is judged to be a defective image block, and the area predicted by the screening model to have a missing condition is assigned to its correct defect category. (2) The area predicted by the screening model to have a missing condition is judged to be an uncertain image block, and the area predicted by the screening model to have a missing condition is assigned to an undetermined defect category. (3) The area predicted by the screening model to have a missing condition is judged to be an actual OK image block, and the area predicted by the screening model to have a missing condition is assigned to no category.

[0128] In some embodiments of the present invention, the processing unit 101 checks whether the area predicted by the screening model to have a missing condition on the screening training image overlaps with the image block originally marked as a defective image block in the screening training image based on the following steps: the processing unit 101 calculates the intersection over union (IoU) of the area predicted by the screening model to have a missing condition on the screening training image and the image block originally marked as a defective image block in the screening training image; if the IoU calculated above is greater than a preset overlap rate (e.g., 50%), the processing unit 101 determines that the area predicted by the screening model to have a missing condition on the screening training image overlaps with the image block originally marked as a defective image block in the screening training image; if the IoU calculated above is less than or equal to the preset overlap rate (e.g., 50%), the processing unit 101 determines that there is no overlap. If the IoU calculated by the processing unit 101 is greater than the aforementioned preset overlap ratio (e.g., 50%), the processing unit 101 skips this image block, and finally the user interface 104 only displays the areas predicted by the screening model with an IoU less than or equal to the aforementioned preset overlap ratio (e.g., 50%) on the screening training image as being missing, and asks the executor to perform a re-judgment process. As long as the executor re-judges that the area predicted by the screening model on the screening training image as being missing is one of the defective image block / uncertain image block / actual OK image block, the processing unit 101 automatically completes one of the above-mentioned (1) to (3) labeling operations.

[0129] For the screening training images that belong to the predicted misjudgment set, the executor checks whether the area predicted by the screening model to be misjudged on the screening training images overlaps with the image blocks originally marked as defective image blocks in the screening training images. If not, the executor does not need to process it. If yes, the following cleaning operation is performed: (only the "image blocks originally marked as defective image blocks" and "areas predicted by the screening model to be misjudged" need to be processed): (1) After the image blocks originally marked as defective image blocks are determined to be defective image blocks, check whether "the outer frame area of ​​the original defective image block minus the overlapping part with the area predicted by the screening model" is still confirmed as a defective image block. If so, modify the outer frame area of ​​the original defective image block to the reduced area, that is, modify the outer frame area of ​​the original defective image block to the area remaining after subtracting the outer frame area of ​​the original defective image block from the overlapping part with the area predicted by the screening model. Otherwise, no processing is required. (2) The image block originally marked as a defective image block is determined to be an uncertain image block, and the image block originally marked as a defective image block is designated as an undetermined defect category. (3) The image block originally marked as a defective image block is determined to be an actual OK image block, and the defect category and outer frame of the image block originally marked as a defective image block are deleted to change it to an actual OK image block.

[0130] In some embodiments of the present invention, the processing unit 101 checks whether the area predicted by the screening model on the screening training image to be in a misjudgment condition overlaps with the image block originally labeled as a defective image block in the screening training image based on the following steps: the processing unit 101 calculates the intersection over union (IoU) of the area predicted by the screening model on the screening training image to be in a misjudgment condition and the image block originally labeled as a defective image block in the screening training image; if the IoU calculated above is greater than a preset overlap rate (e.g., 50%), the processing unit 101 determines that the area predicted by the screening model on the screening training image to be in a misjudgment condition overlaps with the image block originally labeled as a defective image block in the screening training image; if the IoU calculated above is less than or equal to the preset overlap rate (e.g., 50%), the processing unit 101 determines that there is no overlap. If the IoU calculated by the processing unit 101 is less than or equal to the aforementioned preset overlap ratio (e.g., 50%), the processing unit 101 skips this image block, and finally the user interface 104 only displays the area after "the outer frame area of ​​the original defective image block minus the overlapping part with the area predicted by the screening model" on the screening training image for the screening model whose IoU is greater than the aforementioned preset overlap ratio (e.g., 50%), and asks the executor to perform a re-judgment process. As long as the executor re-judges the area as one of the defective image block / uncertain image block / actual OK image block, the processing unit 101 automatically completes one of the above-mentioned (1) to (3) labeling operations.

[0131] Fig.13 FIG. 4 is a schematic diagram of a predicted missing set cleaning operation according to an embodiment of the present invention. Fig.14 FIG. 4 is a schematic diagram of a predicted missing set cleaning operation according to an embodiment of the present invention. Fig.15 This is a schematic diagram of a prediction misjudgment set cleaning operation according to an embodiment of the present invention. First of all, it is worth explaining that one of the uses of the aforementioned no-classification is when the screening model finds a region with a missing condition in the training image, but the operator re-judgments that it belongs to the actual OK image block and marks the region as "no-classification". Please refer to Fig.13 , the processing unit 101 displays on the user interface 104 Fig.13 The recorded screening training image 1301 includes regions 13011, 13012, and 13013 predicted by the screening model to be in a missing state, wherein regions 13011, 13012, and 13013 were originally not labeled with defect categories. Region 13011 has an outer frame 130111, region 13012 has an outer frame 130121, and region 13013 has an outer frame 130131.

[0132] After being determined to belong to the defective image block of the foreign body category, region 13011 is designated as the foreign body category (and therefore is labeled as class A to indicate the foreign body category) and its outer frame 130111 is maintained. After being determined to belong to the uncertain image block, region 13012 is designated as the undetermined defect category (and therefore is labeled as class D to indicate the undetermined defect category) and its outer frame 130121 is maintained. After being determined to belong to the actual OK image block, region 13013 is designated as no category (and therefore is labeled as class C to indicate no category) and its outer frame 130131 is maintained. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal. The processing unit 101 modifies the screening training image 1301 with the modification signal received by the user interface 104, and stores the modified screening training image 1301 (as shown in the screening training image 1302).

[0133] See also Fig.14, the training image 1401 includes an image block 14011 originally labeled as belonging to the foreign body category (hence labeled as class A to indicate the foreign body category), an image block 14012 belonging to the uncertain image block, and an image block 14013 belonging to the actual OK image block. The image block 14011 is labeled with an outer frame 140111. The processing unit 101 displays the screening training image 1402 corresponding to the training image 1401. In the screening training image 1402, the image block 14011 is included in the area predicted by the screening model to have a missing condition. Since the intersection and union ratio of the outer frame 140111 and the area of ​​the screening training image originally labeled as the defective image block is less than the preset threshold, but not 0, the processing unit 101 displays the image block 14011 and the outer frame 140112. Image block 14012 is included in the area predicted by the screening model to have a missing condition, so the processing unit 101 displays image block 14012 and outer frame 140121. Image block 14013 is included in the area predicted by the screening model to have a missing condition, so the processing unit 101 displays image block 14013 and outer frame 140131. Image block 14011 is determined to be a defective image block belonging to the foreign body category, and image block 14011 is designated as a foreign body category (therefore marked as class A to indicate the foreign body category) and its outer frame is modified to outer frame 140112. Area 14012 is determined to belong to an uncertain image block, and area 14012 is designated as an undetermined defect category (therefore marked as class D to indicate the undetermined defect category) and its outer frame 140121 is added. Region 14013 is determined to belong to the actual OK image block, and is designated as having no category (hence labeled as class C indicating having no category) and its outer frame 140131 is added (as shown in the screening training image 1403).

[0134] See also Fig.15 , the processing unit 101 displays on the user interface 104 Fig.15The recorded screening training image 1501 includes image blocks 15011, 15012, and 15013, which were originally labeled as belonging to the foreign body category (thus labeled as class A to represent the foreign body category). Image block 15011 has an original outer frame 150111, image block 15012 has an original outer frame 150121, and image block 15013 has an original outer frame 150131. Image block 15011 is determined to be a foreign body image block. After the executor checks the original annotated outer frame 150111 of image block 15011 and subtracts the overlapping part with the area 150112 predicted by the screening model, it is still confirmed that it is a foreign body image block. The outer frame of image block 15011 is modified to a reduced outer frame 150113 (at this time, the outer frame 150113 actually includes the image block belonging to the foreign body category, which is image block 15011'). Image block 15012 has the original annotated outer frame 150121 and the area 150122 predicted by the screening model, but image block 14012 is determined to be an uncertain image block. Image block 15012 is designated as an undetermined defect category, but the original annotated outer frame 150121 is maintained. The image block 15013 has an originally labeled outer frame 150131 and an area 150132 predicted by the screening model, but the image block 15013 is determined to be an actual OK image block, and the foreign body category label (class A) and the outer frame 150131 of the image block 15013 are deleted to change it to an actual OK image block. The signal generated by the executor operating the user interface 104 is the aforementioned modification signal, and the processing unit 101 modifies the screening training image 1501 with the modification signal received by the user interface 104, and stores the modified screening training image 1501 (as shown in the screening training image 1502).

[0135] Please refer to Fig.26 In step S2604, the processing unit 101 obtains an acceptable data set in the data pool based on the screening model and the data pool. In response to the acceptable data set being non-empty, the processing unit 101 displays and processes the currently displayed acceptable image based on the fourth input for each of at least one acceptable image in the acceptable data set. The processing unit 101 integrates the processed training set and the acceptable data set as the training set, and returns to step S2001 to start again. In some embodiments, the aforementioned integration of the processed training set and the acceptable data set as the training set means using the union of the processed training set and the acceptable data set as the training set.

[0136] The "acceptable data set" described in the aforementioned step S2604 is defined in different ways according to different applications. In some embodiments of the present invention, the aforementioned at least one unresolved condition includes a missing condition and a misjudgment condition, and the aforementioned method of obtaining an acceptable data set in the data pool based on the screening model and the data pool includes the following first step, second step, and third step: In the first step, the processing unit 101 inputs all first images in the first category set into the screening model, and collects first condition images in the first category set that are predicted by the screening model to have areas belonging to the missing condition. In response to the set of first condition images being non-empty, the processing unit 101 displays each first condition image on the user interface 104. If the executor determines that the area in the currently displayed first condition image that is predicted by the screening model to have an area belonging to the missing condition is a defective area, the processing unit 101 classifies the currently displayed first condition image into the "acceptable data set", and if not, the processing unit 101 classifies the currently displayed first condition image into the "unacceptable data set".

[0137] Fig.16 FIG. 1 is a schematic diagram of acceptable data screening according to an embodiment of the present invention. Fig.16 For a description of the first step, see Fig.16 , the processing unit 101 displays the first condition image 1601 on the user interface 104, wherein the first condition image 1601 includes an image block 16011 belonging to an actual OK image block and an area 16012 predicted by the screening model to have a missing condition, the area 16012 is marked as class E to indicate missing, and the area 16012 has an outer frame 160121. The executor determines that the area 16012 predicted by the screening model to have a missing condition in the currently displayed first condition image 1601 belongs to one of the multiple defect categories, so the processing unit 101 classifies the currently displayed first condition image 1601 into the "acceptable data set". The processing unit 101 displays the first condition image 1602 on the user interface 104, wherein the first condition image 1602 includes an image block 16021 that is a defective image block and an area 16022 that is predicted by the screening model to have a missing condition, the area 16022 is labeled as class E to indicate missing, and the area 16022 has an outer frame 160221. The executor determines that the area 16022 that is predicted by the screening model to have a missing condition in the currently displayed first condition image 1602 is not a defective area, so the processing unit 101 classifies the currently displayed first condition image 1602 into an "unacceptable data set".

[0138] In the second step, the processing unit 101 inputs all the second images in the second category set into the screening model, and collects the second condition images in the second category set that are predicted by the screening model to have areas belonging to the misjudgment condition. In response to the set composed of the second condition images being non-empty, the processing unit 101 displays each second condition image on the user interface 104. If the executor determines that the area in the second condition image currently displayed that is predicted by the screening model to have the misjudgment condition belongs to the actual OK image block and all the image blocks of the second condition image currently displayed belong to the actual OK image blocks, the processing unit 101 classifies the second condition image currently displayed into the "acceptable data set". If not, the processing unit 101 classifies the second condition image currently displayed into the "unacceptable data set".

[0139] Fig.17 FIG. 1 is a schematic diagram of acceptable data screening according to an embodiment of the present invention. Fig.17 For a description of the second step, see Fig.17, the processing unit 101 displays the second status image 1701 on the user interface 104, wherein the second status image 1701 includes an image block 17011 belonging to the actual OK image block and an area 17012 predicted by the screening model to have a misjudgment condition, the area 17012 is labeled as class F to indicate misjudgment, and the area 17012 has an outer frame 170121. The executor determines that the area 17012 predicted by the screening model to have a misjudgment condition in the currently displayed second status image 1701 belongs to the actual OK image block, and all the image blocks of the currently displayed second status image belong to the actual OK image block, therefore, the processing unit 101 classifies the currently displayed second status image 1701 into the "acceptable data set". The processing unit 101 displays the second status image 1702 on the user interface 104, wherein the second status image 1702 includes an image block 17021 that belongs to the actual OK image block and an area 17022 that is predicted by the screening model to have a misjudged condition, the area 17022 is labeled as class F to indicate a misjudgment, and the area 17022 has an outer frame 170221. It is worth noting that, in theory, the second status image 1702 should not appear in the actual OK category of the data pool, but in actual situations, the data pool may still be mixed with a small number of images belonging to the third category, so this example is a processing method assuming that this situation occurs. The executor determines that the area 17022 that is predicted by the screening model to have a misjudged condition in the currently displayed second status image 1702 does not belong to the actual OK image block, so the processing unit 101 classifies the currently displayed second status image 1702 into the "unacceptable data set". The processing unit 101 displays the second status image 1703 on the user interface 104, wherein the second status image 1703 includes an image block 17031 that is an uncertain image block after review by the executor and an area 17032 that is predicted by the screening model to have a misjudged status. The area 17032 is marked as class F to indicate a misjudgment, and the area 17032 has an outer frame 170321. It is worth noting that, in theory, the second status image 1703 should not appear in the actual OK category of the data pool, but in actual situations, the data pool may still be mixed with a small number of images belonging to the third category, so this example is a processing method assuming that this situation occurs. Although the executor determines that the area 17032 that is predicted by the screening model to have a missing status in the currently displayed second status image 1703 is an actual OK image block, the image block 17031 does not belong to the actual OK image block, so the processing unit 101 classifies the currently displayed second status image 1703 into the "unacceptable data set".

[0140] In the third step, the processing unit 101 takes the aforementioned “acceptable data set” from the data pool.

[0141] In some embodiments of the present invention, a fourth step is further included after the third step. In the fourth step, the processing unit 101 displays the aforementioned "unacceptable data set" on the user interface 104. If the executor finds that a certain image in the "unacceptable data set" has a classification judgment standard problem of actual OK / uncertain image block / defective image block, the classification of this image in the data pool is confirmed and corrected (for example, although the second status image 1703 is taken out from the second data set, after the executor's re-judgment, the second status image 1703 includes an image block 17031 that belongs to an uncertain image block after the executor's re-judgment and an image block 17032 that belongs to an actual OK image block. Therefore, the second status image 1703 belongs to the third category set. Based on the executor's re-judgment, the processing unit 101 corrects the second status image 1703 to belong to the third category set).

[0142] Please refer to Fig.26 ,exist Fig.26 In the illustrated embodiment, the aforementioned step S2604 further includes: the processing unit 101 displays at least one detection area detected by the screening model for each of the at least one acceptable image in the aforementioned acceptable data set, and performs a defect category labeling process on the at least one detection area based on the fourth input, so that the at least one detection area of ​​each of the at least one acceptable image has a defect category label, wherein the defect category label corresponds to one of the defect categories.

[0143] In some embodiments of the present invention, the aforementioned at least one unresolved situation includes a missing situation and a misjudgment situation. The processing unit 101 displays at least one detection area detected by the screening model for each of the at least one acceptable image in the aforementioned acceptable data set on the user interface 104. The executor performs a defect category labeling process on the at least one detection area through the user interface 104, so that at least one detection area of ​​each of the at least one acceptable image has a defect category labeling. The signal generated by the executor operating the user interface 104 is the aforementioned fourth input.

[0144] In this embodiment, for the acceptable images in the acceptable data set that belong to the first condition, the correct defect category is assigned to the area predicted by the screening model to belong to the missing condition, and at the same time, it is checked whether there is a defective image block in the area predicted by other non-screening models to belong to the missing condition. If there is, the correct defect category is also assigned. For the acceptable images in the acceptable data set that belong to the second condition, the area other than the area predicted by the screening model to belong to the misjudgment condition is assigned to a special defect category without a category.

[0145] Fig.18FIG. 4 is a schematic diagram of a process of labeling image defect categories that are acceptable for a missing condition according to an embodiment of the present invention. Fig.19 FIG. 1 is a schematic diagram of a misjudged image defect classification labeling process according to an embodiment of the present invention. Fig.18 as well as Fig.19 Take the example to illustrate the above steps. Fig.18 In the illustrated embodiment, the processing unit 101 displays an acceptable image 1801 belonging to the first condition image on the user interface 104, wherein the acceptable image 1801 includes a region 18012 that is predicted by the screening model to have a missing condition and is classified as a foreign object after review by the operator, the region 18012 is labeled as class E to indicate missing, and the region 18012 has an outer frame 180121. The acceptable image 1801 also includes an image block 18011 that is predicted by the non-screening model to have a missing condition but is classified as a foreign object after review by the operator. The operator assigns the correct defect class to the region 18012 as a foreign object class (labeled as class A) and retains its outer frame 180121 (as shown in the acceptable image 1802), and the operator labels the outer frame 180111 on the region image block 18011 and assigns the correct defect class to the foreign object class (labeled as class A) (as shown in the acceptable image 1803).

[0146] One occasion when the aforementioned unclassified image is used is when the screening model predicts an area with an overkill condition (that is, to solve the overkill problem, the screening model selects an image from the data pool that is recommended to be added to the training set), and the executor re-judges that the area belongs to the actual OK image block. The executor deletes the outer frame and does not give any label to the original area, and designates the area other than the area predicted by the screening model to be an overkill condition as a special defect category without a category. Fig.19 In the illustrated embodiment, the processing unit 101 displays an acceptable image 1901 belonging to the second condition image on the user interface 104, wherein the acceptable image 1901 includes an area 19011 predicted by the screening model to have a misjudgment condition, the area 19011 is labeled as class F to indicate misjudgment, and the area 19011 has an outer frame 190111. The executor re-judged that the area 19011 belongs to the actual OK image block, the executor deleted the outer frame 19011 and did not give any label to the original area 19011, and the executor designated the area 19012 other than the area 19011 predicted by the screening model to have a misjudgment condition as a special defect category without a category (as shown in the acceptable image 1902).

[0147] Although in the aforementioned embodiment, the processing unit 101 obtains the main model and then obtains the test misjudgment value of the main model and the test misjudgment set in the test set based on the main model and the test set. And in response to the test misjudgment value satisfying a test condition, the processing unit 101 outputs the aforementioned main model as an off-line model. However, the aforementioned process may be performed by another system.

[0148] Please refer to Figure 1 In some embodiments of the present invention, the test system is implemented as follows Figure 1 The electronic device 100 is shown. The test system itself stores a test set. The processing unit 101 is configured to perform: receiving a main model, and storing the received main model in a non-volatile memory 103 for subsequent use and processing; obtaining a test misjudgment value of the main model and a test misjudgment set in the test set based on the main model and the test set; in response to the misjudgment value satisfying the test condition, outputting the main model as an offline model; and in response to the test misjudgment value not satisfying the aforementioned test condition, outputting the main model and the test misjudgment set, wherein the test set comprises a plurality of test images, the test images comprising a plurality of first test images belonging to a first category and a plurality of second test images belonging to a second category, wherein, as described above, the first image belonging to the first category indicates that the image is determined to contain at least one defective image block belonging to one of the plurality of standard defect categories, and the image belonging to the second category indicates that the image is determined not to contain an image block belonging to one of the defect categories. In some embodiments of the present invention, the processing unit 101 outputs the main model and the test misjudgment set back to the training system to continue to execute the aforementioned sub-process.

[0149] The processing unit 101 may receive at least one modification signal corresponding to at least one falsely identified image in the falsely identified test set from the user interface 104 as a test verification input, and process the falsely identified test set based on the test verification input. The various embodiments of the functions performed by the test system may refer to the aforementioned embodiments, which will not be described in detail herein.

[0150] Although in the aforementioned embodiment, the training system executes a sub-process based on the test false positive set to update the training set in response to the test false positive value not satisfying the test condition, the aforementioned sub-process may be executed by another system.

[0151] Please refer to Figure 1 In some embodiments of the present invention, the data screening system is implemented as follows Figure 1The electronic device 100 is shown. The processing unit 101 is configured to perform: receiving a main model, a data pool, a training set, and a test misjudgment set, and storing the received main model and the test misjudgment set in a non-volatile memory 103 for subsequent use and processing, the processing unit 101 stores the received data pool and training set in a large memory space, and accesses the data pool and the training set data according to the relevant position when needed, and loads the required data into the internal memory for subsequent processing; selecting a plurality of test misjudgment images corresponding to at least one unresolved condition from the test misjudgment set; displaying the test misjudgment images, and annotating each test misjudgment image with a frame based on the annotation input, and performing an unresolved condition annotation process for a plurality of annotated areas of each test misjudgment image obtained when annotating the frame so that the unresolved condition annotation of the plurality of annotated areas of each test misjudgment image The annotation area has an annotation of a condition to be resolved, wherein the annotation of the condition to be resolved corresponds to one of the at least one condition to be resolved; based on the test misjudged images, a screening model is trained until the screening model completes screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes at least one condition to be resolved; based on the screening model and the training set, a misjudgment set in the training set is obtained, and in response to the misjudgment set being non-empty, the misjudgment set is displayed and processed based on a third input; and based on the screening model and the data pool, an acceptable data set is obtained in the data pool, and in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, a currently displayed acceptable image is displayed and processed based on a fourth input; and the processed misjudgment set and the acceptable data set are integrated as the training set.

[0152] The processing unit 101 uses the user interface 104 to display the aforementioned various images, and receives various operator operation signals received by the user interface 104 to process the aforementioned various data. The various embodiments of the functions executed by the data screening system can refer to the aforementioned embodiments, which will not be repeated here.

[0153] In some embodiments of the present invention, the aforementioned offline model will be output as an online object detection model deployed in the factory production environment to actually perform the detection work. The online object detection model obtained based on the aforementioned embodiment has the detection capabilities required for general production line production data, but as time changes, the factory production environment may derive data with "variant forms". At this time, it is necessary to retrain the online object detection model deployed in the factory production environment with new training set data so that the original offline model can have the ability to respond to data with "variant forms".

[0154] In some embodiments of the present invention, the executive personnel perform a certain ratio of random inspection and re-judgment on the photos that have been detected by the online object detection model (i.e., the current offline model) in the actual production environment according to the production line quality control standards, and determine whether it is necessary to start the model retraining process. The executive personnel divide the sampled re-inspected photos into two types according to the re-judgment results: images with "misjudgment problems" (also called incorrectly judged images) and images with "no misjudgment problems" (also called correctly judged images). The set of images with "no misjudgment problems" is called the online correct set, and the set of images with "misjudgment problems" is called the online misjudgment set, which is further divided into two categories: missing and wrongly judged. The images in the missing category are images that the offline model determines to have no defective image blocks, but the executive personnel re-judgment finds that there are defective image blocks. The images in the wrongly judged category are images that the offline model determines to have defective image blocks, but the executive personnel re-judgment does not find that there are defective image blocks.

[0155] The aforementioned online misjudgment set is combined with the test set originally used to test the offline model into an online test set.

[0156] A portion of the aforementioned online misjudgment set constitutes the initial "retrain set", that is, the retrain set contains some members of the online misjudgment set. In some embodiments of the present invention, the online misjudgment set is randomly divided into 5 parts, and then one of the parts is taken as the initial "retrain set". The initial "retrain set" is annotated by the execution personnel (same as the aforementioned Fig.18 as well as Fig.19 The annotation process shown) is combined with the training set of the original offline model training to form the current training set.

[0157] The aforementioned online correct set is added to the original data pool according to the actual OK image blocks / uncertain image blocks / defective image blocks to form an online data pool, so the online data pool still has the first category set, the second category set and the third category set. The online verification set is composed of an appropriate number (e.g., 100) of images randomly selected from the first category set and the second category set of the online data pool.

[0158] In some embodiments of the present invention, the training system uses the current training set as the training set, replaces the original data pool with the online data pool, and replaces the validation set with the online validation set and then executes Fig. 20 The illustrated steps S2001 to S2007 are used to obtain a main model, which is used as an online candidate main model. The implementation details of steps S2001 to S2007 can be referred to the above description, and will not be repeated here.

[0159] After obtaining the main model as the online candidate main model, the online candidate main model must also go through a testing process, similar to the training process of the offline model. Fig. 27is a flow chart of an online model training method according to some embodiments of the present invention. Fig. 27 In the illustrated embodiment, after obtaining the main model as the online candidate main model (hereinafter referred to as the main model), it further includes steps S2701 to S2707. In step S2701, the processing unit 101 obtains at least one online test misjudgment value of the main model and an online test misjudgment set in the online test set based on the main model and the online test set. In step S2702, the processing unit 101 determines whether at least one online test misjudgment value satisfies the online test condition. If the online test misjudgment value satisfies the online test condition, step S2703 is executed, otherwise step S2704 is executed. In step S2703, the main model is output as an updated online object detection model. In step S2704, the processing unit 101 uses the main model to determine whether there is a re-misjudgment set that the main model will misjudge among all members of the online misjudgment set after removing the initial retraining set. In step S2705, if the re-false positive set is not empty, the process proceeds to step S2706. If the re-false positive set is an empty set, the process proceeds to step S2707. In step S2706, the processing unit 101 uses the re-false positive set as a retraining set and the online main model as an updated online object detection model, and then returns to step S2001 to start over. In step S2707, the processing unit 101 executes the online sub-process based on the online test false positive set.

[0160] The various embodiments of the functions performed by the aforementioned steps S2701 to S2707 can refer to the aforementioned related embodiments and will not be repeated here.

[0161] Fig.28 is a flowchart of an online sub-process according to some embodiments of the present invention. Fig.28In the illustrated embodiment, the aforementioned online sub-process includes steps S2801 to S2804. In step S2801, the processing unit 101 selects a plurality of online test misjudgment images corresponding to at least one unresolved condition from the online test misjudgment set; the processing unit 101 displays the online test misjudgment images, and performs frame annotation on each online test misjudgment image based on the annotation input, and performs a unresolved condition annotation process for the plurality of annotated areas of each online test misjudgment image obtained when performing the frame annotation so that the annotated areas of each online test misjudgment image have unresolved condition annotations, wherein the unresolved condition annotation corresponds to one of at least one unresolved condition. In step S2802, the processing unit 101 trains an online screening model based on the online test misjudgment images until the online screening model completes the online screening training, wherein the architecture of the online screening model is the same as the online main model, and at least one detection category of the online screening model includes at least one unresolved condition. In step S2803, the processing unit 101 obtains a third misjudgment set in the online training set based on the online screening model and the online training set, and in response to the third misjudgment set being non-empty, displays and processes the third misjudgment set based on the third input to obtain a processed training set. In step S2804, the processing unit 101 obtains an acceptable data set in the online data pool based on the online screening model and the online data pool, and in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, displays and processes the currently displayed acceptable image based on the fourth input; and integrates the processed training set and the acceptable data set as the online training set, and returns to step S2001 to restart training.

[0162] Fig.29 FIG. 1 is a flow chart of a training method according to an embodiment of the present invention. Fig.29 What is illustrated is an embodiment in which the previously disclosed steps are integrated to obtain an offline model. In this embodiment, the processing unit 101 executes steps S2001 to S2007 to obtain a master model. After obtaining the master model, the processing unit 101 executes steps S2501 to S2504 and S2601 to S2604 to obtain an offline model when the test error value satisfies the test condition. It can be seen from the above process that steps S2001 to S2007 may be repeated several times before the test error value satisfies the test condition. The master model obtained by each execution of steps S2001 to S2007 is called a generation of master models, and multiple generations of master models may be generated before the offline model is obtained.

[0163] Fig.30 is a flow chart of an online training method according to an embodiment of the present invention. Fig.30The illustrated embodiment integrates the previously disclosed steps to obtain an updated online object detection model. In this embodiment, the processing unit 101 executes steps S2001 to S2007 to obtain a main model. After obtaining the main model, the processing unit 101 executes steps S2701 to S2706 and S2801 to S2804 to obtain an updated online object detection model when the online test misjudgment value meets the online test conditions.

[0164] Please refer to Figure 1 The internal memory 102 is, for example, a random access memory (RAM). The non-volatile memory is, for example, at least one disk storage. Of course, the electronic device 100 may also include hardware required for other functions.

[0165] The processing unit 101 may include processors 101-1 to 101-R, where R is a positive integer. The processors 101-1 to 101-R may be an integrated circuit chip with signal processing capabilities. In the implementation process, the methods and steps disclosed in the aforementioned embodiments can be completed by hardware integrated logic circuits or software instructions in the processors 101-1 to 101-R. The processors 101-1 to 101-R may be general-purpose processors, including central processing units (CPU), tensor processing units (Tensor Processing Unit), digital signal processors (Digital Signal Processor, DSP), application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA), graphics processing units (Graphic Processing Unit, GPU) or other programmable logic devices, which can implement or execute the methods and steps disclosed in the aforementioned embodiments.

[0166] The embodiments of this specification also provide a computer-readable storage medium, which stores at least one instruction. When the at least one instruction is executed by the processing unit 101 of the electronic device 100, the processing unit 101 of the electronic device 100 can execute the methods and steps disclosed in the aforementioned embodiments.

[0167] Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other internal memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory media such as modulated data signals and carrier waves.

[0168] Based on the above, some embodiments of the present invention provide a training system, a training method, a testing system, a testing method, a data screening system, a data screening method, and a computer-readable recording medium with a stored program. By processing and modifying the training set, the domain knowledge of the executor can be continuously integrated to improve the quality of the training set. By continuously judging whether the object detection model has completed the training set learning, it can be ensured that the model has passed the underfitting stage. In addition, by using the verification set to judge whether the object detection model is overfitting, the object detection model in the best training state can be obtained before the object detection model is overtrained as the main model. The test set is used to verify whether the optimized model trained using the current training set meets the application requirements. When the "optimized model trained using the current training set" does not meet the application requirements, the secondary process provides a standardized method of "correcting the training set label" and "adding training set data" to correct a more appropriate training set. After that, the training set can be re-entered into the offline / online training process to retrain or correct a model that better meets the test set data requirements. By finding acceptable data sets, the training set can be automatically analyzed to find the causes of misjudgment in the test set. The program algorithm process, combined with the existing domain knowledge of the executors, can be used to automatically correct existing training set labeling problems and introduce missing new data to complete the data set required for the next training.

[0169] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Any person skilled in the art may make some changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be based on that defined in the claims.

Claims

1. A training system comprising at least one processing unit and an object detection model; the at least one processing unit being configured to perform: (a) before the object detection model completes the training set learning, repeatedly executing: training the object detection model based on the training set for the current number of times, wherein the training set includes a plurality of training images, each of the training images includes at least one annotated outer frame, and the annotated area and the non-annotated area formed by the at least one annotated outer frame of each training image are used to train the object detection model; obtaining a first misjudgment set in the training set based on the trained object detection model and the training set; Modify the training set based on the first misjudgment set; and increase the current number; as well as (b) obtaining an error value of the object detection model and a second error set in the verification set based on the trained object detection model and each of a plurality of verification images in the verification set; The verification set is modified based on the second false positive set; in response to the false positive value satisfying a condition, the current number is increased and steps (a) to (b) are executed; and in response to the false positive value not satisfying the condition, the previous training model is output as the main model.

2. The training system according to claim 1, wherein: The training images are selected from a first category set in a data pool, and the verification images are selected from the first category set and the second category set in the data pool, wherein the first category set includes a plurality of first images belonging to a first category, and the second category set includes a plurality of second images belonging to a second category, wherein the image belonging to the first category indicates that the image is determined to include at least one defective image block belonging to one of a plurality of standard defect categories, and the image belonging to the second category indicates that the image is determined to not include an image block belonging to one of the standard defect categories, and does not include an image block that cannot be determined whether it belongs to one of the standard defect categories; the at least one defective image block of each of the training images is marked with an outer frame.

3. The training system of claim 2, wherein the data pool comprises a third category set, the third category set comprising a plurality of third images, each of the third images not including an image block belonging to one of the standard defect categories but including an image block that cannot be determined whether it belongs to one of the standard defect categories.

4. The training system of claim 2, wherein the standard defect categories include a foreign matter category and a scratch category.

5. The training system of claim 2, wherein the aforementioned step (a) comprises the following steps to determine whether the object detection model has completed the learning of the training set and obtained the first false positive set in the training set: Inputting each of the training images into the trained object detection model, and calculating the number of misses included in the miss set of the at least one defective image block of all the training images that is not detected by the trained object detection model; In response to the number of misses being zero, determining that the object detection model has completed the training set learning, and setting the first misjudgment set to an empty set; as well as In response to the miss count being non-zero, determining that the object detection model has not completed the training set learning, and setting at least one misjudged training image having a member of the miss set in the training images as the first misjudgment set.

6. The training system of claim 2, wherein the aforementioned step (a) comprises the following steps to determine whether the object detection model has completed the training set learning and obtained the first false positive set in the training set: Inputting each of the training images into the trained object detection model, and calculating the number of misses included in the miss set of the at least one defect image block of all the training images that is not detected by the trained object detection model, and calculating the number of errors included in the error set of the at least one defect image block of all the training images that is not labeled as belonging to one of the standard defect categories but is detected by the trained object detection model; In response to the sum of the number of misses and the number of errors being zero, determining that the object detection model has completed the training set learning, and setting the first misjudgment set to an empty set; as well as In response to the sum of the number of misses and the number of errors being non-zero, determining that the object detection model has not completed the training set learning, and setting at least one misjudged training image in the training image having a member of the miss set or a member of the error set as the first misjudgment set.

7. The training system of claim 1, wherein the training system comprises a user interface, and the aforementioned step (a) comprises the following steps: In response to the first false positive set being non-empty, executing: displaying at least one false positive set image in the first false positive set on the user interface; and At least one modification signal corresponding to the at least one false positive set image is received through the user interface, and the training set is modified based on the modification signal.

8. The training system of claim 2, wherein the aforementioned step (b) comprises the following steps to obtain the misjudgment value of the object detection model and the second misjudgment set in the validation set based on the trained object detection model and the validation set: Input each of the verification images into the trained object detection model, and calculate the number of missing images included in the missing set of the verification image, and calculate the number of misjudgments included in the misjudgment set of the verification image, wherein the missing set is composed of images in the verification image that belong to the first category set but are not detected by the trained object detection model as including defect image blocks belonging to the standard defect category, and the misjudgment set is composed of images in the verification image that belong to the second category set but are detected by the trained object detection model as including defect image blocks belonging to the standard defect category; and The misjudgment value is set to the sum of the number of misses and the number of misjudgments; in response to the sum of the number of misses and the number of misjudgments being zero, the second misjudgment set is set to an empty set; and in response to the sum of the number of misses and the number of misjudgments being non-zero, at least one misjudged verification image in the verification image that belongs to the miss set or the misjudgment set is set to the second misjudgment set.

9. The training system of claim 1, wherein the training system comprises a user interface, and the aforementioned (b) comprises the following steps: In response to the second false positive set being non-empty, executing: displaying at least one false positive set image in the second false positive set on the user interface; and At least one second modification signal corresponding to the at least one false positive set image is received through the user interface, and the verification set is modified based on the at least one second modification signal.

10. The training system of claim 2, wherein the at least one processing unit is configured to perform: Based on the main model and the test set, a test false positive value of the main model and a test false positive set in the test set are obtained; in response to the test false positive value satisfying a test condition, the main model is output as an offline model; and in response to the test false positive value not satisfying the test condition, a process is executed based on the test false positive set; wherein the test set includes a plurality of test images, and the test images include a plurality of first test images belonging to the first category and a plurality of second test images belonging to the second category.

11. The training system of claim 10, wherein the at least one processing unit is configured to perform the following steps to execute the process: comprising: (d1) selecting a plurality of test misjudgment images corresponding to at least one condition to be solved from the test misjudgment set; Displaying the test misjudged images, and annotating each of the test misjudged images with an outer frame based on the annotation input, and performing a pending condition annotation process for a plurality of annotated regions of each of the test misjudged images obtained when performing the outer frame annotation so that the annotated region of each of the test misjudged images has a pending condition annotation, wherein the pending condition annotation corresponds to one of the at least one pending condition; (d2) training a screening model based on the test misjudgment image until the screening model completes screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes the at least one unresolved condition; (d3) obtaining a third false positive set in the training set based on the screening model and the training set, and in response to the third false positive set being non-empty, displaying and processing the third false positive set based on a third input to obtain the processed training set; and (d4) based on the screening model and the data pool, obtaining an acceptable data set in the data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, displaying and processing the currently displayed acceptable image based on a fourth input; and integrating the processed training set and the acceptable data set as the training set, and executing steps (a) to (b).

12. The training system of claim 11, wherein the aforementioned step (d4) comprises: For each of the at least one acceptable image in the acceptable data set, the at least one detection region detected by the screening model is displayed, and a defect category labeling process is performed on the at least one detection region based on the fourth input, so that the at least one detection region of each of the at least one acceptable image has a defect category labeling, wherein the defect category labeling corresponds to one of a plurality of defect categories, wherein the defect categories include the standard defect category and a plurality of special defect categories.

13. The training system of claim 11, wherein the at least one condition to be resolved includes a missing condition and a misjudgment condition, wherein: The missing condition refers to the situation where the defect image blocks in the test image that belong to the first category set but are not detected by the main model include those belonging to the standard defect category. The misjudgment condition refers to the situation where the defect image blocks in the test image that belong to the second category set but are detected by the main model include those belonging to the standard defect category.

14. The training system of claim 1, wherein the condition is that the misjudgment value is not equal to zero and the misjudgment value is less than or equal to a previous misjudgment value; and the object detection model is an instance segmentation model.

15. The training system of claim 1, wherein the training set comprises an original training set and a retraining set, the validation set is selected from an online data pool, the online data pool comprises an original data pool and an online correct set, the retraining set comprises a portion of an online misjudgment set, wherein the online correct set is composed of a plurality of correctly judged images correctly judged by the initial object detection model, and the online misjudgment set is composed of a plurality of incorrectly judged images incorrectly judged by the initial object detection model.

16. The training system of claim 15, wherein the at least one processing unit is configured to perform: Based on the main model and the online test set, an online test misjudgment value of the main model and an online test misjudgment set in the online test set are obtained; in response to the online test misjudgment value satisfying the online test condition, the main model is output as an updated online object detection model; and in response to the online test misjudgment value not satisfying the online test condition, the main model is used to determine whether there is a re-misjudgment set that the main model will misjudge among all members of the online misjudgment set after removing the initial retraining set; in response to the re-misjudgment set being non-empty, steps (a) to (b) are performed using the re-misjudgment set as the retraining set and the main model as the updated object detection model; and in response to the re-misjudgment set being an empty set, an online sub-process is performed based on the online test misjudgment set; wherein the online test set includes all members of the original test set and the online misjudgment set.

17. The training system of claim 16, wherein the at least one processing unit is configured to perform the following steps to execute the online sub-process: Selecting a plurality of online test misjudgment images corresponding to at least one unresolved condition from the online test misjudgment set; displaying the online test misjudgment images, and annotating each of the online test misjudgment images with an outer frame based on the annotation input, and performing an unresolved condition annotation process on a plurality of annotated areas of each of the online test misjudgment images obtained when performing the outer frame annotation so that the annotated area of ​​each of the online test misjudgment images has an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the at least one unresolved condition; Based on the online test misjudged images, training an online screening model until the online screening model completes online screening training, wherein the architecture of the online screening model is the same as that of the main model, and at least one detection category of the online screening model includes the at least one unresolved condition; Obtaining a third misjudgment set in the training set based on the online screening model and the training set, and in response to the third misjudgment set being non-empty, displaying and processing the third misjudgment set based on a third input; and Based on the online screening model and the online data pool, an acceptable data set is obtained in the online data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, the currently displayed acceptable image is displayed and processed based on a fourth input; and the processed third false positive set and the acceptable data set are integrated as the training set, and steps (a) to (b) are performed.

18. A training method, adapted for training an object detection model, executed by at least one processing unit, the training method comprising: (a) before the object detection model completes the training set learning, repeatedly executing: training the object detection model based on the training set for the current number of times, wherein the training set includes a plurality of training images, each of the training images includes at least one annotated outer frame, and the annotated area and the non-annotated area formed by the at least one annotated outer frame of each training image are used to train the object detection model; obtaining a first misjudgment set in the training set based on the trained object detection model and the training set; Modify the training set based on the first misjudgment set; and increase the current number; as well as (b) obtaining an error value of the object detection model and a second error set in the verification set based on the trained object detection model and each of a plurality of verification images in the verification set; The verification set is modified based on the second false positive set; in response to the false positive value satisfying a condition, the current number is increased and steps (a) to (b) are executed; and in response to the false positive value not satisfying the condition, the previous training model is output as the main model.

19. The training method according to claim 18, wherein: The training images are selected from a first category set in a data pool, and the verification images are selected from the first category set in the data pool and a second category set in the data pool, wherein the first category set includes a plurality of first images belonging to a first category; the second category set includes a plurality of second images belonging to a second category, wherein the image belonging to the first category indicates that the image is determined to include at least one defective image block belonging to one of a plurality of standard defect categories, and the image belonging to the second category indicates that the image is determined to not include an image block belonging to one of the standard defect categories, and does not include an image block that cannot be determined whether it belongs to one of the standard defect categories; the at least one defective image block of each of the training images is marked with an outer frame.

20. The training method of claim 19, wherein the data pool comprises a third category set, the third category set comprises a plurality of third images, each of the third images does not include an image block belonging to one of the standard defect categories but includes an image block that cannot be determined whether it belongs to one of the standard defect categories.

21. The training method of claim 19, wherein the standard defect categories include a foreign matter category and a scratch category.

22. The training method of claim 19, wherein the aforementioned step (a) comprises the following steps to determine whether the object detection model has completed the learning of the training set and obtained the first false positive set in the training set: Inputting each of the training images into the trained object detection model, and calculating the number of misses included in the miss set of the at least one defective image block of all the training images that is not detected by the trained object detection model; In response to the number of misses being zero, determining that the object detection model has completed the training set learning, and setting the first misjudgment set to an empty set; as well as In response to the miss count being non-zero, determining that the object detection model has not completed the training set learning, and setting at least one misjudged training image having a member of the miss set in the training images as the first misjudgment set.

23. The training method of claim 19, wherein the aforementioned step (a) comprises the following steps to determine whether the object detection model has completed the learning of the training set and obtained the first false positive set in the training set: Inputting each of the training images into the trained object detection model, and calculating the number of misses included in the miss set of the at least one defect image block of all the training images that is not detected by the trained object detection model, and calculating the number of errors included in the error set of the at least one defect image block of all the training images that is not labeled as belonging to one of the standard defect categories but is detected by the trained object detection model; In response to the sum of the number of misses and the number of errors being zero, determining that the object detection model has completed the training set learning, and setting the first misjudgment set to an empty set; as well as In response to the sum of the number of misses and the number of errors being non-zero, determining that the object detection model has not completed the training set learning, and setting at least one misjudged training image in the training image having a member of the miss set or a member of the error set as the first misjudgment set.

24. The training method of claim 18, wherein the aforementioned step (a) comprises the following steps: In response to the first false positive set being non-empty, executing: Displaying at least one false positive set image in the first false positive set on a user interface; and At least one modification signal corresponding to the at least one misjudged set image is received through the user interface, and the training set is modified based on the modification signal.

25. The training method of claim 19, wherein the aforementioned step (b) comprises the following steps to obtain the misjudgment value of the object detection model and the second misjudgment set in the validation set based on the trained object detection model and the validation set: Input each of the verification images into the trained object detection model, and calculate the number of missing images included in the missing set of the verification image, and calculate the number of misjudgments included in the misjudgment set of the verification image, wherein the missing set is composed of images in the verification image that belong to the first category set but are not detected by the trained object detection model as including defect image blocks belonging to the standard defect category, and the misjudgment set is composed of images in the verification image that belong to the second category set but are detected by the trained object detection model as including defect image blocks belonging to the standard defect category; and The misjudgment value is set to the sum of the number of misses and the number of misjudgments; in response to the sum of the number of misses and the number of misjudgments being zero, the second misjudgment set is set to an empty set; and in response to the sum of the number of misses and the number of misjudgments being non-zero, at least one misjudged verification image in the verification image that belongs to the miss set or the misjudgment set is set to the second misjudgment set.

26. The training method according to claim 18, wherein the aforementioned (b) comprises the following steps: In response to the second false positive set being non-empty, executing: Displaying at least one false positive set image in the second false positive set on a user interface; and At least one second modification signal corresponding to the at least one false positive set image is received through the user interface, and the verification set is modified based on the at least one second modification signal.

27. The training method of claim 19, wherein the training method comprises: Based on the main model and the test set, obtain a test misjudgment value of the main model and a test misjudgment set in the test set; In response to the test false positive value satisfying a test condition, outputting the main model as an offline model, and in response to the test false positive value not satisfying the test condition, executing a process based on the test false positive set; The test set includes a plurality of test images, wherein the test images include a plurality of first test images belonging to the first category and a plurality of second test images belonging to the second category.

28. The training method according to claim 27, wherein the process comprises: (d1) selecting a plurality of test misjudgment images corresponding to at least one condition to be solved from the test misjudgment set; Displaying the test misjudged images, and annotating each of the test misjudged images with an outer frame based on the annotation input, and performing a pending condition annotation process for a plurality of annotated regions of each of the test misjudged images obtained when performing the outer frame annotation so that the annotated region of each of the test misjudged images has a pending condition annotation, wherein the pending condition annotation corresponds to one of the at least one pending condition; (d2) training a screening model based on the test misjudgment image until the screening model completes screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes the at least one unresolved condition; (d3) obtaining a third false positive set in the training set based on the screening model and the training set, and in response to the third false positive set being non-empty, displaying and processing the third false positive set based on a third input to obtain the processed training set; and (d4) based on the screening model and the data pool, obtaining an acceptable data set in the data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, displaying and processing the currently displayed acceptable image based on a fourth input; and integrating the processed training set and the acceptable data set as the training set, and executing steps (a) to (b).

29. The training method according to claim 28, wherein the aforementioned step (d4) comprises: For each of the at least one acceptable image in the acceptable data set, the at least one detection region detected by the screening model is displayed, and a defect category labeling process is performed on the at least one detection region based on the fourth input, so that the at least one detection region of each of the at least one acceptable image has a defect category labeling, wherein the defect category labeling corresponds to one of a plurality of defect categories, wherein the defect categories include the standard defect category and a plurality of special defect categories.

30. The training method of claim 28, wherein the at least one condition to be solved comprises a missing condition and a misjudgment condition, wherein: The missing condition refers to the situation where the defect image blocks in the test image that belong to the first category set but are not detected by the main model include those belonging to the standard defect category. The misjudgment condition refers to the situation where the defect image blocks in the test image that belong to the second category set but are detected by the main model include those belonging to the standard defect category.

31. The training method of claim 18, wherein the condition is that the misjudgment value is not equal to zero and the misjudgment value is less than or equal to a previous misjudgment value; and the object detection model is an instance segmentation model.

32. The training method of claim 18, wherein the training set comprises an original training set and a retraining set, the validation set is selected from an online data pool, the online data pool comprises an original data pool and an online correct set, the retraining set comprises a portion of an online misjudgment set, wherein the online correct set is composed of a plurality of correctly judged images correctly judged by the initial object detection model, and the online misjudgment set is composed of a plurality of incorrectly judged images incorrectly judged by the initial object detection model.

33. The training method of claim 32, wherein the training method comprises: Based on the main model and the online test set, an online test misjudgment value of the main model and an online test misjudgment set in the online test set are obtained; In response to the online test misjudgment value satisfying the online test condition, outputting the main model as an updated online object detection model, and in response to the online test misjudgment value not satisfying the online test condition, using the main model to determine whether there is a re-misjudgment set that will be misjudged by the main model among all members of the online misjudgment set after removing the initial retraining set, in response to the re-misjudgment set being non-empty, using the re-misjudgment set as the retraining set and using the main model as the updated object detection model to perform steps (a) to (b), and in response to the re-misjudgment set being an empty set, performing an online sub-process based on the online test misjudgment set; The online test set includes the original test set and all members of the online misjudgment set.

34. The training method of claim 33, wherein the online sub-process comprises: Selecting a plurality of online test misjudgment images corresponding to at least one condition to be solved from the online test misjudgment set; Display the online test misjudged images, and perform outer frame annotation on each of the online test misjudged images based on the annotation input, and perform a pending condition annotation process for a plurality of annotated areas of each of the online test misjudged images obtained when performing the outer frame annotation so that the annotated area of ​​each of the online test misjudged images has a pending condition annotation, wherein the pending condition annotation corresponds to one of the at least one pending condition; Based on the online test misjudged images, training an online screening model until the online screening model completes online screening training, wherein the architecture of the online screening model is the same as that of the main model, and at least one detection category of the online screening model includes the at least one unresolved condition; Obtaining a third misjudgment set in the training set based on the online screening model and the training set, and in response to the third misjudgment set being non-empty, displaying and processing the third misjudgment set based on a third input; and Based on the online screening model and the online data pool, an acceptable data set is obtained in the online data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, the currently displayed acceptable image is displayed and processed based on a fourth input; and the processed third false positive set and the acceptable data set are integrated as the training set, and steps (a) to (b) are performed.

35. A testing system comprising: at least one processing unit configured to execute; Based on the main model and the test set, obtain a test misjudgment value of the main model and a test misjudgment set in the test set; In response to the test false positive value satisfying the test condition, the main model is output as the offline model; and in response to the test false positive value not satisfying the test condition, the main model and the test false positive set are output, wherein the test set includes a plurality of test images, the test images including a plurality of first test images belonging to a first category and a plurality of second test images belonging to a second category, wherein the image belonging to the first category indicates that the image is determined to include at least one defective image block belonging to one of the plurality of defect categories, and the image belonging to the second category indicates that the image is determined not to include an image block belonging to one of the defect categories.

36. A testing method, performed by at least one processing unit, comprising: Based on the main model and the test set, obtain a test misjudgment value of the main model and a test misjudgment set in the test set; In response to the test false positive value satisfying the test condition, the main model is output as the offline model; and in response to the test false positive value not satisfying the test condition, the main model and the test false positive set are output, wherein the test set includes a plurality of test images, the test images including a plurality of first test images belonging to a first category and a plurality of second test images belonging to a second category, wherein the image belonging to the first category indicates that the image is determined to include at least one defective image block belonging to one of the plurality of defect categories, and the image belonging to the second category indicates that the image is determined not to include an image block belonging to one of the defect categories.

37. A data screening system comprising: at least one processing unit configured to execute; Receive the main model, data pool, training set, and test misclassification set; Selecting a plurality of test misjudgment images corresponding to at least one unresolved condition from the test misjudgment set; displaying the test misjudgment images, and annotating each of the test misjudgment images with an outer frame based on an annotation input, and performing an unresolved condition annotation process on a plurality of annotated regions of each of the test misjudgment images obtained when performing the outer frame annotation so that the annotated region of each of the test misjudgment images has an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the at least one unresolved condition; Based on the test misjudgment image, training a screening model until the screening model completes screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes the at least one unresolved condition; Obtaining a misjudgment set in the training set based on the screening model and the training set, and in response to the misjudgment set being non-empty, displaying and processing the misjudgment set based on a first input to obtain the processed training set; and Based on the screening model and the data pool, an acceptable data set is obtained in the data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, the currently displayed acceptable image is displayed and processed based on a second input; and the processed training set and the acceptable data set are integrated as the training set.

38. A data screening method, performed by at least one processing unit, comprising: Receive the main model, data pool, training set, and test misclassification set; Selecting a plurality of test misjudgment images corresponding to at least one unresolved condition from the test misjudgment set; displaying the test misjudgment images, and annotating each of the test misjudgment images with an outer frame based on the annotation input, and performing an unresolved condition annotation process on a plurality of annotated regions of each of the test misjudgment images obtained when performing the outer frame annotation so that the annotated region of each of the test misjudgment images has an unresolved condition annotation, wherein the unresolved condition annotation corresponds to one of the at least one unresolved condition; Based on the test misjudgment image, training a screening model until the screening model completes screening training, wherein the architecture of the screening model is the same as that of the main model, and at least one detection category of the screening model includes the at least one unresolved condition; Obtaining a misjudgment set in the training set based on the screening model and the training set, and in response to the misjudgment set being non-empty, displaying and processing the misjudgment set based on a first input to obtain the processed training set; and Based on the screening model and the data pool, an acceptable data set is obtained in the data pool; in response to the acceptable data set being non-empty, for each of at least one acceptable image in the acceptable data set, the currently displayed acceptable image is displayed and processed based on a second input; and the processed training set and the acceptable data set are integrated as the training set.

39. A computer-readable recording medium having a stored program, which, when at least one processor loads and executes the stored program, completes the method according to any one of claims 18 to 34, 36, and 38.

Images