Model training method, defect type determination method and device, equipment

Abstract

The present application relates to a model training method, a defect type determination method and device, and equipment. The above method includes using the first sample set to train to obtain a benchmark model; for each second sample image in the second sample set, using the benchmark model to obtain a defect prediction result corresponding to the second sample image, and using the defect prediction result to expand at least one second-class object defect label carried by the second sample image to obtain a defect label expansion result; based on the defect label expansion result corresponding to each second sample image, the second sample set is updated to obtain a third sample set; and the feature extraction network is trained using the third sample set to obtain a pre-trained model. The present application is conducive to improving the adaptability of the pre-trained model obtained by training. The embodiments of the present application can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and smart entertainment.

Description

Technical Field

[0001] The present application relates to the field of Internet communication technology, and in particular to a model training method, a defect type determination method and device, and equipment. Background Art

[0002] With the development of Internet communication technology, target models can be used to predict defects in input images, which improves the convenience of determining defect types. The acquisition of target models depends on pre-trained models. In related technologies, the training of pre-trained models focuses on their classification capabilities for general classification data sets. Such pre-trained models will limit the feature extraction capabilities of the corresponding target models, thereby affecting the target model's defect prediction capabilities for images containing areas showing specific objects, such as specific objects in the industrial field. Summary of the invention

[0003] In order to solve at least one of the above-mentioned technical problems, the present application provides a model training method, a defect type determination method and device, and an apparatus:

[0004] According to a first aspect of the present application, a model training method is provided, the method comprising:

[0005] A benchmark model is obtained by training with a first sample set, the benchmark model is used to output a defect type for an input image, the benchmark model includes a feature extraction network, the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class of object defect label;

[0006] For each second sample image in the second sample set, using the benchmark model to obtain a defect prediction result corresponding to the second sample image, and using the defect prediction result to expand at least one second-category object defect label carried by the second sample image to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object;

[0007] Update the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set;

[0008] The feature extraction network is trained using the third sample set to obtain a pre-trained model.

[0009] According to a second aspect of the present application, a defect type determination method is provided, characterized in that the method comprises:

[0010] Acquire an image to be processed, wherein the image to be processed includes an area showing a first category of objects;

[0011] Defect prediction is performed on the image to be processed using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-training model described in the first aspect.

[0012] According to a third aspect of the present application, a model training device is provided, the device comprising:

[0013] A first training module: used to obtain a benchmark model by training with a first sample set, wherein the benchmark model is used to output a defect type for an input image, wherein the benchmark model includes a feature extraction network, wherein the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class of object defect label;

[0014] A label expansion module: used for obtaining, for each second sample image in the second sample set, a defect prediction result corresponding to the second sample image using the reference model, and expanding at least one second-category object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object;

[0015] A sample set updating module: used for updating the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set;

[0016] The second training module is used to train the feature extraction network using the third sample set to obtain a pre-trained model.

[0017] According to a fourth aspect of the present application, a defect type determination device is provided, the device comprising:

[0018] An acquisition module: used for acquiring an image to be processed, wherein the image to be processed includes an area showing an object of the first category;

[0019] Prediction module: used to perform defect prediction on the image to be processed using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-training model as described in the first aspect.

[0020] According to the fifth aspect of the present application, an electronic device is provided, comprising at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the model training method as described in the first aspect, or the defect type determination method as described in the second aspect.

[0021] According to the sixth aspect of the present application, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or at least one program is loaded and executed by a processor to implement the model training method as described in the first aspect, or the defect type determination method as described in the second aspect.

[0022] According to the seventh aspect of the present application, a computer program product is provided, which includes at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the model training method as described in the first aspect, or the defect type determination method as described in the second aspect.

[0023] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.

[0024] Implementing this application has the following beneficial effects:

[0025] The present application provides a training scheme for a pre-trained model, which is beneficial to improving the adaptability of the pre-trained model obtained through training. In the process of training to obtain the pre-trained model, the first sample set is first used to train to obtain a baseline model; then the baseline model is used to predict defects for the second sample set, and the defect labels originally carried by the second sample set are expanded to obtain a third sample set; furthermore, the feature extraction network in the baseline model is trained using the third sample set to obtain the pre-trained model. The pre-trained model starts training based on the defect type of the first category of objects, and uses samples related to the second category of objects to improve the feature extraction capability of the baseline model. This is beneficial to improving the feature extraction capability of the pre-trained model for images containing areas showing the first category of objects, making it more targeted. Accordingly, the target defect type determination model obtained based on the pre-trained model is more accurate when predicting defects in images containing areas showing the first category of objects.

[0026] Other features and aspects of the present application will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0028] Figure 1 A schematic diagram of an application environment according to an embodiment of the present application is shown;

[0029] Figure 2 A schematic diagram showing a flow chart of a model training method according to an embodiment of the present application is shown;

[0030] Figure 3 A schematic diagram showing a process of obtaining defect label expansion results according to an embodiment of the present application;

[0031] Figure 4 A schematic diagram of a process for determining a second type of object according to an embodiment of the present application is shown;

[0032] Figure 5 A schematic diagram showing a process of training the feature extraction network using a third sample set to obtain a pre-trained model according to an embodiment of the present application;

[0033] Figure 6 A schematic diagram showing a flow chart of a defect type determination method according to an embodiment of the present application;

[0034] Figure 7 A schematic diagram showing a process of obtaining a third sample set according to an embodiment of the present application is shown;

[0035] Figure 8 A schematic diagram showing a process of obtaining a target defect type determination model according to an embodiment of the present application;

[0036] Fig. 9 A schematic diagram showing comparison of experimental indicators according to an embodiment of the present application is shown;

[0037] Fig.10 A block diagram of a model training device according to an embodiment of the present application is shown;

[0038] Fig.11 A block diagram showing a defect type determination device according to an embodiment of the present application;

[0039] Fig.12 A schematic diagram of an electronic device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] Various exemplary embodiments, features and aspects of the present application will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0043] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0044] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0045] In addition, in order to better illustrate the present application, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present application can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present application.

[0046] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0047] Parts overkill rate: The numerator is the number of qualified (ok) parts that are misjudged as unqualified (ng) parts by the algorithm model. The denominator is the total number of parts tested or the total number of qualified parts tested.

[0048] Parts missed detection rate: The numerator is the number of unqualified (ng) parts that are misjudged as qualified (ok) parts by the algorithm model. The denominator is the total number of parts tested or the total number of unqualified parts tested.

[0049] Acceptance criteria: The criteria used to determine whether a defect is acceptable. Defects that do not meet the criteria are considered unqualified products and need to be reworked or scrapped.

[0050] See also Figure 1 , Figure 1 A schematic diagram of an application environment according to an embodiment of the present application is shown, and the application environment may include a client 10 and a server 20. The client 10 and the server 20 may be directly or indirectly connected via wired or wireless communication. The user sends a defect type determination request to the server 20 via the client 10. The server 20 obtains an image to be processed based on the received defect type determination request, and the image to be processed includes an area showing a first type of object; then, a target defect type determination model is used to predict defects in the image to be processed, and the target defect type determination model is obtained based on a pre-trained model. The pre-trained model is obtained by training through the following steps: first, the first sample set is used to train to obtain a benchmark model, the benchmark model is used to output the defect type for the input image, the benchmark model includes a feature extraction network, the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class object defect label; then, for each second sample image in the second sample set, the benchmark model is used to obtain a defect prediction result corresponding to the second sample image, and the defect prediction result is used to expand at least one second class object defect label carried by the second sample image to obtain a defect label expansion result, the second sample set indicates a second class of objects, and the second class of objects have an association with the first class of objects; further, the second sample set is updated based on the defect label expansion result corresponding to each second sample image to obtain a third sample set; then, the feature extraction network is trained using the third sample set to obtain the pre-trained model. It should be noted that Figure 1 Just an example.

[0051] The client 10 may be a physical device such as a smart phone, a computer (such as a desktop computer, a tablet computer, a laptop computer), an augmented reality (AR) / virtual reality (VR) device, a digital assistant, an intelligent voice interaction device (such as an intelligent speaker), an intelligent wearable device (such as a smart watch), a smart home appliance, a vehicle-mounted terminal, etc., or may be software running in a physical device, such as a computer program. The operating system of the client 10 may be an Android system (Android system), an iOS system (a mobile operating system developed by Apple), a Linux system (an operating system), a Microsoft Windows system (Microsoft Windows operating system), etc.

[0052] The server end 20 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The server can include a network communication unit, a processor, a memory, etc. The server end 20 can provide background services for the corresponding client.

[0053] In practical applications, the server end that obtains the pre-trained model through the model training method provided in the embodiment of the present application may be server end A, and the server end that processes the image to be processed through the defect type determination method provided in the embodiment of the present application may be server end B. Server end A and server end B may indicate the same server end or two different server ends. In addition, the client may also use the locally stored target defect type determination model to process the image to be processed.

[0054] The model training method and / or defect type determination method provided in the embodiments of the present application can be used in technologies related to cloud computing and computer vision (CV). Cloud computing is a computing model that distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called a "cloud". The resources in the "cloud" are infinitely expandable in the eyes of users, and can be obtained at any time, used on demand, expanded at any time, and paid for by use. As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established, and various types of virtual resources will be deployed in the resource pool for external customers to choose to use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to the use of cameras and computers to replace human eyes to identify and measure targets, and further perform graphics processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in the visual field such as swin-transformer, ViT, V-MOE, and MAE have been fine-tuned (fine tune) can be quickly and widely applied to specific downstream tasks. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0055] It should be noted that for the first sample set, the second sample set, the third sample set, etc. that are associated with user information, when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0056] Figure 2 A flow chart of a model training method according to an embodiment of the present application is shown as follows: Figure 2 As shown, the method includes:

[0057] S201: obtaining a benchmark model by training with a first sample set, the benchmark model being used to output a defect type for an input image, the benchmark model comprising a feature extraction network, the first sample set indicating a first class of objects, and each first sample image in the first sample set carrying at least one first class of object defect label;

[0058] In an embodiment of the present application, the server side uses the first sample set to train and obtain a benchmark model. The model training method provided in the embodiment of the present application is used to train and obtain a pre-trained model, and the subsequent target defect type determination model is obtained based on the pre-trained model. The target defect type determination model is used to predict defects in images containing areas showing first-class objects. Accordingly, the pre-trained model is also for the first-class objects, and the benchmark model obtained by training here also focuses on the defect types of the first-class objects. The benchmark model can be obtained by training based on the first sample set and adjusting the parameters of the preset machine learning model during training. The training focuses on enabling the benchmark model to have defect prediction capabilities. The training focuses on enabling the benchmark model to determine whether the image defect belongs to the defect type of the first-class object; if so, the specific defect type is determined. The determined defect type corresponds one-to-one to at least one first-class object defect label. Accordingly, the benchmark model needs to be able to extract features from the image and classify defects based on the extracted features. The benchmark model can extract features from the image through the feature extraction network therein, and the benchmark model can classify defects based on the extracted features through the first classification network therein.

[0059] The first sample set indicates a first type of object, and the first sample set includes a plurality of first sample images, each of which may include an area showing the first type of object. The first sample image may be obtained by photographing the first type of object by an image acquisition device. The first type of object may be mass-produced by industrial means. For example, the first type of object may be a SIM card holder, and the first type of object may be a mobile phone frame.

[0060] The first sample set may be the result of an image acquisition device photographing a plurality of first-category objects. 1) The first sample set includes a plurality of first sample images, each of which may indicate a first-category object. For example, the plurality of first-category objects are N first-category objects, and the first-category object i is the i-th first-category object among the N first-category objects. The plurality of first sample images are N first sample images, and the first sample image i is the i-th first sample image among the N first sample images, where i is 1-N. Then, the first sample image i indicates the first-category object i. 2) The first sample set includes a plurality of first sample images, and there may be at least two first sample images indicating the same first-category object among the plurality of first sample images.

[0061] Each first sample image in the first sample set carries at least one first-class object defect label. If there is one first-class object defect label, then each first sample image carries the one first-class object defect label. If there are at least two first-class object defect labels, the defect label carried by each first sample image may be at least one of the at least two first-class object defect labels.

[0062] Taking the first type of object, which is a sim card holder, as an example, a factory has produced multiple sim card holders. Multiple sim card holders are of the same model and material. Based on the acceptance rules (i.e., acceptance standards), it is determined that N of them need to be reworked or scrapped. The acceptance standards record 4 defect types. These N sim card holders all have defects. For each of the N sim card holders, it can have one defect and the defect belongs to any of the four defect types; it can have at least two defects, and the at least two defects belong to the same defect type; it can have at least two defects, and the at least two defects belong to different defect types. For each of the N sim card holders, the sim card holder can be photographed using an image acquisition device, and more than one image can be obtained for a sim card holder, and the image obtained by the shooting includes an area showing the sim card holder.

[0063] S202: for each second sample image in the second sample set, using the benchmark model to obtain a defect prediction result corresponding to the second sample image, and using the defect prediction result to expand at least one second-category object defect label carried by the second sample image to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object;

[0064] In an embodiment of the present application, for each second sample image in the second sample set, the server side uses the benchmark model to obtain the defect prediction result corresponding to the second sample image, and expands at least one second-class object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result. It can be understood that the second sample set is defect predicted using the benchmark model, and the defect label originally carried by the second sample image is expanded. If the second sample set includes M second sample images, the second sample image j is the jth second sample image among the M second sample images, and j is 1-M. At least one first-class object defect label corresponds to defect type ac. At least one second-class object defect label carried by the second sample image j corresponds to defect type de. The second sample image j is input into the benchmark model, and the benchmark model outputs the defect prediction result for the second sample image j. The defect prediction result is the defect type for the first-class object, that is, defect type ac. If it is determined that the image defect of the second sample image j belongs to defect type ac, the defect label originally carried by the second sample image j can be expanded based on the defect type ac. As a result, the second sample image j carries a second-class object defect label indicating defect type de and a first-class object defect label indicating defect type ac. Of course, the defect prediction result can also determine that the image defects of the second sample image j belong to at most two of the defect types ac, and thereby expand the defect label originally carried by the second sample image j.

[0065] The second sample set indicates a second class of objects, and the second sample set includes a plurality of second sample images, each of which may include an area showing the second class of objects, and each of which carries at least one second class of object defect label. For obtaining the second sample images, the relationship between the second class of objects, the second sample images, and the second class of object defect labels, or between any two of them, can refer to the related records of the first sample set in the aforementioned step S201, and will not be repeated here.

[0066] The second type of object has an association relationship with the first type of object. The second type of object can also be mass-produced through industrial means. The association relationship between the second type of object and the first type of object can be in terms of object type, such as the second type of object and the first type of object both belong to the mobile phone frame type; or in terms of object material, such as the second type of object and the first type of object use the same metal material.

[0067] Compared with the annotation of the defect label of the first category object based on the first acceptance rule, the annotation of the defect label of the second category object is based on the second acceptance rule. There are differences between the second acceptance rule and the first acceptance rule. Exemplarily, 1) Both acceptance rules record defect A, but the two acceptance rules have different definitions of defect A, which can be reflected in the form of the described defect A. For defect A in the first acceptance rule, it can correspond to the first category object defect label a; for defect A in the second acceptance rule, it can correspond to the second category object defect label b. 2) The first acceptance rule records the definition of defect B, and the second acceptance rule records the definition of defect C, but the two definitions are the same. For defect B in the first acceptance rule, it can correspond to the first category object defect label c; for defect C in the second acceptance rule, it can correspond to the second category object defect label d.

[0068] S203: updating the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set;

[0069] In an embodiment of the present application, the server updates the second sample set based on the defect label expansion result corresponding to each second sample image to obtain a third sample set. Combined with the example in the aforementioned step S202, each second sample image in the second sample set carries at least one second-class object defect label. Each second sample image in the third sample set not only carries at least one second-class object defect label, but also carries at least one first-class object defect label. Due to the defect prediction result for the second sample image j output by the benchmark model, it can also be determined that the image defect of the second sample image j does not belong to the defect type ac, that is, the image defect of the second sample image j does not belong to the defect type of the first-class object. The defect label carried by such a second sample image j does not need to be expanded, it still carries at least one second-class object defect label, and it is not a constituent element of the third sample set. The second sample set includes M second sample images. If, according to the defect prediction of the benchmark model, there are K second sample images whose image defects do not belong to the defect type of the first-class object, then the number of second sample images constituting the third sample set is MK.

[0070] S204: Using the third sample set to train the feature extraction network to obtain a pre-trained model.

[0071] In the embodiment of the present application, the server uses the third sample set to train the feature extraction network to obtain a pre-trained model. Combined with the record in the aforementioned step S201, the feature extraction network focuses on extracting features from the image, and it is expected that the first classification network can accurately classify defects related to the first class of objects based on the extracted features. When the feature extraction network is trained using the third sample set, the first class of object defect labels and the second class of object defect labels carried by the second sample image are used as constraints to optimize the feature extraction ability of the feature extraction network. Accordingly, the feature extraction network is trained to obtain a pre-trained model, and the features extracted from the image by the pre-trained model need to be used to achieve defect classification related to the first class of objects and the second class of objects. Compared with the feature extraction network, the pre-trained model has stronger feature extraction and feature expression capabilities for images. From the purpose of the benchmark model, the third sample set can be regarded as images from real application scenarios, and the image information carried by these images is more complex. At the same time, considering the association between the second class of objects and the first class of objects, the first defect class information (corresponding to the first class of object defect labels) provided by the first sample image is also associated with the second defect class information (corresponding to the second class of object defect labels) provided by the second sample image. Using the third sample set as training data can improve the ability of the feature extraction network to capture features involved in defect classification of the first class of objects from complex image information. The feature extraction network can effectively distinguish and identify the first defect class information and the second defect class information in the second sample image.

[0072] The following is a detailed description of the pre-trained model obtained through training. Model training is ongoing, and the pre-trained model can be the result of the previous training or the basis for the next training.

[0073] In one embodiment, Figure 3 As shown, for the aforementioned step S202, the defect prediction result includes the predicted probability of each first-class object defect label corresponding to the second sample image, and the defect prediction result is used to expand at least one second-class object defect label carried by the second sample image to obtain a defect label expansion result, including:

[0074] S301: Obtaining a probability threshold;

[0075] S302: for the predicted probability corresponding to each first-category object defect label, when the predicted probability is greater than the probability threshold, determining that the first-category object defect label is an expanded label, so as to obtain at least one of the expanded labels;

[0076] S303: Obtain the defect label expansion result based on the at least one expanded label and the at least one second-category object defect label.

[0077] Here, a method for expanding the defect labels originally carried by the second sample image is provided. The second sample set can be regarded as a candidate set, which provides images for mining the defect types of the first type of objects. This can increase the number of sample images that participate in subsequent training to obtain the pre-trained model. The second type of object defect labels originally carried by the second sample image are also conducive to improving the quality of sample images that participate in subsequent training to obtain the pre-trained model.

[0078] Exemplarily, there are 4 first-class object defect labels, namely, defect labels ad. Each first-class object defect label corresponds to a defect type of the first-class object. There are 2 second-class object defect labels, namely, defect labels ef. Each second-class object defect label corresponds to a defect type of the second-class object. The second sample set includes M second sample images, and the second sample image m is any second sample image among the M second sample images. At least one second-class object defect label carried by the second sample image m is a defect label e.

[0079] The second sample image m is input into the benchmark model, and the benchmark model outputs the defect prediction result for the second sample image m. The defect prediction result includes the prediction probabilities of the second sample image m corresponding to the defect labels ad, such as the prediction probability of 60% corresponding to the defect label a, 10% corresponding to the defect label b, 70% corresponding to the defect label c, and 80% corresponding to the defect label d. When the probability threshold is 15%, the defect labels a, c, and d can be three expanded labels of the second sample image m. Thus, the defect label expansion result of the second sample image m is composed of defect labels a, c, d, and e.

[0080] In practical applications, for the predicted probability output by the subsequent target defect type determination model, a probability lower limit can be used to characterize the defect type, that is, when the predicted probability is greater than the probability lower limit, the image defect is determined to belong to a defect type of a first-class object. Of course, the subsequent target defect type determination model can also output a defect type qualitative result based on the probability lower limit. The probability threshold used above is generally less than the probability lower limit here, and the value range of the probability threshold can be 10%-20%.

[0081] In addition, different probability thresholds and probability lower limits may be set for different first-category object defect labels (corresponding to defect types of the first-category objects) according to actual feedback.

[0082] In one embodiment, Figure 4 As shown, the method may further include the step of determining the second category of objects:

[0083] S401: Determine object information of the first type of object, where the object information includes description information of multiple dimensions;

[0084] S402: Determine a first dimension and a second dimension from the multiple dimensions;

[0085] S403: Based on the description information matching results of the first category dimension and the description information matching results of the second category dimension, determine the second category object from multiple candidate objects, the description information matching degree between the second category object and the first category object in the first category dimension is greater than a first threshold, and the description information matching degree between the second category object and the first category object in the second category dimension is less than a second threshold.

[0086] The association relationship between the second category object and the first category object is established based on the matching result between the object information. Here, the basis for establishing the association relationship is provided, which is conducive to effectively locating the second category object from multiple candidate objects when the first category object is known, thereby ensuring that the second sample set is timely involved in the training process to obtain the pre-trained model.

[0087] Taking the first type of object as the anchor point, the multiple dimensions involved in the object information are divided around how to better describe the first type of object. The first type of dimension and the second type of dimension involve different matching functions. When establishing an association relationship, at the first type of dimension level, the matching degree of the relevant description information needs to be large enough; at the second type of dimension level, the matching degree of the relevant description information needs to be small enough. Therefore, there are parts that are similar enough to the second type of object and the first type of object, and there are also parts that are significantly different. Generally speaking, the first threshold is greater than or equal to the second threshold.

[0088] For the first dimension and the second dimension, the first dimension may include at least one dimension, and the second dimension may include at least one dimension. The union of the first dimension and the second dimension may be the above-mentioned multiple dimensions themselves, or may be a part of the above-mentioned multiple dimensions. For the calculation of the description information matching degree at the level of a certain type of dimension (such as the first dimension and the second dimension), if the dimension includes one dimension, then the description information of the first object and the second object in the dimension is determined respectively, and then the similarity of the two description information is calculated to obtain the description information matching degree. If the dimension includes at least two dimensions, then for each dimension, the description information of the first object and the second object in the dimension is determined respectively, and then the similarity of the two description information is calculated to obtain the similarity corresponding to each dimension; then the description information matching degree is obtained based on the similarity corresponding to each dimension, and the mean, median, etc. of these similarities can be taken as the description information matching degree, or the weight coefficient of each dimension can be introduced to obtain the description information matching degree.

[0089] Furthermore, the first category of objects is a product composed of at least one industrial part, and the multiple dimensions include a product name dimension, a product material dimension, a product version dimension, and a product origin dimension. Determining the first category of dimensions and the second category of dimensions from the multiple dimensions may include the following two methods: 1) determining that the product name dimension and the product material dimension are the first category of dimensions, and determining that the product version dimension is the second category of dimensions; or, 2) determining that the product name dimension and the product material dimension are the first category of dimensions, and determining that the product origin dimension is the second category of dimensions.

[0090] Combined with the specificity of products composed of industrial parts, a specific dimension division method is provided, which can improve the pertinence and adaptability of determining the second type of objects. The description information of the product name dimension can limit the type of object and even the style of the object. The description information of the product material dimension can limit the material used by the object. The product version dimension can limit the version and model of the object. The product origin dimension can limit the factory information where the object is produced, such as the factory identification, the equipment information used by the factory to produce the object, etc. Taking the product name dimension and the product material dimension as the first type of dimension, it can guide the parts of the first type of object and the second type of object that are sufficiently similar to be limited to the material used by the object, the type of the object, and even the style of the object, so as to ensure that the defects of the two types of objects can have a certain degree of similarity. If the materials, types of objects, and even styles of objects used by the first type of object and the second type of object produced by industrial means are significantly different, then the defects of the two types are likely to point to different directions. Taking the product version dimension or the product origin dimension as the second type of dimension, it can guide the parts of the first type of object and the second type of object that have obvious differences to be limited to the version, model, or factory where the object is produced, so as to ensure that the differences in the defects of the two types of objects are within a controllable range. This is helpful to improve the contribution of the second sample image to training.

[0091] In actual applications, in the current Industry 4.0 automated quality inspection, it is often necessary to copy and migrate automated quality inspection solutions. In the replication scenario of the same material and product, but across models and factories, since the products are the same and the materials are the same, there are conditions for replication and migration. However, due to differences in models and differences in processes in different factories, there are huge differences in defect distribution locations, defect morphology, etc., and even different types of defects may occur. If the same set of detection algorithms is reused directly, there will often be a large number of overkills and missed detections, as well as complete missed detections on some new defects. By obtaining a pre-trained model through the model training method provided in the embodiment of the present application, the data accumulated from previous projects can be used to improve the performance indicators and convergence speed of the replicated project.

[0092] In addition, the first category of objects and the second category of objects respectively indicate the current version and historical version of the same product, and the method also includes the step of obtaining the second sample set: first, obtaining a plurality of historical sample images used for training a historical defect type determination model, the historical sample images including an area showing the second category of objects, and the plurality of historical sample images are annotated with defect labels based on product acceptance rules for the historical versions; then, determining that the historical sample image is the second sample image, and determining that the defect label carried by the historical sample image is the at least one second category object defect label.

[0093] For different versions of the same product, the existing historical sample images can be used without labeling defects for the historical sample images, which can improve the utilization rate of the historical sample images and the efficiency of training to obtain the pre-trained model.

[0094] For example, the first type of object is the current version of the sim card holder, and the second type of object is the previous version of the sim card holder. The training to obtain the historical defect type determination model can be based on the historical pre-training model, and the training process of the historical pre-training model can use the model training method provided in the embodiment of the present application. Training to obtain the historical defect type determination model will use multiple historical sample images, and these historical sample images contain areas showing the previous version of the sim card holder. The defect labels carried by these historical sample images point to the defect types of the previous version of the sim card holder. Here, the historical sample images can be used as the second sample images, and the defect labels carried by these historical sample images can be used as the second type of object defect labels, thereby achieving the acquisition of the second sample set.

[0095] In one embodiment, Figure 5 As shown, for the aforementioned step S204, the use of the third sample set to train the feature extraction network to obtain a pre-trained model includes:

[0096] S501: Obtain a classification network;

[0097] S502: constructing a first association model based on the feature extraction network and the classification network;

[0098] S503: Using the third sample set to train the first association model to obtain a second association model;

[0099] S504: Determine that the feature extraction network after parameter adjustment in the second association model is the pre-trained model.

[0100] Since the defect labels of the second-class objects are different from the defect labels of the first-class objects, the first classification network above classifies the defects of the first-class objects based on the extracted features. Here, in order to use the third sample set to train the feature extraction network, a new classification network needs to be introduced. The classification network here (which can be called the second classification network) needs to classify the defects of the first-class objects and the second-class objects based on the extracted features. Therefore, the feature extraction network and the second classification network constitute a first association model that can be used as a training basis.

[0101] The second association model is obtained by training based on the third sample set and adjusting the parameters of the first association model during training. The training focuses on enabling the second association model to determine whether the image defect belongs to the defect type of the first category object and the second category object; if so, the specific defect type is determined. The second association model needs to be able to extract features from the image and classify defects based on the extracted features. The second association model can extract features from the image through the pre-trained model therein, and the benchmark model can classify defects based on the extracted features through the second classification network therein.

[0102] The model training method provided in the embodiment of the present application can be Figure 7 , 8 Reflection. Taking the same material and the same product but different models or cross-factory replication as an example, there are now two products with different models but the same material, namely Model A (corresponding to the second type of object mentioned above) and Model B (corresponding to the first type of object mentioned above). The data of Model A is called Source data, corresponding to the second sample set mentioned above. The data of Model B is called Target data, corresponding to the first sample set mentioned above. Assume that there are four defect labels in the Source data (i.e., defect types of Model A), namely Crack (LW), Scratch (HS), Collapse (BD), Bright Mark (LY), corresponding to at least one defect label of the second type of object mentioned above. There are five defect labels in the Target data (i.e., defect types of Model B), namely Crack (LW), Bright Mark (LWY), Dimple (MD), Scratch (HS), White Spot (BD), corresponding to at least one defect label of the first type of object mentioned above. Although these defect labels have overlaps in name, there are differences in the defect labels due to different acceptance standards.

[0103] like Figure 7As shown, a basic version detection model can be obtained by training with Target Data, called Basemodel, which corresponds to the above-mentioned baseline model. Next, the Base model is used to predict defects in the Source data to obtain object pseudo-labels belonging to the Target data domain, namely the above-mentioned LW, LWY, MD, HS, and BD. In order to distinguish duplicate labels in the Source data, a fixed prefix can be added to the predicted object pseudo-labels (such as target_prefix, then LW will be recorded as target_prefix_LW). Here, a filtering threshold score_thresh (corresponding to the above-mentioned probability threshold) can be selected, which is generally set to between 0.1 and 0.2. Then, all object pseudo-labels with a predicted probability greater than score_thresh are added to the defect label of the original Source data to obtain New Source data, corresponding to the above-mentioned third sample set.

[0104] like Figure 8 As shown, an association detection model is obtained by training with New source data, corresponding to the above-mentioned second association model. Compared with the classification head part of the basic version detection model (corresponding to the above-mentioned first classification network), the classification head part of the second association model (corresponding to the above-mentioned second classification network) has 4 more classification tasks. The parameters of the backbone (corresponding to the above-mentioned pre-trained model) in the association detection model and the parameters of the backbone (corresponding to the above-mentioned feature extraction network) in the basic version detection model can be initialized to obtain the above-mentioned pre-trained model. The pre-trained model is then trained on the Target data to obtain the final application model (corresponding to the subsequent target defect type determination model).

[0105] In practical applications, 1) for the detection of tiny defects in industry (such as cracks that are often less than 0.015mm, about two pixels), due to the large resolution of the input image, the area of ​​the defect in the high-resolution (for example, 2500*2500) image accounts for a very small proportion. The pre-trained model provided in the embodiment of the present application has a strong ability to extract tiny features. 2) The performance of the target defect type determination model obtained by the above method and the target defect type determination model obtained by the pre-training model obtained in the relevant technology at different magnitudes was compared by experiment. The indicators used are the overkill rate of parts and the missed detection rate of parts, and specifically the missed detection rate of parts is compared under the condition of aligning the overkill rate of parts. If the full amount of Target data training data is called 1.0train data. On this basis, the 1.0train data can be proportionally extracted to obtain 0.02train data, 0.05train data, 0.15train data, and 0.5train data. The data scale of the 1.0train data in the experiment here is 11094. 0.02 means that the data level is only 0.02 of 1.0 train data. The same is true for 0.05, 0.15, and 0.5. Fig. 9 The comparison results of part missed detection rate under the overkill rate of aligned parts at different levels of 0.02train data, 0.05train data, 0.15train data, 0.5train data and 1.0train data are shown. It can be seen that under the same level, the index performance of the target defect type determination model obtained by the above method is better than the index performance of the target defect type determination model obtained based on the pre-training model obtained in the relevant technology, and the missed detection rate is lower under the same overkill rate. Accordingly, in order to achieve the same level of overkill missed detection, the model training method provided in the embodiment of the present application can greatly reduce the amount of training data.

[0106] It can be seen from the technical solutions provided by the above embodiments of the present application that the embodiments of the present application provide a training scheme for a pre-trained model, which is conducive to improving the adaptability of the pre-trained model obtained through training. In the process of training to obtain the pre-trained model, the first sample set is first used to train to obtain a baseline model; then the baseline model is used to predict defects for the second sample set, and the defect labels originally carried by the second sample set are expanded to obtain a third sample set; furthermore, the feature extraction network in the baseline model is trained using the third sample set to obtain a pre-trained model. The pre-trained model starts training based on the defect type of the first category of objects, and uses samples related to the second category of objects to improve the feature extraction capability of the baseline model. This is conducive to improving the feature extraction capability of the pre-trained model for images containing areas showing the first category of objects, and is more targeted.

[0107] Figure 6 A schematic diagram of a defect type determination method according to an embodiment of the present application is shown. Figure 6 As shown, the method includes:

[0108] S601: Acquire an image to be processed, where the image to be processed includes an area showing a first type of object;

[0109] S602: Defect prediction is performed on the image to be processed using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-training model described in the aforementioned steps S201-S204.

[0110] As the starting point for training the target defect type determination model, the pre-trained model can accelerate the convergence of the new model and alleviate the direct training of the new model from being confined to the local optimum. The pre-trained model can be used as the backbone of the training to obtain the target defect type determination model. 1) The feature extraction network can be extracted from the benchmark model for training. After the pre-trained model is obtained through training, the pre-trained model can be connected to the first classification network in the benchmark model to serve as the starting point model for training to obtain the target defect type determination model. 2) The feature extraction network in the benchmark model can be copied, and the copied feature extraction network can be trained to obtain the pre-trained model. Then, the parameters of the feature extraction network in the benchmark model are adjusted according to the parameters of the pre-trained model to obtain the starting point model for training to obtain the target defect type determination model.

[0111] It can be seen from the technical solutions provided by the above embodiments of the present application that the target defect type determination model obtained based on the above pre-trained model is more accurate when predicting defects in images containing areas showing the first type of object. The target defect type determination model is a model with high generalization ability obtained through training. Using the target defect type determination model for defect prediction can improve the adaptability and reliability of defect prediction.

[0112] The present application also provides a model training device, such as Fig.10 As shown, the model training device 100 includes:

[0113] The first training module 1001 is used to obtain a benchmark model by training with a first sample set, wherein the benchmark model is used to output a defect type for an input image, wherein the benchmark model includes a feature extraction network and a first classification network, wherein the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class of object defect label;

[0114] The label expansion module 1002 is used to obtain, for each second sample image in the second sample set, a defect prediction result corresponding to the second sample image using the reference model, and to expand at least one second-category object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object;

[0115] A sample set updating module 1003 is configured to update the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set;

[0116] The second training module 1004 is used to train the feature extraction network using the third sample set to obtain a pre-trained model.

[0117] In one embodiment, the defect prediction result includes the predicted probability of each first-class object defect label corresponding to the second sample image, and the method of expanding at least one second-class object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result includes: obtaining a probability threshold; for the predicted probability corresponding to each first-class object defect label, when the predicted probability is greater than the probability threshold, determining that the first-class object defect label is an expanded label to obtain at least one expanded label; and obtaining the defect label expansion result based on the at least one expanded label and the at least one second-class object defect label.

[0118] In one embodiment, the method also includes determining the second category object: determining object information of the first category object, the object information including description information of multiple dimensions; determining a first category dimension and a second category dimension from the multiple dimensions; based on the description information matching results of the first category dimension and the description information matching results of the second category dimension, determining the second category object from multiple candidate objects, the description information matching degree between the second category object and the first category object in the first category dimension is greater than a first threshold, and the description information matching degree between the second category object and the first category object in the second category dimension is less than a second threshold.

[0119] Furthermore, the first category of objects is a product composed of at least one industrial part, and the multiple dimensions include a product name dimension, a product material dimension, a product version dimension, and a product origin dimension, and determining the first category of dimensions and the second category of dimensions from the multiple dimensions includes: determining that the product name dimension and the product material dimension are the first category of dimensions, and determining that the product version dimension is the second category of dimensions; or, determining that the product name dimension and the product material dimension are the first category of dimensions, and determining that the product origin dimension is the second category of dimensions.

[0120] In addition, the first category of objects and the second category of objects respectively indicate the current version and historical version of the same product, and the method also includes obtaining the second sample set: obtaining multiple historical sample images used to train the historical defect type determination model, the historical sample images containing areas showing the second category of objects, and the multiple historical sample images are labeled with defect labels based on product acceptance rules for the historical versions; determining that the historical sample image is the second sample image, and determining that the defect label carried by the historical sample image is at least one of the second category object defect labels.

[0121] In one embodiment, the method of training the feature extraction network using the third sample set to obtain a pre-trained model includes: obtaining a second classification network; constructing a first association model based on the feature extraction network and the second classification network; training the first association model using the third sample set to obtain a second association model; and determining that the feature extraction network after parameter adjustment in the second association model is the pre-trained model.

[0122] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0123] The present application also provides a defect type determination device, such as Fig.11 As shown, the defect type determination device 110 includes:

[0124] An acquisition module 1101 is used to acquire an image to be processed, where the image to be processed includes an area showing a first type of object;

[0125] Prediction module 1102: used to perform defect prediction on the image to be processed by using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-training model described in the aforementioned steps S201-S204.

[0126] It should be noted that the device in the device embodiment and the method embodiment are based on the same inventive concept.

[0127] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0128] The embodiment of the present application also provides a computer-readable storage medium, wherein at least one instruction or at least one program is stored in the computer-readable storage medium, and the at least one instruction or at least one program is loaded and executed by a processor to implement the above method. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0129] An embodiment of the present application also provides an electronic device, which includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the above method.

[0130] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0131] Fig.12 1 shows a block diagram of an electronic device according to an embodiment of the present application. For example, the electronic device 1900 can be provided as a server. Fig.12 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0132] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0133] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0134] The present application may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present application.

[0135] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0136] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0137] The computer program instructions for performing the operation of the present application can be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C+, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions can be executed completely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer, partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). In some embodiments, by using the state information of computer-readable program instructions to personalize electronic circuits, such as programmable logic circuits, field programmable gate arrays (FPGAs) or programmable logic arrays (PLAs), the electronic circuits can execute computer-readable program instructions, thereby realizing various aspects of the present application.

[0138] Various aspects of the present application are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0139] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0140] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0141] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present application. In this regard, each square frame in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the above-mentioned module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the standard function in the square frame can also occur in a sequence different from the standard in the accompanying drawings. For example, two continuous square frames can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square frame in the block diagram and / or flow chart, and the combination of the square frames in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0142] The embodiments of the present application have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A model training method, characterized in that: The method comprises: A benchmark model is obtained by training with a first sample set, the benchmark model is used to output a defect type for an input image, the benchmark model includes a feature extraction network, the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class of object defect label; For each second sample image in the second sample set, using the benchmark model to obtain a defect prediction result corresponding to the second sample image, and using the defect prediction result to expand at least one second-category object defect label carried by the second sample image to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object; Update the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set; Using the third sample set to train the feature extraction network to obtain a pre-trained model: obtaining a classification network; constructing a first association model based on the feature extraction network and the classification network; using the third sample set to train the first association model to obtain a second association model; determining that the feature extraction network after parameter adjustment in the second association model is the pre-trained model; Wherein, the defect prediction result includes the predicted probability of each first-class object defect label corresponding to the second sample image, and the method of expanding at least one second-class object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result includes: obtaining a probability threshold; for the predicted probability corresponding to each first-class object defect label, when the predicted probability is greater than the probability threshold, determining that the first-class object defect label is an expanded label to obtain at least one expanded label; and obtaining the defect label expansion result based on the at least one expanded label and the at least one second-class object defect label.

2. The method according to claim 1, characterized in that The method further comprises determining the second type of object: Determining object information of the first category of objects, the object information including description information of multiple dimensions; Determining a first dimension and a second dimension from the plurality of dimensions; Based on the description information matching results of the first category dimension and the description information matching results of the second category dimension, the second category object is determined from multiple candidate objects, the description information matching degree between the second category object and the first category object in the first category dimension is greater than a first threshold, and the description information matching degree between the second category object and the first category object in the second category dimension is less than a second threshold.

3. The method according to claim 2, characterized in that The first type of object is a product composed of at least one industrial part, the multiple dimensions include a product name dimension, a product material dimension, a product version dimension, and a product origin dimension, and determining the first type of dimension and the second type of dimension from the multiple dimensions includes: Determine that the product name dimension and the product material dimension are the first-type dimensions, and determine that the product version dimension is the second-type dimension; or, It is determined that the product name dimension and the product material dimension are the first type of dimensions, and it is determined that the product origin dimension is the second type of dimension.

4. The method according to claim 3, characterized in that The first type of object and the second type of object respectively indicate a current version and a historical version of the same product, and the method further includes obtaining the second sample set: Acquire a plurality of historical sample images used for training a historical defect type determination model, wherein the historical sample images include an area showing the second type of object, and the plurality of historical sample images are annotated with defect labels based on a product acceptance rule for the historical version; It is determined that the historical sample image is the second sample image, and it is determined that the defect label carried by the historical sample image is the at least one second-category object defect label.

5. A defect type determination method, characterized in that: The method comprises: Acquire an image to be processed, wherein the image to be processed includes an area showing a first category of objects; Defect prediction is performed on the image to be processed using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-trained model according to any one of claims 1 to 4.

6. A model training device, characterized in that: The device comprises: A first training module: used to obtain a benchmark model by training with a first sample set, wherein the benchmark model is used to output a defect type for an input image, wherein the benchmark model includes a feature extraction network, wherein the first sample set indicates a first class of objects, and each first sample image in the first sample set carries at least one first class of object defect label; A label expansion module: used for obtaining, for each second sample image in the second sample set, a defect prediction result corresponding to the second sample image using the reference model, and expanding at least one second-category object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result, wherein the second sample set indicates a second-category object, and the second-category object has an association relationship with the first-category object; A sample set updating module: used for updating the second sample set based on the defect label expansion result corresponding to each of the second sample images to obtain a third sample set; A second training module: used for training the feature extraction network and the second classification network using the third sample set to obtain a pre-trained model; Wherein, the second training module is used to obtain a classification network; construct a first association model based on the feature extraction network and the classification network; train the first association model using the third sample set to obtain a second association model; and determine that the feature extraction network after parameter adjustment in the second association model is the pre-trained model; The defect prediction result includes the predicted probability of each first-class object defect label corresponding to the second sample image, and the method of expanding at least one second-class object defect label carried by the second sample image with the defect prediction result to obtain a defect label expansion result includes: obtaining a probability threshold; for the predicted probability corresponding to each first-class object defect label, when the predicted probability is greater than the probability threshold, determining that the first-class object defect label is an expanded label to obtain at least one expanded label; and obtaining the defect label expansion result based on the at least one expanded label and the at least one second-class object defect label.

7. The device according to claim 6, characterized in that The device is also used to determine the second type of object: Determining object information of the first category of objects, the object information including description information of multiple dimensions; Determining a first dimension and a second dimension from the plurality of dimensions; Based on the description information matching results of the first category dimension and the description information matching results of the second category dimension, the second category object is determined from multiple candidate objects, the description information matching degree between the second category object and the first category object in the first category dimension is greater than a first threshold, and the description information matching degree between the second category object and the first category object in the second category dimension is less than a second threshold.

8. The device according to claim 7, characterized in that The first type of object is a product composed of at least one industrial part, the multiple dimensions include a product name dimension, a product material dimension, a product version dimension, and a product origin dimension, and determining the first type of dimension and the second type of dimension from the multiple dimensions includes: Determine that the product name dimension and the product material dimension are the first-type dimensions, and determine that the product version dimension is the second-type dimension; or, It is determined that the product name dimension and the product material dimension are the first type of dimensions, and it is determined that the product origin dimension is the second type of dimension.

9. The device according to claim 8, characterized in that The first type of object and the second type of object respectively indicate the current version and the historical version of the same product, and the device further includes a sample set acquisition module; The sample set acquisition module is used to acquire multiple historical sample images used for training the historical defect type determination model, wherein the historical sample images include an area showing the second category of objects, and the multiple historical sample images are labeled with defect labels based on the product acceptance rules for the historical version; determine that the historical sample image is the second sample image, and determine that the defect label carried by the historical sample image is at least one of the second category object defect labels.

10. A defect type determination device, characterized in that: The device comprises: An acquisition module: used for acquiring an image to be processed, wherein the image to be processed includes an area showing an object of the first category; Prediction module: used to predict defects of the image to be processed by using a target defect type determination model, wherein the target defect type determination model is obtained based on the pre-trained model as described in any one of claims 1 to 4.

11. An electronic device, characterized in that: The electronic device includes at least one processor and a memory communicatively connected to the at least one processor; wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the at least one processor to implement the model training method as described in any one of claims 1 to 4, or the defect type determination method as described in claim 5.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the model training method as described in any one of claims 1 to 4, or the defect type determination method as described in claim 5.

13. A computer program product, characterized in that The computer program product comprises at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by a processor to implement the model training method as described in any one of claims 1 to 4, or the defect type determination method as described in claim 5.

Images