Model training method and apparatus, and detection method and apparatus

Abstract

The present disclosure provides a model training method, comprising: obtaining a sample image comprising repeatable features and a sample label for describing a target feature in the sample image; on the basis of the sample image, obtaining a processed sample image comprising an input label, and a target label, wherein the input label describes areas corresponding to the repeatable features, and the target label describes a target area corresponding to the target feature; inputting the sample image into a target model to obtain a target feature prediction output, wherein the target model can extract feature data and generate the target feature prediction output on the basis of the feature data; inputting the processed sample image into an auxiliary model to obtain a target area prediction output, wherein the auxiliary model comprises the same model structure as the target model, and can extract auxiliary feature data and generate the target area prediction output on the basis of the auxiliary feature data and the feature data; and on the basis of the target feature prediction output and the target area prediction output as well as the sample label and the target label, updating model parameters of the target model and the auxiliary model.

Description

Model training method and device, detection method and device Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular, to a model training method and apparatus, a detection method and apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background Art

[0002] With the booming development of computer technology, artificial intelligence (AI) is increasingly being used to meet the many needs people encounter in various situations, from industrial production to daily life. For example, during the production, transportation, and use of various products, there is often a need to inspect their appearance, structure, and other aspects. Conventionally, this inspection can be performed manually, for example, by sampling products on an assembly line, after production, or during transportation and having them inspected by professionals. With the introduction of AI, images of relevant products can be obtained and analyzed using machine learning models to identify relevant features, such as the presence of defects. Compared to manual inspection, this can significantly improve inspection efficiency and reduce costs. Furthermore, when using machine learning models for inspection, how to further improve inspection efficiency and accuracy has become a highly sought-after issue.

[0003] Summary of the Invention

[0004] In view of this, the present disclosure provides a model training method and apparatus, a detection method and apparatus, a computing device, a computer-readable storage medium, and a computer program product, which can alleviate, mitigate, or even eliminate the above-mentioned problems.

[0005] According to one aspect of the present disclosure, a model training method is provided, including: obtaining a sample image and a sample label, the sample image containing repetitive features, and the sample label being configured to describe a target feature contained in the sample image; obtaining a processed sample image and a target label based on the sample image, wherein the processed sample image contains an input label, the input label is configured to describe multiple regions corresponding to the repetitive features, and the target label is configured to describe a target region corresponding to the target feature in the multiple regions; inputting the sample image into a target model to obtain a target feature prediction output, wherein the target model is configured to extract feature data based on the sample image and generate a target feature prediction output based on the feature data; inputting the processed sample image into an auxiliary model to obtain a target region prediction output, wherein the auxiliary model includes the same model structure as the target model, and is configured to extract auxiliary feature data based on the processed sample image, and generate a target region prediction output based on the auxiliary feature data and the feature data; updating model parameters of the target model and the auxiliary model based on the target feature prediction output, the target region prediction output, the sample label, and the target label to obtain a trained target model.

[0006] In some embodiments, the processed sample image includes a first processed image and a second processed image, the first processed image includes a first input label, the second processed image includes a second input label, the first input label is configured to describe a plurality of first regions corresponding to a first portion of repetitive features included in the sample image, the second input label is configured to describe a plurality of second regions corresponding to a second portion of repetitive features included in the sample image, the first portion of repetitive features and the second portion of repetitive features are interlaced in the sample image, wherein the target label includes a first target label and a second target label, the first target label is configured to describe a first target region corresponding to the target feature in the plurality of first regions, and the second target label is configured to describe a first target region corresponding to the target feature in the plurality of second regions. The method comprises the steps of: inputting the first processed image into the first auxiliary model to obtain the first target area prediction output, and inputting the second processed image into the second auxiliary model to obtain the second target area prediction output, wherein the first auxiliary model comprises the same model structure as the target model, and is configured to extract first auxiliary feature data based on the first processed image, and generate the first target area prediction output based on the first auxiliary feature data and the feature data, wherein the second auxiliary model comprises the same model structure as the target model, and is configured to extract second auxiliary feature data based on the second processed image, and generate the second target area prediction output based on the second auxiliary feature data and the feature data.

[0007] In some embodiments, updating the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target area prediction output, the sample label, and the target label includes: in response to the number of updates of the model parameters of the target model not reaching a preset threshold, synchronously updating the model parameters of both the target model and the auxiliary model based on the target feature prediction output and the target area prediction output, the sample label, and the target label; in response to the number of updates of the model parameters of the target model reaching a preset threshold, only updating the model parameters of the target model based on the target feature prediction output and the target area prediction output, the sample label, and the target label.

[0008] In some embodiments, based on the sample image, obtaining a processed sample image and a target label includes: extracting at least one feature map based on the sample image through a convolutional neural network; determining the feature size and feature center coordinates of the repetitive feature based on the at least one feature map; determining a two-dimensional grid map based on the feature size and feature center coordinates, the two-dimensional grid map including multiple grid areas, each grid area corresponding to an area of ​​a repetitive feature in the sample image; determining the processed sample image and the target label based on the two-dimensional grid map.

[0009] In some embodiments, each feature map in at least one feature map includes multiple feature points, which are probability peak points of corresponding features, and wherein, based on at least one feature map, determining the feature size and feature center coordinates of the repetitive feature includes: determining at least one set of offsets for at least one feature map, wherein each set of offsets in at least one set of offsets corresponds to a feature map, and each offset in a set of offsets represents the distance between two feature points in the feature map; based on at least one set of offsets, determining the offset with the highest probability of occurrence as the feature size; and determining the feature center coordinates based on the feature size.

[0010] In some embodiments, based on at least one set of offsets, determining the offset with the highest probability of occurrence as a characteristic size includes: substituting each offset in the at least one set of offsets into a target function, the function value of the target function being configured to be negatively correlated with the absolute value of the difference between the target amount and the offset; determining the target amount as the characteristic size so that the weighted sum of the function values ​​of the target function for each offset is maximized.

[0011] In some embodiments, determining a target quantity so that a weighted sum of function values ​​of the objective function for each offset is maximized includes: determining, for each group of offsets in at least one group of offsets, the sum of function values ​​of the objective function for each offset in the group of offsets; determining a weighted sum of the sum of function values ​​corresponding to each group of offsets, wherein the weight of the sum of function values ​​corresponding to each group of offsets is determined based on the number of offsets in the group of offsets; and determining the target quantity so that the weighted sum is maximized.

[0012] In some embodiments, determining the feature center coordinates based on the feature size of the repetitive feature includes: determining the coordinate values ​​of the feature center coordinates so that the coordinate values ​​satisfy the following conditions: the sum of the differences between each offset remainder and half of the feature size is minimized, where each offset remainder is the remainder of an offset difference divided by the feature size, and each offset difference is the difference between the coordinate value and an offset amount.

[0013] In some embodiments, updating the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target area prediction output, the sample label, and the target label includes: determining a first loss based on the target feature prediction output and the sample label; determining a second loss based on the target area prediction output and the target label; and updating the model parameters of the target model and the auxiliary model based on a combination of the first loss and the second loss.

[0014] In some embodiments, determining the first loss based on the target feature prediction output and the sample label includes: determining an adjustment factor corresponding to each predicted value in the target feature prediction output, the adjustment factor being negatively correlated with the predicted value; determining the first loss based on the target feature prediction output, the sample label and the corresponding adjustment factor, wherein determining the second loss based on the target area prediction output and the target label includes: determining an adjustment factor corresponding to each predicted value in the target area prediction output, the adjustment factor being negatively correlated with the predicted value; determining the second loss based on the target area prediction output, the target label and the corresponding adjustment factor.

[0015] In some embodiments, the target model includes a target encoder and a target decoder, the target encoder is configured to extract shallow feature data and deep feature data from the sample image, and the target decoder is configured to generate a target feature prediction output based on the deep feature data and the shallow feature data, wherein the auxiliary model includes an auxiliary encoder and an auxiliary decoder, the auxiliary encoder is configured to extract shallow auxiliary feature data and deep auxiliary feature data from the processed sample image, and the auxiliary decoder is configured to generate a target area prediction output based on the deep auxiliary feature data, the auxiliary shallow feature data, the deep feature data and the shallow feature data.

[0016] In some embodiments, the target encoder includes multiple target coding layers, and the auxiliary encoder includes multiple auxiliary coding layers, wherein the target decoder is configured to: receive shallow feature data from an intermediate target coding layer among the multiple target coding layers and deep feature data from an end target coding layer among the multiple target coding layers; splice the upsampled deep feature data with the shallow feature data to obtain spliced ​​feature data; and generate a target feature prediction output based on the spliced ​​feature data, wherein the auxiliary decoder is configured to: receive shallow feature data and deep feature data; receive shallow auxiliary feature data from an intermediate auxiliary coding layer among the multiple auxiliary coding layers and deep auxiliary feature data from an end auxiliary coding layer among the multiple auxiliary coding layers; fuse the shallow feature data with the shallow auxiliary feature data to obtain shallow fused auxiliary feature data; fuse the deep feature data with the deep auxiliary feature data to obtain deep fused auxiliary feature data; splice the upsampled deep fused auxiliary feature data with the shallow fused auxiliary feature data to obtain spliced ​​auxiliary feature data; and generate a target area prediction output based on the spliced ​​auxiliary feature data.

[0017] In some embodiments, at least one coding layer among multiple target coding layers and multiple auxiliary coding layers includes a feature extraction unit, wherein the feature extraction unit is configured to: receive upper-level feature data; generate a set of feature maps through convolution operation based on the upper-level feature data; generate another set of feature maps through linear transformation based on the set of feature maps; and output feature data at this level based on the set of feature maps and the other set of feature maps.

[0018] In some embodiments, the sample image is an image of a target object, and the target feature is a defect feature of the target object.

[0019] According to another aspect of the present disclosure, a detection method is provided, comprising: acquiring an image of an object to be detected; providing the image to a target model, wherein the target model is trained according to the method described in any embodiment of the aforementioned aspects; and determining whether the object to be detected contains target features based on the output of the target model.

[0020] According to another aspect of the present disclosure, a model training device is provided, comprising: a first acquisition module, configured to: acquire a sample image and a sample label, wherein the sample image contains repetitive features, and the sample label is configured to describe a target feature contained in the sample image; a second acquisition module, configured to: acquire a processed sample image and a target label based on the sample image, wherein the processed sample image contains an input label, the input label is configured to describe a plurality of regions corresponding to the repetitive features, and the target label is configured to describe a target region corresponding to the target feature in the plurality of regions; a first prediction module, configured to: input the sample image into a target model, and obtain a target feature prediction. Output, wherein the target model is configured to extract feature data based on the sample image and generate a target feature prediction output based on the feature data; the second prediction module is configured to: input the processed sample image into the auxiliary model to obtain a target area prediction output, wherein the auxiliary model includes the same model structure as the target model, and is configured to extract auxiliary feature data based on the processed sample image, and generate a target area prediction output based on the auxiliary feature data and the feature data; the update module is configured to: update the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target area prediction output, the sample label, and the target label to obtain a trained target model.

[0021] According to another aspect of the present disclosure, a detection device is provided, including: an acquisition module, configured to acquire an image of an object to be detected; a providing module, configured to provide the image to a target model, wherein the target model is trained using the device described in the above aspects; and a determination module, configured to determine whether the object to be detected contains target features based on the output of the target model.

[0022] According to another aspect of the present disclosure, a computing device is provided, comprising: a memory configured to store computer-executable instructions; and a processor configured to execute the method described in any one of the embodiments of the aforementioned aspects when the computer-executable instructions are executed by the processor.

[0023] According to another aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer-executable instructions. When the computer-executable instructions are executed, the method described in any embodiment of the aforementioned aspects is executed.

[0024] According to another aspect of the present disclosure, a computer program product is provided, comprising computer executable instructions, which, when executed by a processor, implement the steps of the method described in any one of the embodiments of the aforementioned aspects.

[0025] These and other aspects of the disclosure will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Further details, features and advantages of the present disclosure are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0027] FIG1 schematically illustrates an example of an image containing repetitive features according to some embodiments of the present disclosure;

[0028] FIG2 schematically illustrates an example scenario in which the technical solutions according to some embodiments of the present disclosure may be applied;

[0029] FIG3 schematically shows an example flow chart of a model training method according to some embodiments of the present disclosure;

[0030] FIG4 schematically shows an example architecture diagram for model training according to some embodiments of the present disclosure;

[0031] FIG5 schematically shows another example architecture diagram for model training according to some embodiments of the present disclosure;

[0032] FIG6 schematically shows an example structural block diagram of a target model and an auxiliary model according to some embodiments of the present disclosure;

[0033] FIG7 schematically shows another example structural block diagram of a target model and an auxiliary model according to some embodiments of the present disclosure;

[0034] FIG8 schematically shows another example structural block diagram of a target model according to some embodiments of the present disclosure;

[0035] 9A and 9B schematically illustrate example structural block diagrams of a G-bneck unit according to some embodiments of the present disclosure;

[0036] FIG10 schematically shows another example architecture diagram for model training according to some embodiments of the present disclosure;

[0037] 11A and 11B schematically illustrate example feature diagrams according to some embodiments of the present disclosure;

[0038] 12A and 12B exemplarily illustrate schematic diagrams of target feature segmentation according to some embodiments of the present disclosure;

[0039] FIG13 schematically shows an example flow chart of a detection method according to the present disclosure;

[0040] FIG14 schematically shows an example block diagram of a model training device according to the present disclosure;

[0041] FIG15 schematically shows an example block diagram of a detection device according to the present disclosure;

[0042] FIG16 schematically illustrates an example block diagram of a computing device according to the present disclosure. DETAILED DESCRIPTION

[0043] Before introducing the embodiments of the present disclosure in detail, some related concepts are first explained.

[0044] In the present disclosure, repetitive features can be understood as elements that appear repeatedly (or periodically) in an image, such as completely identical (or approximately completely identical) graphics or patterns that appear repeatedly along the length and width of the image. Exemplarily, FIG1 shows an image 100 containing repetitive features, wherein the area of ​​the repetitive features is marked with a dotted line. Exemplarily, the image containing repetitive features mentioned in the present disclosure can be an image of an object to be detected, wherein the object to be detected can be an object containing repetitive patterns, elements or structures, such as a textile containing a repetitive textile structure, a cloth formed with a repetitive pattern, a device provided with repetitive devices, a circuit board formed with a repetitive circuit structure, etc.

[0045] In the present disclosure, a repeating pattern can be understood as a pattern in which a repetitive feature repeatedly appears, such as the size and position of the region in which the repetitive feature appears. For example, for the repetitive feature in FIG1 , its repeating pattern can be presented by boxes divided by dashed lines, where each box has a length dx and a width dy, and the position of each box can be determined based on the coordinates of the center point or other methods.

[0046] As mentioned above, when implementing detection with the help of machine learning models, how to further improve the detection efficiency and accuracy is a problem that has attracted much attention. The applicant has found that in many detection scenarios, such as the detection scenarios of textiles containing repetitive patterns, devices provided with repetitive devices, circuit boards formed with repetitive circuit structures, etc. mentioned above, the image of the object to be detected often contains repetitive features, and the detection of target features with the help of such repetitive features can help improve the detection accuracy. In the conventional training scheme of machine learning, the data of the expected model output can be provided as a label, and the model parameters are iteratively updated so that the difference between the model output and the label is minimized or reduced to within the expected threshold range. However, if the repetitive features of the image are directly applied as additional labels to the training process of the model, the model will consume too much computing power on the calculation of labels related to the repetitive features, and rely on such label calculations in the subsequent detection process, which will have an adverse effect on the training speed of the model and the subsequent detection speed. Based on the above analysis, the applicants proposed a new model training scheme that can introduce information related to repetitive features through an additional network in a semi-supervised manner, thereby avoiding the model's reliance on the calculation of labels related to repetitive features. This helps to improve the model's detection accuracy while achieving satisfactory detection speed. In addition, the applicants also proposed a new detection scheme that can achieve fast and accurate detection of target features based on images containing repetitive features.

[0047] FIG2 schematically illustrates an example application scenario 200 to which the technical solution proposed in the present disclosure can be applied. As shown in the figure, the scenario 200 may include a server 210. The server 210 may be a single server or a server cluster, and may be configured to execute various embodiments of the model training method described in the present disclosure. For example, a target model to be trained may be deployed on the server 210, which may automatically or under user control perform training of the target model based on images and label data stored locally or from other devices (such as a terminal device 220, a camera 230, a database 240, etc.) by the methods described in the various embodiments of the present disclosure to obtain a trained model. Alternatively or additionally, the server 210 may also be configured to execute various embodiments of the detection method described in the present disclosure. For example, the server 210 may detect target features in an image based on locally stored images, images taken by the camera 230, images stored in the database 240, or images sent by the terminal device 220, using a model determined by the method according to an embodiment of the present disclosure. Optionally, the obtained detection results can be stored locally, stored in other devices (such as database 240), or sent to other devices (such as terminal device 220), etc.

[0048] Scenario 200 may also include terminal device 220. Terminal device 220 includes, but is not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, and in-vehicle terminals. For example, a user may access or use applications on server 210, send data to or receive data from it, etc., through terminal device 220. For example, a user may instruct server 210, through terminal device 220, to begin executing a method for determining a defect detection model, send image data or label data or related data identifiers to server 210 for model training, instruct server 210 to perform detection tasks using the trained model, and receive detection results generated by server 210.

[0049] Scene 200 may also include a camera 230. In this disclosure, a camera refers to any device with imaging capabilities. It can be a standalone imaging device, such as a digital camera, infrared camera, or display, or an imaging component integrated into another device, such as a camera in a mobile phone or computer. For example, in a scenario involving product inspection on an assembly line, camera 230 can be installed at a designated location on the assembly line to capture images of the products being inspected. The captured images can be stored locally, stored in a database 240, or transmitted to a server 210 or terminal device 220.

[0050] Scenario 200 may also include a database 240. For example, database 240 can be viewed as an electronic filing cabinet, i.e., a location for storing electronic files. Users can perform operations such as adding, querying, updating, and deleting data within these files. For example, database 240 can be used to store data or files such as text and images. For example, server 210 or terminal device 220 can obtain required data from database 240 or send (e.g., upload) generated or updated data to database 240.

[0051] Two or more of the server 210, terminal device 220, camera 230, and database 240 can communicate with each other via a network 250. The network 250 can be a wired network connected via a cable, optical fiber, or the like, or a wireless network such as 2G, 3G, 4G, 5G, Wi-Fi, Bluetooth, ZigBee, Li-Fi, or a combination of both.

[0052] Figure 3 schematically illustrates an example flow chart of a model training method 300 according to some embodiments of the present disclosure. For example, method 300 can be deployed in a computing device, such as an application, such as server 210, terminal device 220, or a combination thereof, in scenario 200 shown in Figure 2 . As shown in Figure 3 , method 300 may include steps 310 to 350, as follows.

[0053] In step 310, a sample image and a sample label are obtained. The sample image contains repetitive features, and the sample label is configured to describe the target features contained in the sample image. Exemplarily, the sample image and sample label can be obtained from a sample dataset currently stored or stored in an external memory. The sample dataset can include multiple samples, each of which can consist of a sample image and a sample label. Exemplarily, the sample image can be an image of an object that is desired to be detected by the trained target model, which can be captured by a camera or generated by other means. Exemplarily, the sample label can be obtained by manually annotating the target features in the sample image, or it can be obtained through model recognition and manual review. Exemplarily, the sample label can be image data of the same size as the sample image, where the target feature region and non-target feature region can be labeled with different colors. Alternatively, the sample label can be an array corresponding to pixels in the sample image, where each numerical value corresponds to a pixel point, and different numerical values ​​can be used to indicate whether the corresponding pixel point belongs to the target feature region or the non-target feature region, for example, 1 can be used to indicate a target feature and 0 can be used to indicate a non-target feature. Alternatively, the sample label can only contain data about the target feature region, such as the coordinates of the pixels in the target feature region. Alternatively, the sample label may take other appropriate forms as long as it can reflect the target features in the sample image.

[0054] At step 320, a processed sample image and target labels can be obtained based on the sample image. The processed sample image includes an input label, the input label is configured to describe multiple regions corresponding to the repetitive feature, and the target label is configured to describe a target region corresponding to the target feature within the multiple regions. Optionally, the repetitive features in the sample image can be analyzed manually, using a repetitive pattern extraction algorithm, or a combination of both to extract the input label and the target label. For example, the input label can be used to reflect information related to the repetitive pattern of the repetitive feature in the sample image. For example, the input label can describe multiple grid regions, each of which corresponds to a region containing a repetitive feature, the size of the grid region can correspond to the size of the repetitive feature, and the position of the grid region can correspond to the location of the repetitive feature. Alternatively, the repetitive features contained in the sample image can be described in other ways. For example, the processed sample image can be an image containing such an input label, for example, it can be a superposition of the sample image and the input label. For example, the input label can represent each region or region boundary using a specific color or other method. Such an input label can be superimposed on the sample image to form the processed sample image. Exemplarily, the target label can be obtained by manual or machine labeling based on the processed sample image or input label. For example, the processed sample image or input label can be compared with the sample label to determine the area of ​​the repetitive feature corresponding to the pixel corresponding to the target feature described by the sample label. For example, in the case where the processed sample image or input label contains multiple grid areas corresponding to repetitive features, if there are one or more pixels in a grid area that fall within the range of the target feature described by the sample label, then the grid area can be marked as containing the target feature, otherwise, it can be marked as not containing the target feature. Optionally, in the target label, different colors, different values, etc. can be used to distinguish whether the target feature exists. Alternatively, the target label can also only contain data on the area of ​​the repetitive feature where the target feature exists, such as the coordinates of the corresponding grid area. Alternatively, the target label can also take other appropriate forms, as long as it can reflect the area of ​​the repetitive feature corresponding to the target feature in the sample image.

[0055] In step 330, the sample image can be input into the target model to obtain a target feature prediction output, wherein the target model is configured to extract feature data based on the sample image and generate a target feature prediction output based on the feature data. Exemplarily, the target model can have any appropriate network structure so that it can extract feature data based on the input image and output the probability of the target feature existing at each pixel based on the feature data. For example, the target model can have a model structure that can be used for semantic segmentation, that is, semantic segmentation is performed on target feature areas and non-target feature areas. Optionally, the feature data can be a feature vector (such as embedding) or a feature map. Optionally, the target feature prediction output can include multiple numerical values ​​corresponding to each pixel, each numerical value can describe the probability of the target feature existing at the corresponding pixel. For example, the closer the numerical value is to 1, the higher the probability of the target feature existing at the corresponding pixel. On the contrary, the closer the numerical value is to 0, the lower the probability of the target feature existing at the corresponding pixel.

[0056] At step 340, the processed sample image can be input into an auxiliary model to obtain a target region prediction output. The auxiliary model includes the same model structure as the target model and is configured to extract auxiliary feature data based on the processed sample image and generate the target region prediction output based on the auxiliary feature data and the feature data. For example, the auxiliary model can include the same model structure as the target model and can include other additional network structures, such as connections for receiving feature data from the target model. For example, the auxiliary model can extract auxiliary feature data based on the processed sample image, and the extracted auxiliary feature data can be combined with the feature data from the target model, so that the auxiliary model can generate the target region prediction output based on the combined feature data. Optionally, the auxiliary feature data and the feature data can be combined, for example, by averaging, weighted averaging, summing, etc. Optionally, the target region prediction output can include multiple numerical values ​​corresponding to each pixel, each of which can describe the probability of the corresponding pixel belonging to the target region. For example, a value closer to 1 indicates a higher probability that the corresponding pixel belongs to the target region, while conversely, a value closer to 0 indicates a lower probability that the corresponding pixel belongs to the target region.

[0057] In step 350, the model parameters of the target model and the auxiliary model can be updated based on the target feature prediction output, the target area prediction output, the sample label, and the target label to obtain the trained target model. For example, the model parameters of the target model and the auxiliary model can be updated based on the difference between the target feature prediction output and the sample label and the difference between the target area prediction output and the target label to minimize the above-mentioned difference. Optionally, the above-mentioned difference can be measured with the help of a loss function, and the update of the model parameters can be implemented based on a mechanism such as gradient descent, for example, stochastic gradient descent, mini-batch gradient descent, batch gradient descent, etc. can be used. For example, according to experience, the momentum of the stochastic gradient descent optimizer can be set to 0.9, the weight decay parameter can be set to 0.0001, the batch size can be set to 8, and the learning rate can be set to 0.005. It should be understood that the parameters shown here are only exemplary, and different parameter update mechanisms and different parameters can be selected based on experience and actual application requirements.

[0058] Exemplarily, steps 310 to 350 may be iteratively executed. When an iteration cutoff condition is met, the target model may be deemed to have been trained and may be used for target feature detection tasks. When the iteration cutoff condition is not met, steps 310 to 350 may continue to be executed. Optionally, the iteration cutoff condition may include at least one of the following: the difference between the target feature prediction output and the sample label, or the difference between the target region prediction output and the target label, or both, or a combination of the two, is lower than a preset difference threshold, or reaches a preset upper limit on the number of iterations.

[0059] Through the above-described method 300, input labels and target labels related to the repetitive patterns of repetitive features in the sample image can be extracted, and the input labels and target labels can be introduced into the model training process through an additional auxiliary model. On the one hand, by introducing labels related to repetitive features, it is helpful to improve the detection performance of the model for images containing repetitive features; on the other hand, by introducing the repetitive features separately into the model training process through the additional auxiliary model, compared with directly introducing these labels through the target model, it is possible to avoid the model consuming a large amount of computing power to analyze the repetitive features, thereby improving the model's target feature detection performance for images containing repetitive features without affecting the model's detection speed.

[0060] More specifically, in the above-described method 300, knowledge about the repetitive features of an image is utilized in two ways: first, by using target labels to highlight target regions containing target features, and second, by injecting additional repetitive regional information into the model through input labels. Both of these are achieved through additional model channels. This not only leverages knowledge about repetitive features to achieve better model detection results, but also does not result in a loss in detection speed. Therefore, it offers significant advantages over conventional model training methods.

[0061] It should be understood that in the present disclosure, although the steps in the method are numbered and described in sequence and shown in the accompanying drawings, these steps do not have to be performed in the order described or shown, and some steps can be performed in the opposite order to the order shown or described, or some steps can be performed in parallel. Take steps 310 to 360 of method 300 described above as an example, where, for example, steps 330 and 340 can be performed in the opposite order or in parallel, and step 320 can be performed before step 310. For example, the extraction of input labels and target labels can be performed in advance for some or all sample images in the sample data set, or the acquisition of processed sample images and target labels can be performed in advance, and the input labels or processed sample images and target labels can be stored for use.

[0062] To further facilitate understanding, FIG4 schematically shows an architecture 400 for executing the above-mentioned method 300. As shown in the figure, a repetitive pattern extraction module 430 can be used to extract repetitive patterns of repetitive features based on the sample image to form a processed sample image containing an input label, and then the processed sample image or the input label can be labeled based on the target feature in the sample image to form a target label, wherein the repetitive pattern extraction module 430 can perform step 320 described above, as well as various embodiments of step 320 described below. The sample image can be input to the target model 410 to obtain a target feature prediction output. The processed sample image can be input to the auxiliary model 420 to obtain a target area prediction output. The sample label, target feature prediction output, target label, and target area prediction output can be input to the loss function module 440. Subsequently, the model parameters of the target model 410 and the auxiliary model 420 can be updated based on the output of the loss function module.

[0063] In some embodiments, when extracting input labels and target labels based on a sample image, repetitive features can be grouped to generate two or more groups of input labels and target labels, and introduced into the training process of the target model through two or more auxiliary models, so that different parts of the repetitive features are introduced through different model channels, and the features of different parts are processed separately to improve the training effect. Exemplarily, the processed sample image may include a first processed image and a second processed image, the first processed image may include a first input label, and the second processed image may include a second input label, wherein the first input label may be configured to describe multiple first regions corresponding to the first part of the repetitive features contained in the sample image, and the second input label may be configured to describe multiple second regions corresponding to the second part of the repetitive features contained in the sample image. Exemplarily, the first part of the repetitive features and the second part of the repetitive features are staggered in the sample image. Staggered distribution can be understood as, with reference to Figure 1, if a dotted box area belongs to the first area, then the dotted boxes above, below, to the left, and to the right of it all belong to the second area, and vice versa. By dividing the repetitive features into two groups for labeling in this staggered manner, the repetitive features can be better utilized. For example, the spatial distortion information captured by the elastic grid can be better utilized to improve the model training effect. Alternatively, the repetitive features can be grouped in other ways according to application requirements. Accordingly, the target label can include a first target label and a second target label, and the first target label can be configured to describe a first target area corresponding to the target feature in multiple first areas, and the second target label can be configured to describe a second target area corresponding to the target feature in multiple second areas.

[0064] In the above embodiment, step 340 may include: inputting the first processed image into a first auxiliary model to obtain a first target area prediction output, and inputting the second processed image into a second auxiliary model to obtain a second target area prediction output. The first auxiliary model may include the same model structure as the target model and is configured to extract first auxiliary feature data based on the first processed image, and generate the first target area prediction output based on the first auxiliary feature data and the feature data. The second auxiliary model may include the same model structure as the target model, which may have exactly the same model structure as the first auxiliary model, and is configured to extract second auxiliary feature data based on the second processed image, and generate the second target area prediction output based on the second auxiliary feature data and the feature data. For example, the first auxiliary model may extract first auxiliary feature data based on the first processed image, and the extracted first auxiliary feature data may be combined with the feature data from the target model, so that the first auxiliary model can generate the first target area prediction output based on the combined feature data. Optionally, the first auxiliary feature data and the feature data may be combined, for example, by averaging, weighted averaging, summing, etc. Optionally, the first target region prediction output may include multiple numerical values ​​corresponding to each pixel, each numerical value may describe the probability of the corresponding pixel belonging to the target region. For example, the closer the numerical value is to 1, the higher the probability that the corresponding pixel belongs to the first target region. Conversely, the closer the numerical value is to 0, the lower the probability that the corresponding pixel belongs to the first target region. The second auxiliary model may have exactly the same structure as the first auxiliary model and be configured to perform the same operation based on the second processed image.

[0065] For example, FIG5 schematically illustrates an architecture 500 for executing the above-described embodiment. As shown in FIG5 , a repetitive pattern extraction 530 can be used to extract repetitive patterns of repetitive features based on a sample image, and the repetitive features (or multiple regions corresponding to the repetitive features) can be grouped and labeled to form a first processed image containing a first input label and a second processed image containing a second input label, and then the first processed image or the first input label, and the second processed image or the second input label can be labeled based on the target features in the sample image to form a first target label and a second target label, wherein the repetitive pattern extraction module 530 can execute step 320 described above, and various embodiments of step 320 described below. The sample image can be input to the target model 510 to obtain a target feature prediction output. The first processed image can be input to the first auxiliary model 521 to obtain a first target region prediction output. The second processed image can be input to the second auxiliary model 522 to obtain a second target region prediction output. The sample label, target feature prediction output, first target label, second target label, first target region prediction output, and second target region prediction output may be input to the loss function module 540. Subsequently, the model parameters of the target model 510 and the first auxiliary model 521 and the second auxiliary model 522 may be updated based on the output of the loss function module.

[0066] In some embodiments, the target model and the auxiliary model (or the first auxiliary model and the second auxiliary model) may have initial parameters, which may be set according to experience or may be determined by a pre-training step. Subsequently, as the model training steps (such as at least part of the steps 310 to 350 described above) are iteratively executed, the model parameters of the target model and the auxiliary model (or the first auxiliary model and the second auxiliary model) may be synchronously and gradually updated. However, in order to speed up the training, the model parameters of the auxiliary model (or the first auxiliary model and the second auxiliary model) may be frozen in advance. In other words, in some embodiments, the above step 350 may include: in response to the number of updates of the model parameters of the target model not reaching a preset threshold, based on the target feature prediction output and the target area prediction output and the sample label, the target label, the model parameters of both the target model and the auxiliary model are synchronously updated; in response to the number of updates of the model parameters of the target model reaching a preset threshold, based on the target feature prediction output and the target area prediction output and the sample label, the target label, only the model parameters of the target model are updated. The preset threshold value of the number of parameter updates can be set according to experience and needs. For example, it can be set to one-quarter, one-half or other values ​​of the expected total number of iterations to achieve a balance between model training accuracy and training speed. For example, assuming that the number of training iterations of the target model is 10,000, the model parameters of the auxiliary model 420 shown in Figure 4 or the first auxiliary model 521 and the second auxiliary model 522 shown in Figure 5 can be frozen at one-quarter of the number of training iterations, that is, their parameters can stop updating and be frozen after being updated 2,500 times.

[0067] As shown in Figures 4 and 5, in some embodiments, the loss function can be used to measure the difference between the target feature prediction output and the sample label, and the target area prediction output and the target label. For the convenience of calculation, the loss of the target feature prediction output relative to the sample label, and the loss of the target area prediction output relative to the target label can be calculated separately. Specifically, the above step 350 may include: determining a first loss based on the target feature prediction output and the sample label; determining a second loss based on the target area prediction output and the target label; and updating the model parameters of the target model and the auxiliary model based on the combination of the first loss and the second loss. Optionally, the loss functions for determining the first loss and determining the second loss may be the same or different. Exemplarily, the cross entropy (CE) loss function can be used to determine the first loss and the second loss. Specifically, the cross entropy loss can be calculated by the following formula:

[0068] Where W and H represent the width and height of the image, respectively, which can be represented by the number of pixels; p i,jIndicates the probability that the pixel at position (i, j) belongs to a certain feature. In the embodiment of the present disclosure, it can be the probability that each pixel output by the target model belongs to the target feature or the probability that each pixel belongs to the non-target feature, or the probability that each pixel output by the auxiliary model belongs to the target area or the probability that each pixel belongs to the non-target area; Indicates the label probability that the pixel at the position (i, j) belongs to a certain feature. In the embodiment of the present disclosure, it can be determined based on the sample label or the target label. Taking the sample label as an example, if the sample label indicates that a certain pixel belongs to the target feature, then the label probability of the pixel belonging to the target feature can be regarded as 1, and the label probability of belonging to the non-target feature can be regarded as 0. Conversely, if the sample label indicates that a certain pixel does not belong to the target feature, then the label probability of the pixel belonging to the target feature can be regarded as 0, and the label probability of belonging to the non-target feature can be regarded as 1. For the target label, it can be processed similarly. Further exemplarily, in a binary classification scenario, that is, in a scenario where pixels in an image are classified as belonging to a target feature or not belonging to a target feature, or as belonging to a target area or not belonging to a target area, the binary cross entropy loss shown below can be used to calculate the aforementioned first loss and second loss:

[0069] in, represents the label probability that the pixel at position (i, j) belongs to the target feature (or target area) determined by the sample label (or target label), p i,j It represents the probability that the pixel at position (i, j) output by the target model (or auxiliary model) belongs to the target feature (or target area). In addition, other types of loss functions can also be selected according to specific application requirements.

[0070] In some embodiments, the number of pixels of target features and non-target features may be highly unbalanced. For example, in a defect detection scenario for textiles, in some sample images, the number of defective pixels may be much lower than the number of background pixels (i.e., non-defective pixels). In the process of measuring the difference between the target feature prediction output and the sample label, and the target area prediction output and the target label through the loss function, and iteratively updating the model parameters to minimize the value of the loss function, the model may overfit the non-target features. For example, the model may gradually tend to predict each pixel in such samples as a non-target feature, so that the value of the loss function gradually decreases. This is obviously not desirable and will have an adverse effect on model accuracy and training efficiency. To avoid this problem, a dynamic adjustment factor can be introduced to enhance the model's ability to mine such samples. Specifically, when determining the first loss based on the target feature prediction output and the sample label, an adjustment factor corresponding to each predicted value in the target feature prediction output can be determined, the adjustment factor is negatively correlated with the predicted value, and the first loss is determined based on the target feature prediction output, the sample label and the corresponding adjustment factor; when determining the second loss based on the target area prediction output and the target label, an adjustment factor corresponding to each predicted value in the target area prediction output can be determined, the adjustment factor is negatively correlated with the predicted value; the second loss is determined based on the target area prediction output, the target label and the corresponding adjustment factor. Generally speaking, when determining the above-mentioned first loss or second loss according to the loss function, each predicted value and the corresponding label value can be substituted into the loss function to obtain the corresponding loss value, and then the loss values ​​corresponding to each predicted value are added to obtain the above-mentioned first loss or second loss. Exemplarily, when determining the corresponding loss value for each predicted value, the corresponding adjustment factor can be multiplied by the loss value calculated based on the loss function. Since the adjustment factor is negatively correlated with the predicted value, when the predicted value is high, the adjustment factor is low, and when the predicted value is low, the adjustment factor is high, so that the loss value corresponding to each predicted value can be adjusted to avoid or at least reduce the impact of the predicted value on the corresponding loss value. For example, the adjustment factor can be selected as (1-p i,j ) γ , where p i,j Indicates the probability that the pixel at the (i, j) position belongs to a certain feature. In the embodiment of the present disclosure, it can be the probability that each pixel output by the target model or the auxiliary model belongs to the target feature or the target area. γ can be a preset parameter, which can be set to a value greater than or equal to 1, for example, set to 2 or other values. In this way, for a certain pixel, when the prediction probability is high, the adjustment factor can be close to 0, and when the prediction probability is low, the adjustment factor can be close to 1, thereby helping to improve the model's ability to mine samples with a highly unbalanced number of pixels between target features and non-target features, and improving the model training effect.

[0071] In some embodiments, as shown in FIG6 , the target model 410 shown in FIG4 includes a target encoder 411 and a target decoder 412, and the auxiliary model 420 may include an auxiliary encoder 421 and an auxiliary decoder 422. The target encoder 411 may be configured to extract shallow feature data (indicated by a dotted line) and deep feature data (indicated by a solid line) from the sample image, and the target decoder 412 may be configured to generate a target feature prediction output based on the deep feature data and the shallow feature data. The auxiliary model 420 may include an auxiliary encoder 421 and an auxiliary decoder 422, the auxiliary encoder 421 may be configured to extract shallow auxiliary feature data (indicated by a dotted line) and deep auxiliary feature data (indicated by a solid line) from the processed sample image, and the auxiliary decoder 422 may be configured to generate a target region prediction output based on the deep auxiliary feature data, the auxiliary shallow feature data, and the deep feature data and shallow feature data from the target encoder 411. By extracting shallow and deep features from the image at the same time on the encoder side, and providing the shallow and deep features to the decoder side for merging, and then generating prediction output, it helps to improve the efficiency of model detection.

[0072] In some embodiments, as shown in FIG7 , the target encoder 411 may include multiple target coding layers 411-1 to 411-N, where N is an integer greater than 1. The target decoder 412 may be configured to: receive shallow feature data from an intermediate target coding layer 411-n among the target coding layers 411-1 to 411-N and deep feature data from an end target coding layer 411-N among the target coding layers 411-1 to 411-N, where n is an integer greater than or equal to 1 and less than N; concatenate the upsampled deep feature data with the shallow feature data to obtain concatenated feature data; and generate a target feature prediction output based on the concatenated feature data. Exemplarily, as shown in FIG7 , the target decoder may include multiple decoding layers 412-1 to 412-M, where M is an integer greater than 1. The deep feature data from the target encoder 411 can be received by the first decoding layer 412 of the target decoder 411, and the first decoding layer 412 can upsample the deep feature data, or it can perform other operations on the deep feature data, and the upsampling is performed by other decoding layers thereafter. The shallow feature data from the target encoder 411 can be received by the decoding layer 412-m of the target decoder 421, where m is an integer greater than 1 and less than M. The decoding layer 412-m can splice the upsampled (optionally, also processed otherwise) deep feature data with the received shallow feature data, and the spliced ​​feature data can be processed by subsequent decoding layers to obtain the target feature prediction output. Optionally, the layer structure of each encoding layer in the target encoding layers 411-1 to 411-N and each decoding layer in the target decoding layers 412-1 to 412-M can be selected according to specific application requirements, and can be selected as a combination of one or more of a convolutional layer, a pooling layer, a fully connected layer, an upsampling layer, a splicing layer or other layer structures.

[0073] The auxiliary encoder 421 and the auxiliary decoder 422 may have the same structure as the target encoder 411 and the target decoder 412, except that the auxiliary decoder 422 also receives feature data from the target encoder. Specifically, as shown in FIG7 , the auxiliary encoder 421 may include a plurality of auxiliary coding layers 421-1 to 421-N, and the auxiliary decoder 422 may be configured to: receive shallow feature data and deep feature data from the target encoder 411; receive shallow auxiliary feature data from an intermediate auxiliary coding layer 421-n among the plurality of auxiliary coding layers 421-1 to 421-N and deep auxiliary feature data from an end auxiliary coding layer 421-n among the plurality of auxiliary coding layers 421-1 to 421-N; and process the shallow feature data and the shallow auxiliary feature data. The shallow feature data are fused to obtain shallow fusion auxiliary feature data, and the deep feature data and the deep auxiliary feature data are fused to obtain deep fusion auxiliary feature data, wherein the fusion can be completed by summing, averaging, weighted averaging and other operations on the shallow feature data and the shallow auxiliary feature data (or the deep feature data and the deep auxiliary feature data); the upsampled deep fusion auxiliary feature data and the shallow fusion auxiliary feature data are spliced ​​with the shallow fusion auxiliary feature data to obtain spliced ​​auxiliary feature data; and the target area prediction output is generated based on the spliced ​​auxiliary feature data.

[0074] Exemplarily, the target encoder 411 and the target decoder 412 can adopt a structure similar to that shown in Figure 8. As shown in the figure, the target encoder 411 may include a backbone network (Backbone) for extracting shallow features and deep features and a feature enhancement module for enhancing deep features. Here, the feature enhancement module is implemented using an ASPP (Atrous spatial pyramid pooling) module. As shown in Figure 8, the backbone network can obtain shallow features and deep features in the sample image, and the ASPP module can enhance the deep features. The shallow features and the enhanced deep features can be passed to the target decoder 412, wherein the enhanced deep features can be spliced ​​with the shallow features after upsampling, and then output the target feature prediction through 3×3 convolution and bilinear upsampling operations. In this process, the deep features can provide high-level semantic information, and the shallow features can provide boundary information. Through the combination of the two, the target feature area and the non-target feature area can be segmented more efficiently and accurately. Optionally, the backbone network may include a plurality of layer structures, which may be a convolutional network or a deep convolutional network including a plurality of convolutional layers, or it may also have other types of network structures. It should be understood that the auxiliary encoder 421 and the auxiliary decoder 422 may also have a structure similar to that shown in FIG8 , which will not be described here for the sake of brevity.

[0075] In some embodiments, at least one of the aforementioned multiple target coding layers and multiple auxiliary coding layers (such as the target coding layers 411-1 to 411-N and the auxiliary coding layers 421-1 to 421-N shown in FIG7 , or the layers in the backbone network shown in FIG8 ) may include a feature extraction unit. The feature extraction unit may be configured to: receive upper-level feature data, which may be data from an upper-level feature extraction unit, data from an upper coding layer, or an input sample image or a processed sample image; based on the upper-level feature data, generate a set of feature maps through a convolution operation, wherein each feature map can be represented as a series of feature values ​​with a preset dimension; based on the set of feature maps, generate another set of feature maps through a linear transformation; based on the two sets of feature maps, output feature data of this level, for example, the two sets of feature maps may be stacked together and output as feature data of this level, or the two sets of feature maps may be further processed to generate feature data of this level. In this type of embodiment, unlike the general method of using point-by-point convolution and depth-wise convolution to process cross-channel features and spatial information in sequence, a part of the feature map is generated by means of convolution operation, and then another part of the feature map is generated by means of relatively cheap linear operation, and then the results of the two are stacked on the channel, so that the channel contains both channel features and spatial information. Linear operation can replace part of the convolution operation, reducing the computational cost in the process of generating redundant feature maps, effectively improving feature extraction efficiency and saving computing power.

[0076] Exemplarily, the above-mentioned feature extraction unit can be implemented with the help of a Ghost module. The Ghost module first generates a plurality of eigenmaps through ordinary convolution, then performs a linear transformation based on the eigenmap to generate a corresponding Ghost map, and stacks the results of the two to obtain the extracted features. Further exemplarily, two or more Ghost modules can be stacked to form a G-bneck unit, and then a GhostNet network can be formed based on multiple G-bneck units. The GhostNet network can be used as the backbone network shown in Figure 8, or used to form the target encoder 411 and the auxiliary encoder 421 shown in Figure 6 or Figure 7.

[0077] Exemplarily, the above-mentioned G-bneck can have a structure as shown in Figure 9A or Figure 9B. Figure 9A shows a G-bneck unit 900A with a stride (Stride) of 1, wherein two Ghost modules are stacked together, the first Ghost module can be used to increase the feature dimension, and the second Ghost module can be used to reduce the feature dimension, so that the output dimension of the G-bneck unit can be consistent with the input dimension. Optionally, there can be a normalization layer (BN, BatchNorm) and an activation layer after the first Ghost module, and a normalization layer after the second Ghost module, wherein the activation layer can, for example, use the ReLU shown in the figure, or other activation functions can also be used. In addition, there is a jump connection path (Shortcut), which adds the unit input to the result processed by the two Ghost modules and the relevant normalization layer and activation function layer to obtain the output of the unit, which can also be called a residual connection. Exemplarily, the jump connection can be an identity connection. Figure 9B shows a G-bneck unit with a stride of 2. Its structure is similar to that shown in Figure 9A. The only difference is that a depthwise convolution layer (DWConv) with a stride of 2 is added between the two Ghost modules, and the jump connection can be a downsampling connection. Therefore, in the G-bneck unit with a stride of 2, the output dimension can be different from the input dimension.

[0078] Exemplarily, the above-mentioned GhostNet network can adopt the layer structure shown in Table 1, wherein the entire network is divided into 6 stages, the first stage is a two-dimensional convolutional layer (Conv2d), and the remaining stages are all composed of G-bneck units. In each stage, the feature dimension remains consistent, the stride of the last G-bneck unit in each stage is 2, and the strides of the remaining G-bneck units are all 1. In addition, the SE (Squeeze and Excitation) attention module can be applied to certain G-bneck units to enhance channel features, optimize feature extraction effects, and improve the overall performance of the model. It should be understood that the network structure shown in Table 1 is merely exemplary, and different network structures can be designed according to the inventive concept of the present disclosure to meet actual application requirements.

[0079] Table 1 Network structure of an exemplary GhostNet network

[0080] By way of further example, in an embodiment where the target encoder 411 and target decoder 412 employ a structure similar to that shown in FIG8 and use the GhostNet network structure shown in Table 1 as the backbone network, shallow features may be derived from the output of the fourth stage of the GhostNet network shown in Table 1, or from the output of stages 2, 3, 5, and so on. Deep features may be derived from the output of the last stage of the GhostNet network shown in Table 1 and fed into a feature enhancement module such as ASPP. The enhanced deep and shallow features may then be passed to the target decoder 412 and an auxiliary decoder. For example, the target decoder 412 may include a first upsampling layer, a convolutional layer, and a second upsampling layer. In this example, the enhanced deep features may be passed to the first upsampling layer for upsampling. The shallow features may be passed to the convolutional layer for concatenation with the enhanced deep features and convolution operation performed thereafter. The convolution result may then be fed into the second upsampling layer for upsampling to produce the target feature prediction output. Similarly, the auxiliary decoder can receive the enhanced deep features and the enhanced auxiliary deep features through the first upsampling layer to fuse the two and then perform upsampling. The shallow features and the auxiliary shallow features can be passed to the convolution layer and fused, and spliced ​​with the upsampled deep fusion auxiliary feature data, and then a convolution operation is performed at the convolution layer. Subsequently, the convolution result is sent to the second upsampling layer to obtain the target area prediction output after upsampling. It should be understood that the above description is only exemplary. In fact, the structure and number of layers of the encoder and decoder can be designed according to specific application requirements, and the location where the shallow features and deep features are generated at the encoder and the location where they are passed to the decoder can also be designed according to actual application requirements.

[0081] Various embodiments of target model 410 and auxiliary model 420 shown in FIG4 have been described above with reference to FIG6 to FIG9B . It should be understood that target model 510 shown in FIG5 may have the same or similar embodiments as target model 410, and first auxiliary model 521 and second auxiliary model 522 may each have the same or similar embodiments as auxiliary model 420. For the sake of brevity, a repeated description will not be given.

[0082] Further exemplarily, FIG10 schematically illustrates a model training architecture 1000 according to some embodiments of the present disclosure. As shown in FIG10 , the architecture 1000 may include a target model, a first auxiliary model, a second auxiliary model, and a repetitive pattern extraction module 1040, wherein the target model includes a target encoder 1011 and a target decoder 1012, the first auxiliary model includes a first auxiliary encoder 1021 and a first auxiliary decoder 1022, and the second auxiliary model includes a second auxiliary encoder 1031 and a second auxiliary decoder 1032. The target encoder 1011 may include the GhostNet network and the ASPP module described in the previous embodiment, wherein the GhostNet network may extract shallow features and deep features based on the sample image 1051, the shallow features may be output by the intermediate layer of the GhostNet network, and the deep features may be output by its terminal layer and enhanced by the ASPP module. For example, the GhostNet in the target encoder 1011 shown in FIG10 may have the structure shown in Table 1, which includes five stages consisting of different numbers of G-bneck units, the shallow features may come from the output of the third stage, and the deep features may come from the output of the fifth stage. The target decoder 1012 may include a first upsampling layer, a convolutional layer, and a second upsampling layer, wherein the first upsampling layer may upsample the deep features, and the upsampled deep features may be concatenated with the shallow features and, through the convolutional layer and the second upsampling layer, generate the final target feature prediction output. The first auxiliary encoder 1021 and the second auxiliary encoder 1031 may have the same structure as the target encoder 1011, and the first auxiliary decoder 1022 and the second auxiliary decoder 1032 may have the same structure as the target decoder 1012. The deep features from the target encoder 1011 are also transmitted to the input ends of the first auxiliary decoder 1022 and the second auxiliary decoder 1032 through a skip connection to be fused with the first auxiliary deep features generated by the first auxiliary encoder 1021 and the second auxiliary deep features generated by the second auxiliary encoder 1031 (such as addition, averaging, weighted averaging, etc.). The shallow features from the target encoder 1011 are also transmitted to the intermediate layers of the first auxiliary decoder 1022 and the second auxiliary decoder 1032 through a skip connection to be fused with the first auxiliary shallow features generated by the first auxiliary encoder 1021 and the second auxiliary shallow features generated by the second auxiliary encoder 1031 (such as addition, averaging, weighted averaging, etc.).Through this jump connection, multiple combined networks are formed, wherein the main network is composed of a target encoder 1011, a target decoder 1012, a first auxiliary decoder 1022, and a second auxiliary decoder 1032, which receives a sample image 1051 and outputs a target feature prediction output, a first target region prediction output, and a second target region prediction output; an auxiliary network is composed of a target encoder 1011, a first auxiliary encoder 1021, and a first auxiliary decoder 1022, wherein the first auxiliary encoder 1021 receives a first processed image 1053 (which contains a first input label); another auxiliary network is composed of a target encoder 1011, a second auxiliary encoder 1031, and a second auxiliary decoder 1032, wherein the second auxiliary encoder 1031 receives a second processed image 1054 (which contains a second input label). The first processed image 1053 and the second processed image 1054 can be obtained by grouping and labeling the repeated pattern extraction output 1052 of the repeated pattern extraction module 1040, which can be performed according to the embodiment described above and will be explained in more detail below. During the training process, the model parameters of the target model, the first auxiliary model, and the second auxiliary model can be updated synchronously, or the model parameters of the first auxiliary model and the second auxiliary model can be updated synchronously in the early stage and frozen in the later stage. After the training is completed, the first auxiliary model and the second auxiliary model can be discarded, and the trained target model (i.e., the target encoder 1011 and the target decoder 1012) can be used for the target feature detection task.

[0083] The following will describe in more detail the aforementioned process of obtaining the processed sample image containing the input label and the target label with reference to the relevant figures and embodiments. For example, these embodiments can be performed by the aforementioned repetitive pattern extraction module or other similar modules.

[0084] In some embodiments, the aforementioned method 320 may include: extracting at least one feature map based on the sample image using a convolutional neural network; determining the characteristic size and feature center coordinates of the repetitive feature based on the at least one feature map; determining a two-dimensional grid map based on the feature size and feature center coordinates, the two-dimensional grid map including multiple grid regions, each grid region corresponding to a region of a repetitive feature in the sample image; and determining the processed sample image and target label based on the two-dimensional grid map. Optionally, the convolutional neural network can be selected as needed, for example, AlexNet or other networks can be used. A convolutional neural network can be considered a feature extractor that can perform feature extraction by operating on regular local peaks in the image, that is, calculating the probability that each pixel in the image belongs to a certain feature. For example, different feature extractors can be understood as different filters or different convolution kernels, which can perform different feature extraction operations. Taking AlexNet as an example, feature maps can be generated at each layer of the network. This can be achieved through convolutional layers or a combination of convolutional layers and activation layers, where the activation layers can use sigmoid functions or other activation functions. Figures 11A and 11B show feature maps 1100A and 1100B under different feature extractors, where different colors (grayscale) can represent different feature values ​​or probability values. It can be seen that when the image contains repetitive features, the feature values ​​or probability values ​​in the extracted feature map also have a certain regularity. Therefore, the feature map can be analyzed to determine the feature size and feature center coordinates of the repetitive feature. After determining the feature size and feature center coordinates of the repetitive feature, a two-dimensional grid layout model can be installed on the image. This can be implemented, for example, with the help of an implicit pattern model (IPM), which can be used to detect structure instances based on the feature size and feature center coordinates of the repetitive feature to generate a corresponding grid structure. After obtaining the grid layout, subsequent processing can be performed to convert it into a processed sample image and target label that can be used for the aforementioned model training process. Here, it is assumed that the pattern repetition pattern of the sample image under consideration is static, that is, the repetition pattern of the repetitive pattern in the sample image is unchanged. Therefore, the label information expected to be obtained should contain all parts of the sample image. However, since there may be a lack of information about repetitive features at locations such as the edges of the sample image and the target feature area, these locations are usually not responded to by the selected filter. Therefore, the missing areas can be filled with the already obtained grid layout and the characteristic size of the repetitive features so that the repeated texture grid can be tiled over the entire sample image. Furthermore, this grid map can be processed and labeled to obtain the corresponding processed sample image and target label containing the input label, wherein the input label can include the above-mentioned repeated texture grid to describe the area corresponding to the repetitive feature in the sample image, and the target label can distinguish between the grid containing the target feature and the grid that does not contain the target feature.In some examples, the filter response can capture the non-rigid micro-deformations of the sample to a certain extent. Therefore, adjacent grids can be divided into two groups for annotation to better utilize the spatial distortion information they capture and improve the training effect of the target model.

[0085] In addition, referring to Figures 11A and 11B, the target feature areas are shown in boxes 1101 and 1102, which can be defect areas for example. It can be seen that when considering repetitive features, such target areas can be more obvious and easier to detect accurately. Therefore, in the model training scheme disclosed in the present invention, by considering the repetitive pattern information of repetitive features, the accuracy of the model can be effectively improved. Furthermore, by extracting feature maps with the help of convolutional neural networks, and then extracting the feature size and feature center coordinates of the repetitive features, the cost of extracting and labeling repetitive features can be reduced, thereby helping to improve the overall model training efficiency.

[0086] In some embodiments, each feature map in the aforementioned at least one feature map includes a plurality of feature points, and the feature points may refer to probability peak points corresponding to the features extracted by the corresponding feature extractor. For example, as shown in FIG11B , a plurality of prominent points or point sets 1103 are included, and the pixel points represented by these points or the pixel points falling into these point sets may be regarded as the above-mentioned probability peak points or feature points. Exemplarily, the characteristic size and characteristic center coordinates of the above-mentioned repetitive features may be determined as follows: for at least one feature map, at least one set of offsets is determined, wherein each set of offsets may correspond to a feature map, and each offset in a set of offsets may characterize the distance between two feature points in the feature map; based on the at least one set of offsets, the offset with the highest probability of occurrence may be determined as the characteristic size of the repetitive feature; and further, based on the determined characteristic size, the characteristic center coordinates may be determined.

[0087] For example, taking AlexNet as an example, we can obtain activation layers with the same configuration based on each layer of the network. Through the configured activation layers, we can obtain multiple feature maps, where each feature map can contain multiple feature points. For each feature map, we can construct an offset set in Can represent neural networks Layer l, f in l Can represent the feature extractor at layer 1. Exemplarily, Can be constructed as in, represents the feature extractor f l The probability set of each feature point extracted, p m 、p n Can be expressed The probability value of any two points (point m and point n), d m,n The offset between point m and point n can be represented, for example, in the form of a vector. The offset with the highest probability of occurrence should be the offset between corresponding feature points in adjacent repetitive features. Thus, the offset with the highest probability of occurrence in this offset set can be used to effectively and accurately determine the characteristic size of the repetitive feature. Furthermore, by analyzing multiple different feature maps, the characteristic size of the repetitive feature can be determined more accurately. Alternatively, it is possible to directly count The offset that appears most often in the set, and optionally a weighted average of the offsets that appear most often in different offset sets, for example, weighted based on the number of offsets in each offset set, to obtain the offset with the highest probability of occurrence, or the offset with the highest probability of occurrence can be solved by other means.

[0088] In some embodiments, the characteristic size can be solved in the following manner: each offset in the above-mentioned at least one set of offsets can be substituted into an objective function, and the function value of the objective function can be configured to be negatively correlated with the absolute value of the difference between the target amount and the offset; then the target amount is determined as the characteristic size, so that the weighted sum of the function values ​​of the objective function for each offset is maximized. In the case where the function value of the objective function is negatively correlated with the absolute value of the difference between the target amount and the offset, when the weighted sum of the function values ​​of the objective function for each offset is maximized, the difference between the target amount and each offset can be minimized as a whole. Thus, with the help of the maximization problem related to the objective function, the offset with the highest probability of occurrence can be quantitatively solved, which helps to realize the automated execution of the solution of the characteristic size. At the same time, compared with directly counting the offset with the highest probability of occurrence in the relevant set, this helps to obtain more accurate results.

[0089] For example, after substituting each offset into the objective function, the sum of the function values ​​of the objective function for each offset in the at least one set of offsets can be determined; then, a weighted sum of the sums of the function values ​​corresponding to each set of offsets can be determined, wherein the weight of the sum of the function values ​​corresponding to each set of offsets can be determined based on the number of offsets in the set; and then, a target value can be determined so that the determined weighted sum is maximized. By weighting the sums of the function values ​​corresponding to each offset based on the number of offsets in the set of offsets, the effect of differences in the number of feature points generated by different feature extractors on the result can be eliminated, thereby facilitating the acquisition of more accurate feature sizes.

[0090] For example, the characteristic size can be calculated by the following formula, where the objective function can be:

[0091] in, Represents the feature extractor f l The constructed collection The offset d in m,n The calculated objective function, x, can be regarded as the target quantity to be solved, and Σ can be a quantity related to the activation function. For example, when the Sigmoid function is used, Σ can be:

[0092] Among them, σ l represents the activation function used at layer l, such as the Sigmoid function. In addition, the sum of the function values ​​corresponding to each group of offsets can be weighted by the following formula:

[0093] in, Can represent sets Each d m,n The sum of the moduli, It can represent the set of feature extractors at layer 1. Optionally, at a layer of a convolutional neural network, there can be multiple different feature extractors, such as multiple different filters or convolution kernels. Furthermore, the target quantity can be solved according to the following formula as the feature size to be solved:

[0094] Among them, d * The characteristic size to be solved can be expressed in vector form, H x,0 and H 0,y H can be expressed by substituting (x, 0) and (0, y) as the target quantity x into the above equations (1) to (3). It can be regarded as the characteristic size in the x direction, It can be regarded as a characteristic size in the y direction, and the x direction and the y direction can be directions in a self-defined coordinate system. For example, the x direction and the y direction can extend along the width and height of the sample image respectively.

[0095] In some embodiments, the aforementioned feature center coordinates can be determined by determining the coordinate values ​​of the feature center coordinates so that the coordinate values ​​satisfy the following condition: the sum of the differences between each offset remainder and half of the feature size is minimized, where each offset remainder is the remainder of dividing an offset difference by the feature size, and each offset difference is the difference between the coordinate value and an offset. In this way, the feature center coordinates can be determined with higher efficiency and lower complexity. For example, after determining the feature size, the feature center coordinates can be determined according to the following formula:

[0096] Among them,* It can represent the feature center coordinates, Can represent weights, which can be set or omitted as needed, It can be used to simplify vectors in the moduli space to reduce computational complexity and improve computational efficiency. Specifically, It can be used to perform calculations such as:

[0097] in, Can represent d * The value in the x direction, Can represent d * The value in the y direction, v x It can represent the value of v in the x direction, v y It can represent the value of v in the y direction, and mod can represent the remainder operation. For example, A mod B can refer to the remainder when A is divided by B.

[0098] It should be understood that in the present disclosure, the target feature may refer to any feature that is desired to be detected. In some embodiments, the aforementioned sample image may be an image of the target object, and the target feature may be a defect feature of the target object. For example, a defect may refer to an area in the target object where cracks, device or structure defects, dirt or other problems exist. For example, the area shown in boxes 1101 and 1102 in Figures 11A and 11B may be a defect area, and it can be seen from the figure that there are no repetitive features in this area that exist in other non-defective areas. In such embodiments, the trained target model can be used for defect detection of the target object. For example, Figure 12A shows an image 1200A of the target object, and Figure 12B shows a result image 1200B obtained by processing image 1200A using the target object, wherein the defect area can be identified by a specific color or in other ways, such as the circular area 1201 in the figure. Optionally, the above process can also be described as a semantic segmentation process for target features or defects.

[0099] Figure 13 schematically illustrates an example flow chart of a detection method 1300 according to some embodiments of the present disclosure. For example, method 1300 can be deployed in a computing device, such as an application, such as server 210, terminal device 220, or a combination thereof, in scenario 200 shown in Figure 2 . As shown in Figure 13 , method 1300 may include steps 1310 to 1330, as follows.

[0100] At step 1310, an image of the object to be inspected may be acquired. For example, the object to be inspected may be any object containing repetitive features that is desired to be inspected, such as the aforementioned textile containing a repetitive textile structure, fabric with a repetitive pattern, a device with repetitive components, a circuit board with a repetitive circuit structure, and the like. Alternatively, the image of the object to be inspected may be acquired using a camera or other image acquisition device, or the image of the object to be inspected may be pre-acquired and stored in a local or external storage device.

[0101] In step 1320, the image may be provided to a target model, which may be trained according to various embodiments of the model training method described above.

[0102] At step 1330, based on the output of the target model, a determination can be made as to whether the object to be inspected contains the target feature. For example, the output of the target model can be a series of predicted probability values ​​for the target feature. If some of these probability values ​​are above a preset threshold, the pixel region corresponding to these probability values ​​is considered to contain the target feature. If all probability values ​​are below the preset threshold, the target feature is considered absent from the image. For example, the target feature can be a defect, meaning that detection method 1300 can be used for defect detection.

[0103] It should be understood that the method 1300 may have the same or similar features and advantages as the method 300 described above, and may have corresponding embodiments, which will not be described in detail here for the sake of brevity.

[0104] 14 schematically shows an exemplary block diagram of a model training apparatus 1400 according to some embodiments of the present disclosure. As shown in the figure, the apparatus 1400 includes a first acquisition module 1410, a second acquisition module 1420, a first prediction module 1430, a second prediction module 1440, and an update module 1450.

[0105] Specifically, the acquisition module 1410 can be configured to: acquire a sample image and a sample label, the sample image contains repetitive features, and the sample label is configured to describe the target features contained in the sample image; the extraction module 1420 can be configured to: based on the sample image, acquire a processed sample image and a target label, wherein the processed sample image contains an input label, the input label is configured to describe multiple regions corresponding to the repetitive features, and the target label is configured to describe a target region corresponding to the target features in multiple regions; the first prediction module 1430 can be configured to: input the sample image into the target model to obtain a target feature prediction output, wherein the target model is Configured to extract feature data based on the sample image and generate a target feature prediction output based on the feature data; the second prediction module 1440 can be configured to: input the processed sample image into the auxiliary model to obtain a target area prediction output, wherein the auxiliary model includes the same model structure as the target model, and is configured to extract auxiliary feature data based on the processed sample image, and generate a target area prediction output based on the auxiliary feature data and the feature data; the update module 1450 can be configured to: update the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target area prediction output, the sample label, and the target label to obtain a trained target model.

[0106] Fig. 15 schematically shows an exemplary block diagram of a detection apparatus 1500 according to some embodiments of the present disclosure. As shown in the figure, the apparatus 1500 includes an acquisition module 1510, a providing module 1520, and a determination module 1530.

[0107] Specifically, the acquisition module 1510 can be configured to: acquire an image of the object to be detected; the providing module 1520 can be configured to: provide the image to the target model, which is trained using the device 1400; the determination module 1530 can be configured to: determine whether the object to be detected contains target features based on the output of the target model.

[0108] It should be understood that the apparatus 1400, 1500 can be implemented in software, hardware, or a combination of software and hardware. Multiple different modules can be implemented in the same software or hardware structure, or one module can be implemented by multiple different software or hardware structures.

[0109] Furthermore, apparatus 1400 can be used to implement method 300 described above, and apparatus 1500 can be used to implement method 1300 described above. The relevant details have been described in detail above and will not be repeated here for the sake of brevity. Apparatuses 1400 and 1500 can have the same features and advantages as those described with respect to the aforementioned methods.

[0110] Figure 16 schematically shows an example block diagram of an electronic device 1600 according to some embodiments of the present disclosure. For example, it may represent the server 210 or the terminal device 220 in Figure 2, or may represent other types of computing devices for deploying the apparatus 1400 or 1500 provided by the present disclosure.

[0111] As shown, example computing device 1600 includes a processing system 1601, one or more computer-readable media 1602, and one or more I / O interfaces 1603 that are communicatively coupled to one another. Although not shown, computing device 1600 may also include a system bus or other data and command transmission system that couples the various components to one another. The system bus may include any one or a combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures, or may also include, for example, control and data lines.

[0112] Processing system 1601 represents functionality that uses hardware to perform one or more operations. Thus, processing system 1601 is illustrated as including hardware elements 1604 that can be configured as processors, functional blocks, and the like. This can include implementing application-specific integrated circuits (ASICs) or other logic devices formed using one or more semiconductors in hardware. Hardware elements 1604 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor can be comprised of semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor-executable instructions can be electronically executable instructions.

[0113] Computer-readable medium 1602 is illustrated as including memory / storage device 1605. Memory / storage device 1605 represents a memory / storage device associated with one or more computer-readable media. Memory / storage device 1605 may include volatile storage media (such as random access memory (RAM)) and / or non-volatile storage media (such as read-only memory (ROM), flash memory, optical disk, magnetic disk, etc.). Memory / storage device 1605 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Exemplarily, memory / storage device 1605 can be used to store the sample images mentioned in the above embodiments, input labels and target labels extracted based on the sample images, processed sample images, images of objects to be detected, etc. The computer-readable medium 1602 can be configured in various other ways as further described below.

[0114] One or more input / output interfaces 1603 represent functions that allow a user to type commands and information into the computing device 1600 and also allow the information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, a touch function (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., a device that can detect motion that does not involve touch as a gesture using visible or invisible wavelengths (such as infrared frequencies)), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a monitor or projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like. Exemplarily, in the embodiments described above, the input device can be used to allow the user to provide sample images for model training, control the model training process, provide images of objects to be detected to the trained model, and the like, and the output device can be used to allow the user to view the model training process, view model output results, and the like.

[0115] The computing device 1600 also includes a model training application or a detection application 1606. The model training application or the detection application 1606 can be stored as computer program instructions in the memory / storage device 1605. The model training application or the detection application 1606 can, together with the processing system 1601 and the like, implement all the functions of the various modules of the model training device 1400 described with respect to FIG. 14 or the detection device 1500 described with respect to FIG. 15.

[0116] Various techniques may be described herein in the general context of software, hardware, elements, or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. As used herein, the terms "module," "function," etc. generally refer to software, firmware, hardware, or a combination thereof. A feature of the techniques described herein is that they are platform-independent, meaning that these techniques can be implemented on a variety of computing platforms with a variety of processors.

[0117] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include various media accessible by computing device 1600. By way of example, and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0118] As opposed to a mere signal transmission, carrier wave, or signal itself, "computer-readable storage media" refers to media and / or devices, and / or tangible storage devices, capable of persistently storing information. Thus, computer-readable storage media refers to non-signal-bearing media. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-executable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, hard disks, cassettes, magnetic tape, magnetic disk storage devices or other magnetic storage devices, or other storage devices, tangible media, or articles of manufacture suitable for storing desired information and accessible by a computer.

[0119] "Computer-readable signal media" refers to signal-bearing media configured to transmit instructions to the hardware of computing device 1600, such as via a network. Signal media typically embodies computer-executable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also includes any information transmission media. By way of example, and not limitation, signal media include wired media such as a wired network or direct connection, and wireless media such as acoustic, RF, infrared, and other wireless media.

[0120] As previously described, hardware elements 1604 and computer-readable media 1602 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include other implementations in integrated circuits or systems on a chip, application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and silicon or other hardware devices. In this context, hardware elements can be used as processing equipment for executing program tasks defined by the instructions, modules, and / or logic embodied by the hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media previously described.

[0121] The aforementioned combinations may also be used to implement the various techniques and modules described herein. Thus, software, hardware, or program modules and other program modules may be implemented as one or more instructions and / or logic on some form of computer-readable storage medium and / or embodied by one or more hardware elements 1604. Computing device 1600 may be configured to implement specific instructions and / or functions corresponding to software and / or hardware modules. Thus, for example, by using a computer-readable storage medium and / or hardware elements 1604 of a processing system, a module may be implemented as a module executable by computing device 1600 as software, at least in part, in hardware. Instructions and / or functions may be executed / operable by, for example, one or more computing devices 1600 and / or processing system 1601 to implement the techniques, modules, and examples described herein.

[0122] The techniques described herein may be supported by these various configurations of computing device 1600 and are not limited to the specific examples of the techniques described herein.

[0123] It should be understood that, for the sake of clarity, embodiments of the present disclosure have been described with reference to different functional units. However, it will be apparent that, without departing from the present disclosure, the functionality of each functional unit can be implemented in a single unit, in multiple units, or as a part of other functional units. For example, the functionality described as being performed by a single unit can be performed by multiple different units. Therefore, reference to a specific functional unit is only considered as a reference to the appropriate unit for providing the described functionality, rather than indicating a strict logical or physical structure or organization. Therefore, the present disclosure can be implemented in a single unit, or can be physically and functionally distributed between different units and circuits.

[0124] The present disclosure provides a computer-readable storage medium having computer-executable instructions stored thereon, which implement the above-mentioned model training method or detection method when executed.

[0125] The present disclosure provides a computer program product or computer program, which includes computer-executable instructions stored in a computer-readable storage medium. A processor of a computing device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the computing device to perform the model training method or detection method provided in the various embodiments described above.

[0126] Variations on the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure, and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. A model training method, comprising: Acquire a sample image and a sample label, wherein the sample image contains repetitive features, and the sample label is configured to describe target features contained in the sample image; Based on the sample image, obtaining a processed sample image and a target label, wherein the processed sample image includes an input label, the input label is configured to describe a plurality of regions corresponding to the repetitive feature, and the target label is configured to describe a target region corresponding to the target feature among the plurality of regions; Inputting the sample image into a target model to obtain a target feature prediction output, wherein the target model is configured to extract feature data based on the sample image and generate the target feature prediction output based on the feature data; inputting the processed sample image into an auxiliary model to obtain a target region prediction output, wherein the auxiliary model comprises the same model structure as the target model and is configured to extract auxiliary feature data based on the processed sample image, and generate the target region prediction output based on the auxiliary feature data and the feature data; Based on the target feature prediction output, the target region prediction output, the sample label, and the target label, the model parameters of the target model and the auxiliary model are updated to obtain a trained target model.

2. The method according to claim 1, wherein The processed sample image includes a first processed image and a second processed image, the first processed image includes a first input label, the second processed image includes a second input label, the first input label is configured to describe a plurality of first regions corresponding to a first portion of repetitive features included in the sample image, the second input label is configured to describe a plurality of second regions corresponding to a second portion of repetitive features included in the sample image, and the first portion of repetitive features and the second portion of repetitive features are alternately distributed in the sample image. The target tag includes a first target tag and a second target tag, the first target tag is configured to describe a first target area corresponding to the target feature in the plurality of first areas, and the second target tag is configured to describe a second target area corresponding to the target feature in the plurality of second areas. The step of inputting the processed sample image into the auxiliary model to obtain the target area prediction output comprises: inputting the first processed image into the first auxiliary model to obtain the first a target region prediction output, and inputting the second processed image into a second auxiliary model to obtain a second target region prediction output, wherein the first auxiliary model comprises the same model structure as the target model and is configured to extract first auxiliary feature data based on the first processed image, and generate the first target region prediction output based on the first auxiliary feature data and the feature data, The second auxiliary model includes the same model structure as the target model, and is configured to extract second auxiliary feature data based on the second processed image, and generate the second target area prediction output based on the second auxiliary feature data and the feature data.

3. The method according to claim 1, wherein The updating of the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target region prediction output, the sample label, and the target label includes: In response to the number of updates of the model parameters of the target model not reaching a preset threshold, synchronously updating the model parameters of both the target model and the auxiliary model based on the target feature prediction output and the target region prediction output, the sample label, and the target label; In response to the number of updates of the model parameters of the target model reaching a preset threshold, only the model parameters of the target model are updated based on the target feature prediction output and the target area prediction output as well as the sample label and the target label.

4. The method according to claim 1, wherein The acquiring, based on the sample image, a processed sample image and a target label comprises: Based on the sample image, extract at least one feature map through a convolutional neural network; Determining a feature size and a feature center coordinate of the repetitive feature based on the at least one feature map; Determining a two-dimensional grid map based on the feature size and the feature center coordinates, wherein the two-dimensional grid map includes a plurality of grid regions, each grid region corresponding to a region of a repetitive feature in the sample image; Based on the two-dimensional grid map, the processed sample image and the target label are determined.

5. The method according to claim 4, wherein Each of the at least one feature map includes a plurality of feature points, each of which is a probability peak point of a corresponding feature, and wherein, based on the at least one feature map, determining the characteristics of the repetitive feature Dimensions and feature center coordinates include: Determining at least one set of offsets for the at least one feature map, wherein each set of offsets in the at least one set of offsets corresponds to a feature map, and each offset in the set of offsets represents a distance between two feature points in the feature map; Based on the at least one set of offsets, determining an offset with the highest probability of occurrence as the characteristic size; Based on the feature size, the feature center coordinates are determined.

6. The method according to claim 5, wherein: The determining, based on the at least one set of offsets, an offset with the highest probability of occurrence as the characteristic size comprises: Substituting each offset in the at least one set of offsets into a target function, wherein a function value of the target function is configured to be negatively correlated with an absolute value of a difference between the target value and the offset; The target amount is determined as the characteristic size so that a weighted sum of function values of the target function for each offset amount is maximized.

7. The method according to claim 6, wherein: Determining the target amount so that a weighted sum of function values of the target function for each offset is maximized includes: For each set of offsets in the at least one set of offsets, determining a sum of function values of the objective function for each offset in the set of offsets; Determining a weighted sum of the sums of the function values corresponding to each set of offsets, wherein the weight of the sum of the function values corresponding to each set of offsets is determined based on the number of offsets in the set of offsets; The target quantity is determined so that the weighted sum is maximized.

8. The method according to claim 5, wherein Determining the feature center coordinates based on the feature size of the repetitive feature includes: Determine the coordinate values of the feature center coordinates so that the coordinate values satisfy the following conditions: the sum of the differences between each offset remainder and half of the feature size is minimized, wherein each offset remainder is the remainder of dividing an offset difference by the feature size, and each offset difference is the difference between the coordinate value and an offset amount.

9. The method according to claim 1, wherein The updating of the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target region prediction output, the sample label, and the target label includes: Determining a first loss based on the target feature prediction output and the sample label; Determining a second loss based on the target region prediction output and the target label; Based on a combination of the first loss and the second loss, model parameters of the target model and the auxiliary model are updated.

10. The method according to claim 9, wherein: The determining of the first loss based on the target feature prediction output and the sample label includes: Determining an adjustment factor corresponding to each predicted value in the target feature prediction output, wherein the adjustment factor is negatively correlated with the predicted value; Determine a first loss based on the target feature prediction output, the sample label and the corresponding adjustment factor, The determining of the second loss based on the target region prediction output and the target label includes: Determining an adjustment factor corresponding to each predicted value in the target area prediction output, wherein the adjustment factor is negatively correlated with the predicted value; A second loss is determined based on the target region prediction output, the target label, and a corresponding adjustment factor.

11. The method according to claim 1, wherein The target model includes a target encoder and a target decoder, wherein the target encoder is configured to extract shallow feature data and deep feature data from the sample image, and the target decoder is configured to generate the target feature prediction output based on the deep feature data and the shallow feature data. The auxiliary model includes an auxiliary encoder and an auxiliary decoder, the auxiliary encoder is configured to extract shallow auxiliary feature data and deep auxiliary feature data from the processed sample image, and the auxiliary decoder is configured to generate the target area prediction output based on the deep auxiliary feature data, the auxiliary shallow feature data, the deep feature data and the shallow feature data.

12. The method according to claim 11, wherein The target encoder includes a plurality of target coding layers, the auxiliary encoder includes a plurality of auxiliary coding layers, The target decoder is configured to: receive shallow feature data from an intermediate target coding layer among the multiple target coding layers and deep feature data from an end target coding layer among the multiple target coding layers; concatenate the upsampled deep feature data and the shallow feature data to obtain concatenated feature data; and generate the target feature prediction output based on the concatenated feature data. The auxiliary decoder is configured to: receive the shallow feature data and the deep feature data; receive the shallow feature data from the middle auxiliary coding layer in the plurality of auxiliary coding layers; layer auxiliary feature data and deep auxiliary feature data from the terminal auxiliary coding layer among the multiple auxiliary coding layers; fusing the shallow feature data and the shallow auxiliary feature data to obtain shallow fused auxiliary feature data; fusing the deep feature data and the deep auxiliary feature data to obtain deep fused auxiliary feature data; splicing the upsampled deep fused auxiliary feature data and the shallow fused auxiliary feature data to obtain spliced auxiliary feature data; generating the target area prediction output based on the spliced auxiliary feature data.

13. The method according to claim 11, wherein At least one coding layer among the plurality of target coding layers and the plurality of auxiliary coding layers comprises a feature extraction unit, wherein the feature extraction unit is configured to: Receive superior feature data; Based on the upper-level feature data, a set of feature maps are generated through a convolution operation; Based on the set of feature maps, generating another set of feature maps through linear transformation; Based on the set of feature maps and the other set of feature maps, feature data of this level is output.

14. A method according to any one of the preceding claims, wherein The sample image is an image of a target object, and the target feature is a defect feature of the target object.

15. A detection method comprising: Acquire an image of the object to be detected; Providing the image to a target model, wherein the target model is trained according to the method of any one of claims 1 to 14; Based on the output of the target model, it is determined whether the object to be detected contains target features.

16. A model training device comprising: A first acquisition module is configured to: acquire a sample image and a sample label, wherein the sample image includes repetitive features, and the sample label is configured to describe target features included in the sample image; A second acquisition module is configured to: acquire a processed sample image and a target label based on the sample image, wherein the processed sample image includes an input label, the input label is configured to describe a plurality of regions corresponding to the repetitive feature, and the target label is configured to describe a target region corresponding to the target feature among the plurality of regions; a first prediction module configured to: input the sample image into a target model to obtain a target feature prediction output, wherein the target model is configured to extract feature data based on the sample image and generate the target feature prediction output based on the feature data; a second prediction module, configured to: input the processed sample image into an auxiliary model to obtain a target area prediction output, wherein the auxiliary model includes the same model structure as the target model, and is configured to extract auxiliary feature data based on the processed sample image, and generate the target area prediction output based on the auxiliary feature data and the feature data; The updating module is configured to update the model parameters of the target model and the auxiliary model based on the target feature prediction output, the target area prediction output, the sample label, and the target label to obtain a trained target model.

17. A detection device comprising: The acquisition module is configured to: acquire an image of the object to be detected; A providing module is configured to: provide the image to a target model, wherein the target model is trained using the apparatus according to claim 16; The determination module is configured to determine whether the object to be detected contains target features based on the output of the target model.

18. A computing device comprising: a memory configured to store computer-executable instructions; A processor configured to perform the method according to any one of claims 1 to 15 when the computer executable instructions are executed by the processor.

19. A computer-readable storage medium storing computer-executable instructions, which, when executed, perform the method according to any one of claims 1 to 15.

20. A computer program product comprising computer executable instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images