Anomaly detection model training method and apparatus, and item anomaly detection method and apparatus

Abstract

An anomaly detection model training method, which is executed by means of a computer device. The method comprises: a computer device acquiring a plurality of first sample images, and generating a plurality of sample pairs on the basis of the plurality of first sample images, wherein each first sample image includes a sample item image, and each sample pair comprises a sample image block and at least one piece of sample text, the sample image block including sample item local images of the sample item image, and the sample text being used for describing anomalies of the sample item local images (201); for each sample pair, the computer device performing feature extraction on the sample image block in the sample pair by means of an anomaly detection model, so as to obtain an image feature of the sample image block, performing feature extraction on the sample text in the sample pair by means of the anomaly detection model, so as to obtain a text feature of the sample text, and determining the similarity between the image feature and the text feature, wherein the anomaly detection model is used for performing anomaly detection on target item local images in an input image block (202); and on the basis of the respective similarities of the plurality of sample pairs and a preset similarity, the computer device iteratively training the anomaly detection model (203).

Description

Training methods for anomaly detection models, methods and devices for anomaly detection of objects

[0001] Related applications

[0002] This application claims priority to Chinese patent application filed on April 7, 2024, with application number 202410405801.X, entitled "Training method for an anomaly detection model, object anomaly detection method and apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0003] This application relates to the field of artificial intelligence technology, and in particular to a training method for an anomaly detection model, an object anomaly detection method, and an apparatus. Background Technology

[0004] Anomaly detection plays a crucial role in actual production. When detecting anomalies in industrial products, the general process involves first establishing the distribution characteristics of normal products. Then, by judging the similarity between the characteristics of the product to be tested and these distribution characteristics, the normality of the product is determined. However, before establishing the distribution characteristics of normal products, a large number of normal products need to be collected; that is, abnormal products need to be detected and removed from this large pool of products.

[0005] In related technologies, clustering is used to detect anomalous items. The basic assumption of this method is that anomalous items account for a low percentage in actual production. Therefore, this method groups items into clusters based on similarity. Because the proportion of anomalous items is low, after clustering, small, isolated clusters are highly likely to contain anomalous items, which can then be removed. However, the proportion of anomalous items in actual production may not be low. This can lead to some anomalous items clustering into larger clusters after clustering, causing confusion with normal items and rendering the detection method inaccurate. Summary of the Invention

[0006] This application provides a method for training an anomaly detection model, a method for detecting anomalies in objects, and an apparatus for such detection. Anomaly detection based on the trained model obtained through this method can improve the accuracy of anomaly detection. The technical solution is as follows.

[0007] On the one hand, a method for training an anomaly detection model is provided, executed by a computer device, the method comprising:

[0008] Multiple first sample images are acquired, and multiple sets of sample pairs are generated based on the multiple first sample images. Each first sample image contains a sample item image, and each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a sample item partial image of the sample item image, and the sample text is used to describe the abnormal situation of the sample item partial image.

[0009] For each sample pair, features are extracted from the sample patches in the sample pair using an anomaly detection model to obtain the image features of the sample patches. Similarly, features are extracted from the sample text in the sample pair using the same anomaly detection model to obtain the text features of the sample text. The similarity between the image features and the text features is determined. The anomaly detection model is used to perform anomaly detection on the local image of the target object in the input patch.

[0010] The anomaly detection model is iteratively trained based on the similarity of the multiple sets of samples and a preset similarity.

[0011] On the other hand, a training device for an anomaly detection model is provided, the device comprising:

[0012] The acquisition module is used to acquire multiple first sample images and generate multiple sets of sample pairs based on the multiple first sample images. Each first sample image contains a sample item image. Each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a sample item partial image of the sample item image, and the sample text is used to describe the abnormal situation of the sample item partial image.

[0013] The extraction module is used to extract features from the sample patches in each sample pair using an anomaly detection model to obtain the image features of the sample patches, and to extract features from the sample text in the sample pair using the anomaly detection model to obtain the text features of the sample text, and to determine the similarity between the image features and the text features. The anomaly detection model is used to perform anomaly detection on the local image of the target item in the input patch.

[0014] The training module is used to iteratively train the anomaly detection model based on the similarity of the multiple sets of samples and a preset similarity.

[0015] On the other hand, a method for detecting anomalies in items is provided, the method comprising:

[0016] Acquire multiple tiles of an image, wherein the image contains an image of a target object, and each tile contains a partial image of the target object from the image of the target object;

[0017] For each tile, the target text corresponding to the tile is determined by an anomaly detection model. The anomaly detection model is obtained by the above training method. The target text is used to describe the anomaly of the local image of the target item in the tile.

[0018] Based on the target text corresponding to the multiple image tiles, the abnormal information of the target item image in the image is determined, and the abnormal information is used to describe the abnormal situation of the target item image.

[0019] Furthermore, an anomaly detection device for an item is provided, the device comprising:

[0020] The acquisition module is used to acquire multiple tiles of an image, wherein the image contains an image of a target item, and each tile contains a partial image of the target item from the image of the target item;

[0021] The determination module is used to determine the target text corresponding to each tile through an anomaly detection model. The target text is used to describe the anomaly of the local image of the target item in the tile.

[0022] The determining module is further configured to determine the abnormal information of the target item image in the image based on the target text corresponding to the plurality of image tiles respectively, and the abnormal information is used to describe the abnormal situation of the target item image.

[0023] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one program, the at least one program being loaded and executed by the processor to implement the training method of the anomaly detection model or the item anomaly detection method in the embodiments of this application.

[0024] On the other hand, a computer-readable storage medium is provided, wherein at least one program is stored in the computer-readable storage medium, the at least one program being loaded and executed by a processor to implement the training method of the anomaly detection model or the item anomaly detection method in the embodiments of this application.

[0025] On the other hand, a computer program product is provided, the computer program product comprising at least one program segment stored in a computer-readable storage medium, a processor of a computer device reading the at least one program segment from the computer-readable storage medium, the processor executing the at least one program segment, causing the computer device to execute the training method for the anomaly detection model or the item anomaly detection method described in any of the above implementations.

[0026] Details of one or more embodiments of this application are set forth in the following drawings and description. Other features, objects, and advantages of this application will become apparent from the specification, drawings, and claims. Attached Figure Description

[0027] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the published drawings without creative effort.

[0028] Figure 1 is a schematic diagram of an implementation environment provided in an embodiment of this application;

[0029] Figure 2 is a flowchart of a training method for an anomaly detection model provided in an embodiment of this application;

[0030] Figure 3 is a flowchart of another training method for an anomaly detection model provided in an embodiment of this application;

[0031] Figure 4 is a schematic diagram of an image calibration provided in an embodiment of this application;

[0032] Figure 5 is a flowchart of an article anomaly detection method provided in an embodiment of this application;

[0033] Figure 6 is a flowchart of another article anomaly detection method provided in an embodiment of this application;

[0034] Figure 7 is a schematic diagram of an industrial anomaly detection process provided in an embodiment of this application;

[0035] Figure 8 is a flowchart of the training phase in industrial testing provided in an embodiment of this application;

[0036] Figure 9 is a flowchart of the testing phase in industrial testing provided in an embodiment of this application;

[0037] Figure 10 is a block diagram of a training device for an anomaly detection model provided in an embodiment of this application;

[0038] Figure 11 is a block diagram of an anomaly detection device for an article provided in an embodiment of this application;

[0039] Figure 12 is a block diagram of a terminal provided in an embodiment of this application;

[0040] Figure 13 is a block diagram of a server provided in an embodiment of this application. Detailed Implementation

[0041] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0042] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are they limiting in terms of quantity or execution order. In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0043] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample images involved in this application were all obtained with full authorization.

[0044] The following describes the implementation environment involved in this application:

[0045] The training method for the anomaly detection model provided in this application can be executed by a computer device, which can be a server or a terminal. The following is a schematic diagram illustrating the implementation environment of the training method for the anomaly detection model provided in this application.

[0046] Referring to Figure 1, which is a schematic diagram of the implementation environment of a training method for an anomaly detection model provided in an embodiment of this application, the implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected via wired or wireless communication, which is not limited herein. In some embodiments, the server 102 is used to train an anomaly detection model, and the trained anomaly detection model is used to determine the text corresponding to the image patch, which describes the anomaly of a local image of a sample item included in the image patch. A target application is installed on the terminal 101, which is used to perform anomaly detection on the target item. In some embodiments, the trained anomaly detection model is embedded on the terminal 101, and the terminal 101 performs anomaly detection on the item using the anomaly detection model. In other embodiments, the terminal 101 performs anomaly detection on the item using the anomaly detection model on the server 102.

[0047] In some embodiments, terminal 101 may be a smartphone, tablet, laptop, desktop computer, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, VR (Virtual Reality) device, AR (Augmented Reality) device, etc., but is not limited thereto. In some embodiments, server 102 may be an independent server, a server cluster composed of multiple servers, or a distributed system. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. In some embodiments, server 102 mainly undertakes computing work, and terminal 101 undertakes secondary computing work; or, server 102 undertakes secondary computing services, and terminal 101 undertakes primary computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0048] Referring to Figure 2, which is a flowchart of a training method for an anomaly detection model provided in an embodiment of this application, the method includes the following steps.

[0049] 201. A computer device acquires multiple first sample images and generates multiple sets of sample pairs based on the multiple first sample images. Each first sample image contains a sample item image. Each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a partial image of the sample item image, and the sample text is used to describe the abnormal situation of the partial image of the sample item.

[0050] In this embodiment, the sample item image can be an image formed by imaging the target item. The target item can be any item to be detected as having an anomaly, such as various parts produced in industry. Multiple sample patches are obtained based on the segmentation of the first sample image, and one first sample image can be segmented into multiple sample patches.

[0051] The multiple first sample images all contain images of sample items. A sample item image is an image that visually represents the shape of the sample item. Sample item images may also show the colors or patterns on the sample item. The sample items in the first sample images may be in the same or different states; differences in state include differences in size, shooting angle, brightness, etc. The multiple first sample images include images of sample items without anomalies and images of sample items with anomalies.

[0052] In this embodiment, "abnormal situation" refers to either the presence or absence of anomalies in a localized area. If the sample item is a part, then "local abnormality" means that there are defects in a localized part, such as an uneven surface, scratches on the surface, or insufficient dimensions. Furthermore, in the case of local abnormalities in the sample image, the sample text is also used to describe the category of the abnormality, such as an uneven surface, scratches on the surface, or insufficient dimensions.

[0053] Each sample pair includes one sample patch and at least one sample text. When multiple sample texts exist, the average feature value of the multiple sample texts is used as the text feature of the sample pair for training. For each sample pair, if the local image of the sample item included in the sample patch of the sample pair does not contain anomalies, then the sample text in the sample pair describes that the local image of the sample patch does not contain anomalies. If the local image of the sample item included in the sample patch of the sample pair contains anomalies, then the sample text in the sample pair describes that the local image of the sample patch contains anomalies, or the sample text in the sample pair describes the type of anomaly present in the local image of the target item in the sample patch.

[0054] It should be noted that, since the sample item has multiple different parts, such as the beginning and end, the middle part, the head of the part, the tail of the part, etc., the sample text can optionally also describe the name of the sample item's local image included in the sample patch, so that the trained anomaly detection model can also determine which part the patch includes.

[0055] 202. For each sample pair, the computer equipment uses an anomaly detection model to extract features from the sample image blocks and sample text in the sample pair, obtaining the image features of the sample image blocks and the text features of the sample text, and determining the similarity between the image features and the text features. The anomaly detection model is used to detect anomalies in the local image of the target item in the input image block.

[0056] In this embodiment, the anomaly detection model includes an image encoder and a text encoder. The image encoder is used to encode the image to obtain image features. The text encoder is used to encode the text to obtain text features.

[0057] In this embodiment, image features are vectors or matrices used to represent sample image patches. Text features are vectors or matrices used to represent sample text. Similarity can be cosine similarity.

[0058] 203. The computer equipment iteratively trains the anomaly detection model based on the similarity of multiple sample pairs and the preset similarity.

[0059] In this embodiment, the preset similarity can be set and changed as needed. The training objective of the anomaly detection model is to achieve a preset similarity between the feature similarity of sample image patches and sample text in a sample pair. In this embodiment, the computer device determines a loss value based on the similarity of the sample pair and the preset similarity, and iteratively adjusts the model parameters of the anomaly detection model based on the loss value. The model parameters of the anomaly detection model include the parameters of the image encoder and the parameters of the text encoder. The preset similarity can be determined through multiple experiments based on the distribution of the actual training data and the model performance requirements, such as taking a value in the range of [0.8, 0.9].

[0060] In this embodiment, the computer device iteratively trains the anomaly detection model based on the similarity of multiple samples and a preset similarity, until a preset requirement is met. Meeting the preset requirement can be achieved by the loss value converging, the loss value reaching a preset threshold, the number of iterations reaching a preset number, or the similarity reaching a preset similarity; no specific limitation is made here.

[0061] In this embodiment, the trained anomaly detection model is used to extract image features from the input image patch, determine target text features from multiple preset text features that meet preset similarity requirements with the image features, and use the text corresponding to the target text features as the text corresponding to the image patch. This text is used to describe the anomaly in the local image of the target item in the image patch. Optionally, the multiple preset text features are obtained by extracting features from the multiple preset texts separately using the trained anomaly detection model. The target item can be an item that is the same as or similar to the sample item.

[0062] This application provides a training method for an anomaly detection model. This method trains the anomaly detection model based on sample image tiles and sample text describing anomalies in local images of sample items within the sample image tiles. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between image and text features and a preset similarity. This allows the anomaly detection model to learn the rule of high similarity between pairs of image and text features. Therefore, the trained anomaly detection model can accurately extract features. For any image tile containing a local image of a target item, the extracted image features can determine text features with high similarity. The text corresponding to this text feature is then the text describing the anomaly in the local image of the target item within the image tile. Therefore, the anomaly detection model trained using this method has high accuracy in detecting anomalies in local images of target items within image tiles. Furthermore, since this method is trained on image tiles, and image tiles contain local images of sample items, the anomaly detection model can specifically detect which local image of the target item contains anomalies, thus improving the accuracy of anomaly detection. Therefore, using an anomaly detection model trained based on this method can improve the accuracy and precision of anomaly detection.

[0063] Figure 2 above illustrates the basic flow of the training method for the anomaly detection model. The training method for the anomaly detection model will be further described below based on Figure 3. Referring to Figure 3, which is a flowchart of a training method for an anomaly detection model provided in an embodiment of this application, the method includes the following steps.

[0064] 301. A computer device acquires multiple second sample images, each second sample image containing a sample item image. For each second sample image, a sample item image is segmented from the second sample image to obtain multiple first sample images, each first sample image containing a sample item image.

[0065] In this embodiment, the second sample image includes not only the sample item image but also a background containing other interfering elements. Therefore, the sample item image is segmented from the second sample image to distinguish it from the background, thereby removing the background and leaving only the sample item image in the second sample image, thus obtaining the first sample image.

[0066] Optionally, the computer device performs image segmentation using a SAM (Segment Anything Model). The computer device inputs a second sample image into the SAM model, which segments the sample items from the second sample image to distinguish the foreground from the background, thereby removing background noise interference. Further, the computer device sets the pixel values ​​of the segmented background to 0, i.e., sets the background to black. Correspondingly, the sample items in the first sample image are considered the foreground, and the background in the first sample image excluding the sample items is black.

[0067] In this embodiment, by segmenting the sample item from the second sample image, the background in the second sample image is removed, thus avoiding interference caused by the background and facilitating subsequent image processing.

[0068] In some embodiments, the computer device further calibrates the position of the sample item in the overall image of the sample image. Specifically, the computer device acquires multiple third sample images, each containing an image of the sample item; for each third sample image, the position of the sample item in the third sample image is calibrated based on the position of the sample item in the template image to obtain a first sample image, wherein the position of the sample item in the first sample image matches the position of the sample item in the template image.

[0069] The matching of the position of the sample item image in the first sample image with the position of the sample item image in the template image means that the two positions are the same or the coordinate difference between the two positions is within a preset range.

[0070] Optionally, the computer device inputs the third sample image into the image calibration module. The image calibration module calibrates the position of the sample item image in the third sample image based on the position of the sample item image in the template image, thus obtaining the first sample image. The image calibration module is used to calibrate the input image to facilitate accurate extraction of features from the region containing the sample item image in the subsequent image.

[0071] Optionally, the image calibration module uses the minimum error iteration method for calibration. This method constructs an error function, defines the parameters to be estimated, and optimizes the parameters using an iterative algorithm based on the current estimated values ​​to gradually reduce the error function. Specifically, the image calibration module first performs edge detection on the sample items in the input image and the template image to obtain their respective contour maps. Then, it performs non-maximum suppression (NMS) on the contour maps to convert them into two-dimensional contour point sets. Alternatively, it partitions the contour maps to obtain two-dimensional contour point sets for each region in the image, thus partitioning the image. The average error between the two-dimensional contour point sets of the input image and the template image is the error function. The error function is shown in formula (1).

[0072] Where F represents the error function, i.e., the average error, K represents the number of contour points in the template image, and P K The coordinates of the k-th contour point in the input image. This represents the coordinates of the k-th contour point in the template image, and ∑ represents accumulation.

[0073] After setting the error function, an iterative optimization method is used to calibrate the input image. First, the currently estimated transformation matrix, i.e., the initial transformation matrix, is applied to the set of contour points to be registered for registration. Then, a nearest neighbor search method is used to establish the positional matching relationship between the contour point sets of the input image and the template image, and matching pairs with excessive errors are removed, i.e., contour points whose positions are difficult to match are removed. Finally, the RANSAC (Random Sample Consensus) algorithm is used to estimate the transformation matrix again, and the process returns to the first step until the error function converges, obtaining the target transformation matrix. Based on the target transformation matrix, the coordinates of each point in the input image are transformed to obtain the calibrated image.

[0074] The nearest neighbor search method is an algorithm used to establish positional matching relationships between sets of points. In image calibration, it is used to establish positional matching relationships between the contour point sets of the input image and the template image, and it will remove matching pairs with excessive errors, that is, remove contour points whose positions are difficult to match.

[0075] The RANSAC (Random Sample Consensus) algorithm is used in image calibration to re-estimate the transformation matrix. First, the currently estimated transformation matrix is ​​applied to the set of contour points to be registered. Then, a nearest neighbor search method is used to establish the positional matching relationship between the contour point sets of the input image and the template image, eliminating matching pairs with excessive errors. Finally, the algorithm is used again to estimate the transformation matrix until the error function converges, yielding the target transformation matrix.

[0076] In other embodiments, the computer device may also employ a photometric loss function for unsupervised image alignment learning to calibrate the image, without specific limitations. The photometric loss function is a function used for unsupervised image alignment learning, and in this application, it can be used for calibrating the position of a sample item image within a sample image.

[0077] In this embodiment, the position of the sample item image in the sample image is calibrated to facilitate subsequent processing by extracting the area where the sample item image is located in the sample image, such as facilitating the segmentation of the sample item image from the image, facilitating the determination of the area where the sample item image is located in the image, and facilitating the extraction of features of the sample item image in the image.

[0078] For example, referring to Figure 4, which is a schematic diagram of an image calibration provided in an embodiment of this application, the target transformation matrix between the template image and the input image is first determined. Then, the input image is processed based on the target transformation matrix to minimize the positional difference between the input image and the sample item image in the template image, thereby obtaining the calibrated image. The orientation and angle of the sample item image in the calibrated image are the same as those of the sample item image in the template image.

[0079] In some embodiments, the computer device calibrates the position of the sample item image before segmenting the second sample image, thereby facilitating image segmentation. Alternatively, the computer device calibrates the position of the sample item image after segmenting the second sample image; in this case, background interference is eliminated, thus facilitating the calibration of the sample item image's position.

[0080] 302. For each first sample image, the computer device segments the first sample image using sliding windows of multiple sizes to obtain multiple tile sets of different sizes. Each tile set of a size contains multiple sample tiles of a size, and each sample tile contains a partial image of a sample item from a sample item image.

[0081] A sliding window is a window of a certain size used for local processing of an image during image segmentation. This window slides across the image according to certain rules, operating on the regions it passes through to obtain different local information about the image. In this application, multiple sliding windows of different sizes are used to segment the first sample image or the image containing the target object, respectively, to obtain tile sets of different sizes.

[0082] In this embodiment, the size of each sliding window can be set and changed as needed, and is not specifically limited here. For example, the resolution of the first sample image x is h×w, and image segmentation is performed given an encoder f. Multiple sliding windows of different sizes are represented as {w}. ij}, each sliding window {w ij}∈{0,1} h×w It is a binary mask used to represent the local activity of a k×k kernel around pixel (i, j). i, j, h, w, and k are all non-negative integers. The sample patch corresponding to each sliding window is represented as... Segmentation is performed using sliding windows of multiple sizes. The effective area of ​​the first sample image, i.e., a sample patch, is defined as... ⊙ represents element-wise product. The kernel size k corresponds to the amount of surrounding context at each location in the computed image, controlling the balance between local details and global information during segmentation. The kernel size controls the balance between local details and global information during image segmentation when using a sliding window.

[0083] In some embodiments, the image encoder in the anomaly detection model can automatically segment the input image. In other embodiments, if the anomaly detection model is unable to segment the input image, the computer device segments the image before inputting it into the anomaly detection model.

[0084] In some embodiments, the computer device performs image segmentation using the ViT patch (Vision Transformer) method, which includes sliding windows of multiple sizes. For example, a smaller 2×2 sliding window corresponds to an image with a resolution of 32×32 pixels, and a medium-sized 3×3 sliding window corresponds to an image with a resolution of 48×48 pixels. In other embodiments, the computer device performs image segmentation using the ViT token (pattern) method to capture the context of the image.

[0085] In this embodiment, the first sample image is segmented by sliding windows of multiple sizes, so that each first sample image corresponds to multiple sample patches of multiple sizes. The sample patches of different sizes include local parts of the sample items from small to large. Thus, by using sample patches of multiple sizes, defects in the sample items in the sample image from small to large can be captured, improving the diversity of the samples. Based on these samples, the anomaly detection model can be trained, which can improve the generalization ability and accuracy of the anomaly detection model.

[0086] 303. The computer device obtains multiple sets of sample pairs based on the sample tiles in the tile sets of multiple first sample images and the sample text corresponding to each sample tile. Each set of sample pairs includes a sample tile and at least one sample text. The sample text is used to describe the abnormal situation of the local image of the sample item.

[0087] In the embodiments of this application, each first sample image is divided into multiple sample patches, and correspondingly, each first sample image corresponds to multiple sets of sample pairs.

[0088] In some embodiments, the same anomaly can be described by different texts. Therefore, one sample patch can correspond to multiple sample texts. The multiple sample texts describe the anomaly of the local image of the sample item in the sample patch in different texts. Thus, one sample patch can form multiple sample pairs, in which the sample patches are the same but the sample texts are different.

[0089] In some embodiments, each sample patch corresponds to multiple sample texts, and the multiple sample texts describe the anomalies in the local images of sample items within the sample patch in different ways. The process by which the computer device extracts features from the sample texts in a sample pair using an anomaly detection model to obtain the text features of the sample texts includes the following steps: The computer device extracts features from the multiple sample texts in the sample pair using an anomaly detection model to obtain the initial text features corresponding to the multiple sample texts respectively, and determines the mean of the multiple initial text features to obtain the text features.

[0090] In this embodiment, multiple initial text features are multiple vectors of the same dimension, and the text features are also vectors with the same dimension as the initial text features. Therefore, determining the mean of the multiple initial text features means determining the mean of the multiple vectors to obtain the text features. Alternatively, multiple initial text features are multiple matrices of the same dimension, and the text features are also matrices with the same dimension as the initial text features. In this case, determining the mean of the multiple initial text features means determining the mean of the multiple matrices to obtain the text features.

[0091] In this context, multiple sample texts describe anomalies in local images of sample items within sample tiles using different textual descriptions. These multiple sample texts include, but are not limited to, at least one of the following: each sample text contains at least one word that differs from those in other sample texts; the number of words in the multiple sample texts differs; or the word order among multiple words in the multiple sample texts differs.

[0092] For example, sample text describing the absence of local anomalies can be "perfect" or "defect-free". Sample text describing the presence of local anomalies can be "damaged" or "defective".

[0093] In this embodiment of the application, since the same abnormal situation can be described by different texts, averaging the text features of multiple texts describing the same abnormal situation makes the obtained text features more accurate and effective.

[0094] In some embodiments, the computer device acquires multiple sample texts through the following steps. For each sample image patch, the computer device fills multiple text templates based on a partial image of a sample item in the sample image patch and anomaly information of that partial image, resulting in multiple sample texts. The anomaly information describes anomalies in the partial image of the sample item, and the multiple text templates are different.

[0095] A text template is a pre-defined, generic format used to generate text describing anomalies in local images of sample items. Multiple text templates may differ; for example, each template may contain at least one word that differs from others, the number of words may vary, or the word order may differ. Multiple sample texts can be obtained by populating the text templates based on the local image of the sample item and its anomaly information.

[0096] For example, the text template can be 'Cropped photo of [c]', 'Photo of [c]', where c is at least one of the anomaly information of the sample item partial image to be filled and the name of the sample item partial image; it can also be 'Perfect [o]', 'Defect-free [o]', 'Damaged [o]', 'Defective [o]', where o represents the name of the sample item partial image to be filled or the name of the sample item image.

[0097] In the embodiments of this application, the multiple text templates are preset, and the multiple text templates include, but are not limited to, at least one of the following situations: each text template includes at least one word that is different from the words in other text templates; the number of words included in the multiple text templates is different; the word order among the multiple words in the multiple text templates is different.

[0098] The text template can be filled based on a partial image of the sample item and anomaly information of the partial image, or it can be filled based on either the partial image or the anomaly information. For example, the text template could be "Cropped photo of [c]" or "Photo of [c]", where c is at least one of the anomaly information of the partial image of the sample item to be filled and the name of the partial image of the sample item.

[0099] In this embodiment, the text template is a universal text template for various abnormal situations, and multiple sample texts can be obtained based on multiple text templates. It is only necessary to fill in the text template based on the abnormal situation, which improves the convenience of obtaining sample texts.

[0100] In other embodiments, text templates are also provided to describe different anomalies, so that sample text can be obtained simply by filling in the name of the partial image of the sample item in the text template. For example, the text template can be "perfect [o]", "defect-free [o]", "damaged [o]", or "defective [o]", where 'o' represents the name of the partial image of the sample item to be filled, such as "the head of a screw". Optionally, 'o' can also represent the name of the image of the sample item to be filled, such as "screw".

[0101] In this embodiment, steps 302-303 described above achieve the process of generating multiple sets of sample pairs based on multiple first sample images. In this embodiment, the first sample images are segmented using sliding windows of multiple sizes to obtain multiple patches of the first sample images of multiple sizes. This not only improves the diversity of the samples but also captures defects in the sample items from small to large, thereby enabling the anomaly detection model to identify defects in the sample items from small to large and improving the accuracy of the anomaly detection model.

[0102] It should be noted that steps 302-303 above are only one optional implementation of the process. The computer device can also implement the process in other optional ways; for example, the first sample image can be segmented by a sliding window of a certain size to obtain multiple patches of the first sample image, and then sample pairs can be obtained based on the sample patches and sample text.

[0103] 304. For each sample pair, the computer equipment extracts features from the sample image patch and sample text in the sample pair through the anomaly detection model, obtains the image features of the sample image patch and the text features of the sample text, determines the similarity between the image features and the text features, and uses the anomaly detection model to perform anomaly detection on the local image of the target item in the input image patch.

[0104] In this embodiment of the application, feature extraction of sample image blocks and sample text can obtain more expressive representations of sample image blocks and sample text, which facilitates subsequent anomaly detection by the model.

[0105] In some embodiments, a pre-trained feature extraction network, such as a ResNet50 network, is used for initial feature extraction. This feature extraction network is pre-trained on a large-scale dataset and can effectively extract semantic features from images and text. During subsequent learning, the network's parameters remain unchanged; that is, the network is only responsible for feature extraction and its parameters are not updated during subsequent training. Optionally, the network is followed by at least one fully connected layer. The fully connected layer is used to further extract features from the network's output to obtain the corresponding image or text features. This means that only the parameters of the fully connected layer need to be adjusted during training, thereby improving training efficiency.

[0106] A pre-trained model, also known as a foundational model or large model, refers to a deep neural network (DNN) with a large number of parameters. It is trained on massive amounts of unlabeled data. Leveraging the function approximation capabilities of the large-parameter DNN, the pre-trained model (PTM) extracts common features from the data. Through fine-tuning, parameter-efficient fine-tuning (PEFT), and prompt-tuning techniques, it becomes suitable for downstream tasks. Therefore, pre-trained models can achieve ideal results in scenarios with few or zero samples. Based on the data modalities they process, PTMs can be categorized into language models (ELMO, BERT, GPT), visual models (Swin-transformer, ViT, V-MOE), speech models (VALL-E), and multimodal models (ViBERT, CLIP, Flamingo, Gato), among others. Multimodal models refer to models that establish feature representations for two or more data modalities. Pre-trained models are important tools for outputting AI-generated content (AIGC) and can also serve as a universal interface connecting multiple task-specific models.

[0107] Zero-shot transfer ability refers to the ability of a model to directly apply knowledge learned from pre-training on large-scale, multimodal datasets to new tasks without having trained on training samples specific to a particular task. In this application, large models such as CLIP, through pre-training on large-scale, multimodal datasets, exhibit good zero-shot transfer ability and can be used for tasks such as outlier detection and removal.

[0108] 305. The computer equipment iteratively trains the anomaly detection model based on the similarity of multiple sample pairs and the preset similarity.

[0109] In the embodiments of this application, the similarity of each sample pair refers to the similarity between the features of the sample tiles and the sample text in the sample pair.

[0110] During any iteration, the model parameters of the anomaly detection model can be adjusted using the similarity of at least one set of sample pairs and a preset similarity. This involves determining the loss values ​​between the similarity of each of the at least one set of sample pairs and the preset similarity, and adjusting the model parameters based on the average of these loss values. In the next iteration, the next batch of at least one set of sample pairs is input into the adjusted anomaly detection model to obtain the similarity of each of the next batch of at least one set of sample pairs. The model parameters of the anomaly detection model are then adjusted using the similarity of these at least one set of sample pairs and the preset similarity. This iterative process is repeated until the preset requirements are met.

[0111] In other embodiments, the above-mentioned sample pairs are positive sample pairs. The computer device can also construct negative sample pairs and combine the positive and negative sample pairs for training. A negative sample pair includes a sample patch and negative sample text, where the negative sample text is not used to describe anomalies in the local image of a sample item within the sample patch. Furthermore, the negative sample text can be any sample text from any positive sample pair other than the one containing the sample patch, thereby improving the efficiency of sample text acquisition.

[0112] In this embodiment, during the training of the anomaly detection model, image features and text features are extracted by the image encoder and text encoder, respectively. A contrastive learning loss function is used to bring paired image-text pairs closer together in the feature space and push unpaired image-text pairs further apart, thus achieving pre-training of image-text pairs. Therefore, during the usage phase of the anomaly detection model, different preset texts, such as "a defective item" and "a non-defective item," are used to describe the anomalies of the sample item images. The anomaly detection model then extracts the text features of each different text and the image features of the input image. The similarity between the extracted image features and the preset text features is compared to determine which text the image is closer to, thereby achieving anomaly detection.

[0113] Contrastive learning is a machine learning method. In the training process of the anomaly detection model in this application, the loss function of contrastive learning is used to bring paired image-text pairs closer in the feature space and push unpaired image-text pairs further apart in the feature space, thereby achieving pre-training of image-text pairs. This enables the model to learn the pattern that there is a high similarity between paired image features and text features.

[0114] The loss function for contrastive learning can take the following form:

[0115] Where sim(I,T) represents the similarity between image feature I and text feature T, τ is the temperature parameter, and N is the number of samples. The temperature parameter is a parameter in the loss function of contrastive learning, used to adjust the weight of similarity in the loss function, affecting the degree to which the model learns from different samples during training.

[0116] This application provides a training method for an anomaly detection model. This method trains the anomaly detection model based on sample image tiles and sample text describing anomalies in local images of sample items within the sample image tiles. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between image and text features and a preset similarity. This allows the anomaly detection model to learn the rule of high similarity between pairs of image and text features. Therefore, the trained anomaly detection model can accurately extract features. For any image tile containing a local image of a target item, the extracted image features can determine text features with high similarity. The text corresponding to this text feature is then the text describing the anomaly in the local image of the target item within the image tile. Therefore, the anomaly detection model trained using this method has high accuracy in detecting anomalies in local images of target items within image tiles. Furthermore, since this method is trained based on image tiles, and image tiles include local images of sample items, the anomaly detection model can specifically detect which local image of the target item contains anomalies, thus improving the accuracy of anomaly detection. Therefore, using an anomaly detection model trained based on this method can improve the accuracy and precision of anomaly detection.

[0117] An anomaly detection model was trained using the embodiments shown in Figures 2 and 3 above. Anomaly detection is then performed based on the anomaly detection model trained in Figure 2 or 3. Referring to Figure 5, which illustrates an anomaly detection method for items according to an embodiment of this application, the method includes the following steps.

[0118] 501. A computer device acquires multiple tiles of an image, the image containing an image of a target object, and each tile containing a partial image of the target object.

[0119] In this embodiment, the target item can be any item to be detected as abnormal, such as various parts produced in industry. Multiple blocks are obtained based on image segmentation.

[0120] 502. For each tile, the computer equipment uses an anomaly detection model to determine the target text corresponding to the tile. The target text is used to describe the anomalies in the local image of the target item in the tile.

[0121] In this embodiment, the anomaly detection model is used to perform anomaly detection on a local image of a target object in an input image tile. Specifically, a computer device inputs the image tile into the anomaly detection model, extracts image features from the tile, determines text features whose similarity to these image features meets preset requirements, and then uses the text corresponding to these text features as the target text.

[0122] 503. The computer device determines the abnormal information of the target item image in the image based on the target text corresponding to multiple blocks respectively. The abnormal information is used to describe the abnormal situation of the target item image.

[0123] In some embodiments, the anomaly information of the target item image includes target text corresponding to multiple image tiles, so as to obtain the anomaly status of each local image of the target item; or, the anomaly information is used to describe whether there is anomaly in the target item as a whole. For example, the anomaly information can be "the target item has anomalies" or "the target item does not have anomalies".

[0124] This application provides an anomaly detection method for objects. This method is based on an anomaly detection model, which is trained on sample image patches and sample text describing the anomalies in local images of sample objects within those patches. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between these features and a preset similarity. This allows the model to learn the pattern of high similarity between pairs of image and text features, enabling accurate feature extraction. Thus, for any image patch containing a local image of a target object, the extracted image features can identify text features with high similarity. The text corresponding to these text features is then used to describe the anomalies in the local image of the target object within the patch. Therefore, the anomaly detection model trained using this method has high accuracy in detecting anomalies in local images of target objects within patches. Furthermore, based on the anomalies corresponding to each image patch, the overall anomaly situation of the target object image can be obtained, improving the convenience and accuracy of anomaly detection.

[0125] Figure 5 illustrates the basic process of anomaly detection based on an anomaly detection model. The anomaly detection method will be further described below based on the embodiment shown in Figure 6. Referring to Figure 6, which is a flowchart of an anomaly detection method provided in this application, the method includes the following steps.

[0126] 601. A computer device acquires multiple image tiles, the image containing an image of a target object, and each tile containing a partial image of the target object.

[0127] In some embodiments, the computer device acquires an initial image, which includes an overall image of the target item; the target item is segmented from the initial image to obtain the image. Alternatively, the computer device calibrates the position of the target item image in the initial image based on the position of the target item image in a template image to obtain the image. Alternatively, the computer device segments the target item from the initial image to obtain a first image, and then calibrates the position of the target item image in the first image based on the position of the target item image in the template image to obtain the image. Alternatively, the computer device calibrates the position of the target item image in the initial image based on the position of the target item image in the template image to obtain a second image, and then segments the target item from the second image to obtain the image.

[0128] The specific process of image segmentation and calibration by computer equipment is the same as the process of image segmentation and calibration in step 301, and will not be repeated here.

[0129] In some embodiments, the process of a computer device acquiring multiple tiles of an image includes the following steps: the computer device segments the image using sliding windows of multiple sizes to obtain multiple tile sets of different sizes, wherein each tile set of a size includes multiple tiles of that size.

[0130] The process of segmenting the image by the computer equipment is the same as the process of segmenting the first sample image in step 302, and will not be described again here.

[0131] 602. For each map tile, the computer device extracts the image features of the map tile through an anomaly detection model, determines the similarity between the image features and multiple preset text features, determines the target text feature whose similarity meets the preset requirements from the multiple preset text features, obtains the target text based on the target text feature, and the multiple preset text features correspond to preset texts. The target text is the preset text corresponding to the target text feature. The target text is used to describe the anomaly of the local image of the target item in the map tile.

[0132] In this embodiment, multiple preset text features correspond to preset texts. The computer device inputs these preset texts into an anomaly detection model, which then extracts features from each text to obtain its own preset text features. In this embodiment, multiple preset text features are pre-determined through the anomaly detection model, allowing for the reuse of these features to detect a large number of image blocks, thus improving efficiency and saving resources.

[0133] In this embodiment of the application, the similarity between the target text features and the image features meets the preset requirements, which means that the similarity between the target text features and the image features reaches the target similarity, or the similarity between the target text features and the image features is the largest among multiple preset text features.

[0134] In some embodiments, there are two preset texts: normal text and abnormal text, corresponding to two preset text features: normal text features and abnormal text features. Normal text describes the absence of anomalies in a localized area of ​​the image patch, while abnormal text describes the presence of anomalies in a localized area of ​​the image patch. If the similarity between the image features and the normal text features is higher, then the area of ​​the image patch is free of anomalies, and the patch is considered a normal patch, i.e., the target patch. If the similarity between the image features and the abnormal text features is higher, then the area of ​​the image patch contains anomalies, and the patch is considered an abnormal patch.

[0135] In this embodiment, step 602 described above achieves the process of determining the target text corresponding to each image tile using an anomaly detection model. In this embodiment, since the trained anomaly detection model can accurately extract image features of the image tile and accurately extract text features, and the text features used to describe the anomalies in the local image of the target item in the image tile have high similarity to the image features, multiple text features are preset, and the similarity between these multiple text features and the image features is compared. The text corresponding to the text feature whose similarity meets the preset requirements is the text used to describe the anomalies in the local image of the target item in the image tile. That is, through this anomaly detection model, rapid and accurate detection of anomalies in the local image of the target item in the image tile is achieved.

[0136] It should be noted that step 602 is only one optional way to implement the process. The computer device can also implement the process in other optional ways, which will not be elaborated here.

[0137] 603. The computer device determines the abnormal information of the target object image in the image based on the target text corresponding to multiple blocks respectively. The abnormal information is used to describe the abnormal situation of the target object image.

[0138] In some embodiments, the process by which the computer device determines the abnormal information of a target item image in an image based on the target text corresponding to multiple tiles includes the following steps: when the target text corresponding to at least one tile in the multiple tiles indicates that a local image of the target item is abnormal, the computer device determines that the target item in the image is abnormal and determines the local area where the target item is abnormal; when the target text corresponding to each tile in the multiple tiles indicates that a local image of the target item is not abnormal, the computer device determines that the target item image in the image is not abnormal.

[0139] If at least one tile corresponds to a text indicating an anomaly in a local image of the target item, it means that at least one local image of the target item is abnormal, thus indicating that the target item is abnormal. Based on this, since multiple tiles each contain local images of the target item, the location of the anomaly in the target item is determined based on this at least one tile.

[0140] In some embodiments, the target text further indicates the type of anomaly. Based on the target text, the type of anomaly in the local image of the target item can also be determined. Accordingly, the anomaly information includes the local image of the target item with anomalies and the type of anomaly in that local image. In some embodiments, based on the proportion of at least one tile in multiple tiles, the area of ​​the target item with anomalies can also be determined. Accordingly, the anomaly information includes the area of ​​the target item image with anomalies.

[0141] In other embodiments, the anomaly information includes target text corresponding to multiple blocks respectively. Then, by using the target text of each of these multiple blocks in the anomaly information, it can be known which blocks have anomalies in the local images of the target items or which blocks do not have anomalies in the local images of the target items.

[0142] In this embodiment, the overall anomaly of the target item image is determined based on the target text of each of the multiple image tiles. This enables the summarization of anomaly information from multiple image tiles, improves the readability of anomaly detection results, and thus enhances the user experience.

[0143] In some embodiments, the computer device segments the image using sliding windows of multiple sizes to obtain multiple tiles of multiple sizes, each tile containing multiple pixels. The process by which the computer device determines the anomalous information of a target object image in an image based on the target text corresponding to each of the multiple tiles includes the following steps: For each tile of each size, if the target text corresponding to the tile indicates that a local image of the target object in the tile is abnormal, the computer device assigns a similarity value between the image features of the tile and the corresponding target text features to multiple pixels in the tile; for each pixel, based on the similarity of the pixel across multiple sizes, an anomalous value is obtained, which indicates the probability that the pixel is abnormal; based on the anomalous value of each pixel in the image, the anomalous information of the target object image in the image is determined, whereby the anomalous information of the target object image includes at least one of the pixel location and the anomalous area of ​​the abnormal pixel in the target object image.

[0144] Optionally, the process by which the computer device obtains outlier values ​​of a pixel based on the similarity of the pixel across multiple dimensions includes the following steps: The computer device determines the mean of the similarity of the pixel across multiple dimensions to obtain the outlier value of the pixel. This mean can be a harmonic mean or an arithmetic mean. Alternatively, the computer device determines the sum of the similarity of the pixel across multiple dimensions to obtain the outlier value of the pixel. The harmonic mean is a statistical average value; for a set of data x1, x2, ..., x... n The formula for calculating its harmonic mean value H is: In this application, the mean value of the similarity of pixels at multiple sizes can be used to obtain outlier values ​​of pixels.

[0145] Optionally, the process by which the computer device determines the abnormal information of a target object image in an image based on the abnormal value of each pixel in the image includes the following steps: the computer device determines multiple target pixels in the image whose abnormal values ​​are greater than an abnormal threshold, determines the pixels located on the target object among the multiple target pixels, and obtains the positions of the abnormal pixels on the target object. Further, after obtaining the abnormal pixels on the target object, the computer device determines the ratio between the number of these pixels and the number of multiple pixels on the target object, and obtains the abnormal area of ​​the target object image.

[0146] In other embodiments, the computer device determines the average similarity between image features of a patch and multiple preset text features, and assigns this average value to multiple pixels in the patch. Alternatively, the computer device determines a first text feature from among the multiple preset text features that describes a local anomaly, determines the average similarity between the image features of the patch and the multiple first text features, and assigns this average value to multiple pixels in the patch. Alternatively, the computer device may further determine multiple second text features from among the multiple preset text features whose similarity to image features is greater than a similarity threshold, determines the average similarity between the image features of the patch and the multiple second text features, and assigns this average value to multiple pixels in the patch.

[0147] In this embodiment, for each pixel, the similarity of the pixel at multiple sizes is fused to make the outlier value of each pixel more accurate. That is, the probability that the pixel indicated by the outlier value is abnormal is more accurate. Then, the outlier values ​​of multiple pixels are used to determine the abnormal information of the target object image in the image, making the abnormal information more accurate.

[0148] In some embodiments, the anomaly detection model provided in this application is used to screen abnormal samples, and the remaining normal samples are used as training data to construct an anomaly detection network, which is used to detect anomalies in items.

[0149] An anomaly detection network is a network used to detect anomalies in objects. This network constructs non-anomaly features from normal samples to create images of sample objects or parts of sample objects. By comparing the features of the image or patch to be detected with these non-anomaly features, it detects objects or parts that differ significantly from the non-anomaly features, thus identifying them as anomalies.

[0150] In this embodiment, the target item is the sample item. An abnormal sample refers to a sample image containing an anomaly, while a normal sample refers to a sample image containing no anomalies. An anomaly detection model detects abnormal samples from a large number of sample images, and an anomaly detection network is constructed based on the remaining normal samples. Specifically, non-abnormal features of the sample item images are constructed based on the normal samples; these non-abnormal features are the feature distribution of the sample item images in the normal samples. The anomaly detection network then detects abnormal items based on these non-abnormal features, that is, it detects items that significantly deviate from these non-abnormal features. These items are considered abnormal. The construction and use of the anomaly detection network are described in the following steps, which are optional implementation steps.

[0151] 604. The computer device determines multiple target images from multiple images based on the abnormal information of the target items in each of the multiple images, and the target item images in the target images do not have any abnormalities.

[0152] In this embodiment of the application, the anomaly information is used to describe the anomaly of the target item image. Therefore, based on the anomaly information, it can be determined whether there is an anomaly in the target item in the image, and the image in which the anomaly information indicates that there is no anomaly in the target item image is taken as the target image.

[0153] 605. Computer equipment determines the non-abnormal features of a target object image based on multiple target images. Non-abnormal features are the characteristics of a target object that does not have any abnormalities.

[0154] In this process, the computer equipment extracts features from multiple image features to obtain the image features of each of the multiple target images. Based on the image features of each of the multiple target images, the non-abnormal features of the target object image are obtained.

[0155] Alternatively, the computer device may perform feature extraction using an anomaly detection model, or it may perform feature extraction using other feature extraction networks, such as the ResNet50 network.

[0156] The non-abnormal features include the individual image features of each of the multiple target images. Alternatively, the computer device clusters the image features of the multiple target images to obtain multiple clusters, and uses the average of the multiple image features in each cluster as the image feature corresponding to that cluster; the non-abnormal features include the image features corresponding to each of the multiple clusters. Alternatively, the computer device determines the average of the image features of the multiple target images to obtain the non-abnormal features.

[0157] Therefore, the non-abnormal feature can represent the feature distribution of a target object image without anomalies. Optionally, the non-abnormal feature is stored in a memory that directly and discretly represents the feature distribution of a target object image without anomalies.

[0158] 606. Computer equipment performs anomaly detection on images containing images of target objects based on non-abnormal features.

[0159] In some embodiments, the computer device determines that the target item in the image is abnormal if the distance between the image features of the image containing the target item image and the non-abnormal features is greater than a distance threshold; the computer device determines that the target item image in the image is not abnormal if the distance between the image features of the image containing the target item image and the non-abnormal features is not greater than the distance threshold. This distance can be a cosine distance.

[0160] In some embodiments, non-abnormal features include image features of each of multiple target images or image features corresponding to each of multiple clusters. Then, from the multiple image features included in the non-abnormal features, the target image feature that is closest to the image feature of the image is determined. If the distance is not greater than a distance threshold, it is determined that the target item image in the image does not have an anomaly; if the distance is greater than the distance threshold, it is determined that the target item in the image has an anomaly.

[0161] The distance threshold can be set and changed as needed. This threshold can be determined based on the yield requirements of the items in actual production. Optionally, a small amount of verification data (images of target items without anomalies) can be collected and input into the anomaly detection network to calculate the anomaly score (i.e., distance) for each verification data point. The anomaly scores are then sorted in ascending order. Assuming the actual yield rate is m%, the anomaly score at the m% position in the sorted anomaly scores is used as the distance threshold. In anomaly detection, the distance between the image features of the image to be detected and the non-anomaly features (such as cosine distance) can be used as the anomaly score. By collecting a small amount of verification data, inputting it into the anomaly detection network, calculating the anomaly score for each verification data point, and then sorting the anomaly scores in ascending order, the distance threshold can be determined based on the actual yield requirements.

[0162] In this embodiment, an anomaly detection model is used to determine the anomaly information of the image. This model can accurately detect images where the target item is abnormal. Therefore, it can detect images of the target item with anomalies from multiple images, retaining normal images of the target item without anomalies, thus improving the accuracy of anomaly detection. Furthermore, the accuracy of non-abnormal features of normal items established based on these normal images is high. Correspondingly, anomaly detection based on non-abnormal features can detect images that do not match the non-abnormal features, and thus the target items included in these images are the items with anomalies, further improving the accuracy of anomaly detection.

[0163] The above embodiments illustrate an anomaly detection network using an image encompassing the entire target item as the detection object. In other embodiments, the anomaly detection network uses patches containing partial images of the target item as the detection object. Anomaly samples refer to patch samples containing partial anomalies, while normal samples refer to patch samples containing partial anomalies. Specifically, anomaly samples are filtered from a large number of patches using an anomaly detection model, and the anomaly detection network is constructed based on the remaining normal samples. Specifically, non-anomaly features of the sample item image are constructed based on the normal samples. These non-anomaly features are the feature distribution of partial images of the sample item within the normal samples. The anomaly detection network then uses these non-anomaly features to detect abnormal partial images of the target item image, that is, it detects areas that differ significantly from the non-anomaly features; these areas are the areas containing anomalies. Detecting abnormal areas means detecting the abnormal target item.

[0164] The computer device identifies multiple target blocks within multiple blocks based on the target text corresponding to each block, and the local images of the target items in the target blocks do not contain any anomalies; based on the multiple target blocks of each of the multiple images, it determines the non-anomaly features of the local images of the target items, which are the features of the local images of the target items where no anomalies exist; and it performs anomaly detection on the blocks containing the local images of the target items based on the non-anomaly features.

[0165] The target text is used to describe the abnormality of a local image of the target item. Therefore, based on the target text, it is possible to determine whether a local part of the tile is abnormal, and the tile in which the target text indicates that there is no local abnormality is taken as the target tile.

[0166] Optionally, the computer device performs feature extraction on multiple target image tiles separately to obtain the image features of each target image tile. Based on the image features of each target image tile, non-abnormal features of the local image of the target object are obtained. Wherein, if the computer device performs feature extraction using an anomaly detection model, since the image features of the target image tiles have already been extracted when determining the target text, the image features extracted in step 602 can be directly obtained. Alternatively, the computer device can perform feature extraction using other feature extraction networks, such as using a ResNet50 network.

[0167] The non-abnormal features of the local image of the target object include the image features of each of the multiple target tiles. Alternatively, the computer device clusters the image features of the multiple target tiles to obtain multiple clusters, and uses the mean of the multiple image features in each cluster as the image feature corresponding to that cluster; the non-abnormal features include the image features corresponding to each of the multiple clusters. Alternatively, the computer device determines the mean of the tile features of the multiple target tiles to obtain the non-abnormal features.

[0168] In some embodiments, the computer device performs unified processing on multiple tiles comprising different local images of a target object to obtain non-abnormal features of the local images of the target object. In other embodiments, for each local image of the target object, the computer device determines the non-abnormal features of that local image based on multiple target tiles including that local image, thereby obtaining the non-abnormal features of each local area of ​​the target object image. Accordingly, the computer device performs anomaly detection on the tiles comprising that local image based on the non-abnormal features of each local image of the target object.

[0169] The process of computer equipment performing anomaly detection on a patch containing a local image of a target object based on the non-abnormal features of the target object's local image is the same as the process of performing anomaly detection on an image containing a target object's image based on the non-abnormal features of the target object's image, and will not be repeated here.

[0170] In this embodiment, an anomaly detection model is used to determine the anomaly information of the image. This model can accurately detect abnormal patches in the local image of the target object. Furthermore, the model can detect patches with local anomalies among multiple patches, retaining normal patches without local anomalies. This achieves accurate detection of abnormal patches, and the accuracy of the non-abnormal features of the normal local image of the target object built based on these normal patches is high. Correspondingly, anomaly detection based on non-abnormal features can accurately detect patches that do not match the non-abnormal features. These patches include the local areas with anomalies, thus achieving accurate anomaly detection.

[0171] When constructing non-abnormal features for target object images, anomaly detection networks typically use normal samples as training data. Normal samples refer to images containing target object images without any anomalies. Therefore, after calibration, the images must first undergo denoising, i.e., abnormal samples must be detected and removed before the remaining samples can be used as training data to train the anomaly detection network. However, in this embodiment, the method provided can accurately detect abnormal samples, ensuring that the training data used by the anomaly detection network consists entirely of normal samples. This improves the accuracy of the trained anomaly detection network, resulting in high accuracy in anomaly detection based on this network.

[0172] In this application, denoising refers to the process of detecting and removing abnormal samples from a large number of sample images or patches using a specific model (such as an anomaly detection model or a large model) to ensure that the samples used to train the anomaly detection network are normal samples, thereby improving the accuracy of the trained anomaly detection network.

[0173] In this embodiment, an anomaly detection model obtained through the above training method is used as an example for illustration. In other embodiments, the computer device can also directly use a basic model that has been trained on a large-scale dataset, such as the CLIP (Contrastive Language-Image Pre-Training) model, and models improved based on CLIP, such as the APRIL-GAN model and the Anomaly CLIP model, etc., without specific limitations. These large models, through pre-training on large-scale, multimodal datasets, can have good zero-shot transfer capabilities. Therefore, using these models for denoising can avoid a series of problems caused by clustering methods. Denoising refers to detecting and removing abnormal samples. Therefore, the method provided in this embodiment realizes a fully automatic anomaly detection framework based on a basic large model. It utilizes the general capabilities of the large model, does not rely on specific assumptions and pre-training on a large number of normal items, realizes a more reliable denoising process, effectively improves the overall denoising performance of the framework, and thus achieves a more efficient, stable, and universal fully automated industrial anomaly detection process.

[0174] A normal item is an item whose characteristics conform to the distribution characteristics of normal items in actual production, and does not have any abnormalities such as rough surfaces, scratches, or insufficient dimensions. Conversely, an abnormal item is an item whose characteristics differ from the distribution characteristics of normal items in actual production. In industrial production scenarios, items with rough surfaces, scratches, or dimensions that do not meet requirements are all considered abnormal items.

[0175] The method provided in this application does not require updating parameters when constructing an anomaly detection network, resulting in extremely low training time overhead and thus improving efficiency. In other embodiments, the computer device can also construct the anomaly detection network using PatchCore (an image processing algorithm based on convolutional neural networks), UniAD (an anomaly detection algorithm), SimpleNet (an anomaly detection algorithm), etc., which will not be elaborated here.

[0176] The method provided in this application can be applied to automated industrial quality inspection, accurately detect abnormal defects in industrial parts, greatly reduce the labor cost of industrial quality inspection, and is more efficient, stable and versatile due to noise reduction based on the abnormality detection module.

[0177] For example, referring to Figure 7, which is a schematic flowchart of an industrial anomaly detection method provided in an embodiment of this application, the method provided in this embodiment is applied to a fully automated industrial anomaly detection process, which includes a training phase and a testing phase. In the training phase, the anomaly detection model is trained based on parts produced in the factory. First, sample images including parts are aligned with template images; alignment refers to calibrating the position of the sample item image in the sample image based on the position of the sample item image in the template image. Then, the model is trained based on these sample images to obtain the anomaly detection model. The anomaly detection model is then denoised, and an anomaly detection network is constructed based on the remaining normal samples. The anomaly detection network includes non-abnormal features of the parts. In the testing phase and during actual anomaly detection, parts produced in the factory are detected through the anomaly detection network to identify abnormal parts that differ significantly from non-abnormal features.

[0178] For example, referring to Figure 8, which is a flowchart of the training phase in industrial inspection according to an embodiment of this application, a computer device acquires an automatically collected input image, performs image alignment, and then performs denoising based on an anomaly detection model, i.e., removing abnormal samples. An anomaly detection network is then constructed based on the remaining normal samples. As another example, referring to Figure 9, which is a flowchart of the testing phase in industrial inspection according to an embodiment of this application, a computer device acquires an input test image, performs image alignment, performs anomaly detection through an anomaly detection network, and outputs the anomaly detection results.

[0179] It should be noted that the above embodiments illustrate actual anomaly detection using an anomaly detection network as an example. The computer device can also directly perform actual anomaly detection based on the anomaly detection model. Alternatively, actual anomaly detection can be performed separately using both the anomaly detection network and the anomaly detection model, and the target detection result can be obtained by combining the detection results of both. For example, an item detected by either method can be considered an item with anomalies; or, an item detected by both methods can be considered an item with anomalies.

[0180] This application provides an anomaly detection method for items. This method is based on an anomaly detection model, which is trained on sample image patches and sample text describing the anomalies in local images of sample items within the sample image patches. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between image and text features and a preset similarity. This allows the anomaly detection model to learn the pattern of high similarity between pairs of image and text features, thus enabling accurate feature extraction. Because the trained anomaly detection model can accurately extract image features from image patches... This model identifies text features that can accurately extract text. The text features used to describe anomalies in a local image of a target object within a tile have high similarity to the image features. Multiple text features are then preset, and their similarity to the image features is compared. The text corresponding to the text feature whose similarity meets the preset requirements is the text used to describe anomalies in a local image of a target object within a tile. This anomaly detection model enables fast and accurate detection of anomalies in local images of target objects within a tile. Furthermore, based on the anomalies corresponding to each tile in the image, the overall anomaly situation of the target object image can be obtained, improving the convenience and accuracy of anomaly detection.

[0181] Figure 10 is a block diagram of a training apparatus for an anomaly detection model according to an embodiment of this application. Referring to Figure 10, the apparatus includes:

[0182] The acquisition module 1001 is used to acquire multiple first sample images and generate multiple sets of sample pairs based on the multiple first sample images. Each first sample image contains a sample item image. Each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a partial image of the sample item in the sample item image. The sample text is used to describe the abnormal situation of the partial image of the sample item in the sample patch.

[0183] The extraction module 1002 is used to extract features from the sample image blocks and sample text in each sample pair through an anomaly detection model, thereby obtaining the image features of the sample image blocks and the text features of the sample text, and determining the similarity between the image features and the text features. The anomaly detection model is used to perform anomaly detection on the local image of the target item in the input image block.

[0184] Training module 1003 is used to iteratively train the anomaly detection model based on the similarity of multiple sample pairs and a preset similarity.

[0185] In some embodiments, the acquisition module 1001 is used for:

[0186] For each first sample image, the first sample image is segmented using sliding windows of multiple sizes to obtain multiple tile sets of different sizes. A tile set of one size contains multiple sample tiles of one size.

[0187] Based on the sample tiles in the tile sets of each of the multiple first sample images and the sample text corresponding to each sample tile, multiple sets of sample pairs are obtained.

[0188] In some embodiments, each sample patch corresponds to multiple sample texts, and the multiple sample texts describe the anomalies in the local image of the target object in the sample patch using different texts; the extraction module 1002 is used for:

[0189] By using an anomaly detection model, features are extracted from multiple sample texts in a sample pair to obtain initial text features corresponding to each sample text. The mean of the multiple initial text features is then determined to obtain the text features.

[0190] In some embodiments, the apparatus further includes:

[0191] The filling module is used to fill multiple text templates for each sample image patch based on the local image of the target item in the sample image patch and the anomaly information of the local image of the target item, resulting in multiple sample texts. The anomaly information is used to describe the anomaly of the local image of the target item, and the multiple text templates are different.

[0192] In some embodiments, the acquisition module 1001 is further configured to acquire a plurality of second sample images, each second sample image containing a sample item image;

[0193] The device also includes a segmentation module for segmenting sample item images from each second sample image to obtain a first sample image.

[0194] In some embodiments, the acquisition module 1001 is further configured to acquire a plurality of third sample images, each third sample image containing a sample item image;

[0195] The device also includes a calibration module for calibrating the position of the sample item image in the third sample image based on the position of the sample item image in the template image for each third sample image, so as to obtain a first sample image in which the position of the sample item image matches the position of the sample item image in the template image.

[0196] This application provides a training device for an anomaly detection model. This device trains the anomaly detection model based on sample image tiles and sample text describing anomalies in local images of sample items within the sample image tiles. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between image and text features and a preset similarity. This allows the anomaly detection model to learn the rule of high similarity between pairs of image and text features. Therefore, the trained anomaly detection model can accurately extract features. For any image tile containing a local image of a target item, based on its extracted image features, a text feature with high similarity can be determined. The text corresponding to this text feature is then the text describing the anomaly in the local image of the target item within the image tile. Therefore, the anomaly detection model trained using this device has high accuracy in detecting anomalies in local images of target items within image tiles. Furthermore, since the device is trained based on image tiles, and image tiles contain local images of target items, the anomaly detection model can specifically detect which local image of the target item contains anomalies, thus improving the accuracy of anomaly detection. Therefore, using an anomaly detection model obtained through training can improve the accuracy and precision of anomaly detection.

[0197] Figure 11 is a block diagram of an anomaly detection device for an article according to an embodiment of this application. Referring to Figure 11, the device includes:

[0198] The acquisition module 1101 is used to acquire multiple tiles of an image, the image containing an image of a target object, and each tile containing a partial image of the target object.

[0199] The determination module 1102 is used to determine the target text corresponding to each tile through an anomaly detection model. The anomaly detection model is obtained through the above training method, and the target text is used to describe the anomaly of the local image of the target item in the tile.

[0200] The determination module 1102 is also used to determine the abnormal information of the target item image in the image based on the target text corresponding to multiple blocks respectively. The abnormal information is used to describe the abnormal situation of the target item image.

[0201] In some embodiments, the determining module 1102 is configured to:

[0202] In the case where the target text corresponding to at least one of the multiple tiles indicates that there is an anomaly in the local image of the target item, determine that there is an anomaly in the target item in the image and determine the local area where the target item has an anomaly;

[0203] If the target text corresponding to each of the multiple blocks indicates that there are no anomalies in the local image of the target item, then it is determined that there are no anomalies in the image of the target item.

[0204] In some embodiments, the determining module 1102 is configured to:

[0205] For each image patch, the image features of the image patch are extracted using an anomaly detection model. The similarity between the image features and multiple preset text features is determined. From the multiple preset text features, the target text feature that meets the preset similarity requirements is determined. Each of the multiple preset text features corresponds to a preset text. The target text is the preset text corresponding to the target text feature.

[0206] In some embodiments, the acquisition module 1101 is used for:

[0207] The image is segmented using sliding windows of multiple sizes to obtain tile sets of multiple sizes, with each size's tile set consisting of multiple tiles.

[0208] In some embodiments, each tile contains multiple pixels, and the determining module 1102 is used to:

[0209] For each tile of each size, if there is an anomaly in the local image of the target item in the target text indicating the tile, the similarity between the image features of the tile and the corresponding target text features is assigned to multiple pixels in the tile.

[0210] For each pixel, outlier values ​​are obtained based on the similarity of the pixel across multiple dimensions. These outlier values ​​indicate the probability that a pixel is abnormal.

[0211] Based on the outlier values ​​of each pixel in the image, the abnormal information of the target object image is determined. The abnormal information of the target object image includes at least one of the abnormal pixel locations and abnormal areas.

[0212] In some embodiments, there are multiple images. The determining module 1102 is further configured to determine multiple target images in the multiple images based on the abnormal information of the target items in the multiple images, wherein the target item images in the target images do not have abnormalities; and to determine non-abnormal features of the target item images based on the multiple target images, wherein the non-abnormal features are the features of the target items that do not have abnormalities.

[0213] The device also includes a first detection module for anomaly detection of images containing images of target objects based on non-abnormal features.

[0214] In some embodiments, there are multiple images. The determining module 1102 is further configured to determine multiple target blocks among the multiple blocks based on the target text corresponding to the multiple blocks respectively, wherein the local image of the target item in the target block does not have any anomalies; and to determine non-abnormal features of the local image of the target item based on the multiple target blocks of each of the multiple images, wherein the non-abnormal features are the features of the local image of the target item where the target item image does not have any anomalies.

[0215] The device also includes a second detection module for anomaly detection of patches containing local images of the target object based on non-abnormal features.

[0216] This application provides an anomaly detection device. This device performs detection based on an anomaly detection model, which is trained on sample image patches and sample text describing anomalies in local images of sample items within those patches. Since the anomaly detection model extracts image and text features, and pairs of image and text features should have high similarity, the model is trained based on the similarity between these features and a preset similarity. This allows the model to learn the rule that pairs of image and text features have high similarity, enabling accurate feature extraction. Thus, for any image patch containing a local image of a target item, the extracted image features can identify text features with high similarity. The text corresponding to these text features is the text describing the anomalies in the local image of the target item within the patch. Therefore, the anomaly detection model trained by this device has high accuracy in detecting anomalies in local images of target items within patches. Furthermore, based on the anomalies corresponding to each image patch, the anomalies of the target item image can be determined, improving the convenience and accuracy of anomaly detection.

[0217] In the embodiments of this application, the computer device can be a terminal or a server. When the computer device is a terminal, the terminal acts as the execution subject to implement the technical solution provided in the embodiments of this application; when the computer device is a server, the server acts as the execution subject to implement the technical solution provided in the embodiments of this application; or, the technical solution provided in this application can be implemented through the interaction between the terminal and the server. The embodiments of this application do not limit this.

[0218] In summary, the training method, apparatus, computer equipment, computer-readable storage medium, and computer program product of the above-mentioned anomaly detection model involve the computer equipment acquiring multiple first sample images, which include images of sample items. Based on these first sample images, multiple sets of sample pairs are generated. Each sample pair contains a sample patch and at least one sample text. The sample patch contains a local image of the sample item, and the sample text describes the anomaly in that local image. Then, for each sample pair, the anomaly detection model extracts the image features of the sample patch and the text features of the sample text, and determines their similarity. Finally, based on the similarity of multiple sample pairs and a preset similarity, the anomaly detection model is iteratively trained. This method utilizes the feature similarity between sample patches and sample text to train the model, enabling the model to learn the similarity patterns between them and improving the accuracy and precision of anomaly detection in local images of target items within the patch.

[0219] When generating sample pairs, for each first sample image, it is segmented using sliding windows of multiple sizes to obtain tile sets of different sizes. Each tile set contains multiple sample tiles of the corresponding size. Multiple sample pairs are then generated based on these sample tiles and their corresponding sample text. This improves sample diversity, captures defects of different sizes in sample items, and enhances the generalization ability and accuracy of the anomaly detection model.

[0220] When each sample image patch corresponds to multiple sample texts, and these sample texts describe local image anomalies of the sample items in different ways, the anomaly detection model extracts initial text features from multiple sample texts, and the average value is taken to obtain the final text features. This makes the text features more accurate and effective, further optimizing model training.

[0221] For each sample image patch, multiple different text templates are filled based on the local image of the sample item and its anomaly information to obtain multiple sample texts. This method utilizes a general text template and can obtain sample text simply by filling in the anomaly information, thus improving the convenience of obtaining sample text.

[0222] Multiple second sample images are acquired, each containing an image of the sample item and a background. The sample item image is segmented from each second sample image to obtain the first sample image, avoiding background interference and facilitating subsequent image processing.

[0223] Multiple third sample images are acquired. Based on the position of the sample item image in the template image, the position of the sample item image in the third sample image is calibrated to obtain the first sample image. The position of the sample item image in the first sample image is matched with the template image, which facilitates the accurate extraction of the features of the area where the sample item image is located.

[0224] The aforementioned method, apparatus, computer equipment, computer-readable storage medium, and computer program product for anomaly detection involve the computer equipment acquiring multiple image tiles containing an image of a target item, each tile containing a partial image of the target item. Using an anomaly detection model trained using the aforementioned method, target text corresponding to each tile is determined, describing the anomaly in the partial image of the target item within the tile. Finally, based on the target text corresponding to multiple tiles, the anomaly information of the target item image is determined. This method, based on a trained anomaly detection model, determines the anomaly of the target item image through the text corresponding to the tiles, improving the convenience and accuracy of anomaly detection.

[0225] If the target text corresponding to at least one of the multiple image blocks indicates that there is an anomaly in a local part of the target item's image, then the anomaly in the target item's image and the anomaly location are determined. If the target text corresponding to each image block indicates that there is no anomaly in a local part of the target item's image, then the target item's image is determined to be free of anomalies. This method can accurately determine whether the target item as a whole is abnormal and the location of the anomaly.

[0226] For each image patch, an anomaly detection model is used to extract image features. The similarity of these features with multiple preset text features is then determined, and the target text features that meet the preset requirements are identified, thus yielding the target text. This method utilizes the comparison between preset text features and image features to quickly and accurately determine the target text corresponding to an image patch.

[0227] The image containing the target object is segmented using sliding windows of multiple sizes, resulting in tile sets of different sizes. Each tile set contains multiple tiles of the corresponding size. This method allows the obtained tiles to contain different local information about the target object, facilitating more comprehensive anomaly detection.

[0228] When a tile contains multiple pixels, if the target text corresponding to the tile indicates an anomaly in a local image of the target item, the similarity between the tile image features and the target text features is assigned to the tile pixels. Anomaly values ​​are obtained based on the similarity of pixels across multiple dimensions, thereby determining the anomaly information of the target item image, including the location and area of ​​abnormal pixels, making the anomaly information more accurate.

[0229] For multiple images, anomaly information is used to identify target images where no anomalies exist. Then, non-anomaly features of the target object images are determined based on these target images. Finally, anomaly detection is performed on images containing target object images based on these non-anomaly features. By filtering anomalous samples, a more accurate anomaly detection network is constructed, improving anomaly detection accuracy.

[0230] Based on the target text corresponding to multiple image tiles, target tiles without anomalies in the local image of the target item are identified. Then, based on the target tiles of multiple images, non-anomaly features of the local image of the target item are determined. Finally, anomaly detection is performed on the tiles containing the local image of the target item based on the non-anomaly features. The anomaly detection network is further optimized at the tile level to improve detection accuracy.

[0231] Figure 12 shows a structural block diagram of a terminal 1200 provided in an exemplary embodiment of this application.

[0232] Typically, terminal 1200 includes a processor 1201 and a memory 1202.

[0233] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0234] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one program code, which is executed by the processor 1201 to implement the training method for the anomaly detection model or the item anomaly detection method provided in the method embodiments of this application.

[0235] In some embodiments, the terminal 1200 may also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: radio frequency circuitry 1204, display screen 1205, camera assembly 1206, audio circuitry 1207, and power supply 1208.

[0236] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0237] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0238] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of terminal 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of terminal 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0239] The camera assembly 1206 is used to acquire images or videos. Optionally, the camera assembly 1206 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0240] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1201 for processing, or input to the radio frequency circuit 1204 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1200. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0241] Power supply 1208 is used to power the various components in terminal 1200. Power supply 1208 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1208 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0242] In some embodiments, the terminal 1200 further includes one or more sensors 1209. The one or more sensors 1209 include, but are not limited to: an acceleration sensor 1210, a gyroscope sensor 1211, a pressure sensor 1212, an optical sensor 1213, and a proximity sensor 1214.

[0243] Accelerometer 1210 can detect the magnitude of acceleration on the three coordinate axes of a coordinate system established with terminal 1200. For example, accelerometer 1210 can be used to detect the components of gravitational acceleration on the three coordinate axes. Processor 1201 can control display screen 1205 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1210. Accelerometer 1210 can also be used for games or for acquiring user motion data.

[0244] The gyroscope sensor 1211 can detect the orientation and rotation angle of the terminal 1200. The gyroscope sensor 1211 can work in conjunction with the accelerometer sensor 1210 to collect the user's 3D movements on the terminal 1200. Based on the data collected by the gyroscope sensor 1211, the processor 1201 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0245] The pressure sensor 1212 can be disposed on the side bezel of the terminal 1200 and / or on the lower layer of the display screen 1205. When the pressure sensor 1212 is disposed on the side bezel of the terminal 1200, it can detect the user's grip signal on the terminal 1200, and the processor 1201 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1212. When the pressure sensor 1212 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0246] Optical sensor 1213 is used to collect ambient light intensity. In one embodiment, processor 1201 can control the display brightness of display screen 1205 based on the ambient light intensity collected by optical sensor 1213. Specifically, when the ambient light intensity is high, the display brightness of display screen 1205 is increased; when the ambient light intensity is low, the display brightness of display screen 1205 is decreased. In another embodiment, processor 1201 can also dynamically adjust the shooting parameters of camera assembly 1206 based on the ambient light intensity collected by optical sensor 1213.

[0247] The proximity sensor 1214, also known as a distance sensor, is typically located on the front panel of the terminal 1200. The proximity sensor 1214 is used to detect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1214 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from a screen-on state to a screen-off state; when the proximity sensor 1214 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from a screen-off state to a screen-on state.

[0248] Those skilled in the art will understand that the structure shown in FIG12 does not constitute a limitation on the terminal 1200, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0249] Figure 13 is a schematic diagram of a server according to an embodiment of this application. The server 1300 can vary significantly due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memories 1302 are used to store executable program code, and the processors 1301 are configured to execute the executable program code to implement the training method for the anomaly detection model or the item anomaly detection method provided in the various method embodiments described above. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0250] This application also provides a computer-readable storage medium storing at least one program, which is loaded and executed by a processor to implement a training method for an anomaly detection model or an anomaly detection method for items according to any of the above implementations.

[0251] This application also provides a computer program product, which includes at least one program segment stored in a computer-readable storage medium. The processor of a computer device reads the at least one program segment from the computer-readable storage medium and executes the at least one program segment, causing the computer device to execute the training method for the anomaly detection model or the item anomaly detection method of any of the above implementations.

[0252] In some embodiments, the computer program product involved in the present application can be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0253] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0254] The above embodiments merely illustrate several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for training an anomaly detection model, executed by a computer device, comprising: Multiple first sample images are acquired, and multiple sets of sample pairs are generated based on the multiple first sample images. Each first sample image contains a sample item image, and each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a sample item partial image of the sample item image, and the sample text is used to describe the abnormal situation of the sample item partial image. For each sample pair, features are extracted from the sample patches in the sample pair using an anomaly detection model to obtain the image features of the sample patches. Similarly, features are extracted from the sample text in the sample pair using the same anomaly detection model to obtain the text features of the sample text. The similarity between the image features and the text features is determined. The anomaly detection model is used to perform anomaly detection on the local image of the target object in the input patch. The anomaly detection model is iteratively trained based on the similarity of the multiple sets of samples and a preset similarity.

2. The method according to claim 1, wherein generating multiple sets of sample pairs based on the plurality of first sample images comprises: For each first sample image, the first sample image is segmented using sliding windows of multiple sizes to obtain tile sets of each size. A tile set of one size contains multiple sample tiles of one size. Based on the sample tiles in the tile sets of each of the multiple first sample images and the sample text corresponding to each sample tile, the multiple sets of sample pairs are obtained.

3. The method according to claim 1 or 2, wherein each sample image block corresponds to multiple sample texts, and the multiple sample texts describe the abnormal situation of the local image of the sample item in the sample image block in different texts; The anomaly detection model extracts features from the sample text in the sample pair to obtain the text features of the sample text, including: The anomaly detection model is used to extract features from multiple sample texts in the sample pair to obtain initial text features corresponding to each sample text. The mean of the multiple initial text features is then determined to obtain the text features.

4. The method according to claim 3, further comprising: For each sample image patch, multiple text templates are filled based on the partial image of the sample item in the sample image patch and the anomaly information of the partial image of the sample item to obtain the multiple sample texts. The anomaly information is used to describe the anomaly of the partial image of the sample item, and the multiple text templates are different.

5. The method according to any one of claims 1 to 4, further comprising: Acquire multiple second sample images, each of which contains the sample item image; For each second sample image, the sample item image is segmented from the second sample image to obtain the first sample image.

6. The method according to any one of claims 1 to 5, wherein the method further comprises: Acquire multiple third sample images, each of which contains the sample item image; For each third sample image, the position of the sample item image in the third sample image is calibrated based on the position of the sample item image in the template image to obtain the first sample image, wherein the position of the sample item image in the first sample image matches the position of the sample item image in the template image.

7. A method for detecting anomalies in an item, the method comprising: Acquire multiple tiles of an image, wherein the image contains an image of a target object, and each tile contains a partial image of the target object from the image of the target object; For each image tile, the target text corresponding to the image tile is determined by an anomaly detection model. The anomaly detection model is obtained by the training method of any one of claims 1-6. The target text is used to describe the anomaly of the local image of the target item in the image tile. Based on the target text corresponding to the multiple image tiles, the abnormal information of the target item image in the image is determined, and the abnormal information is used to describe the abnormal situation of the target item image.

8. The detection method according to claim 7, wherein determining the abnormal information of the target object image in the image based on the target text corresponding to the plurality of image tiles respectively includes: If the target text corresponding to at least one of the plurality of image tiles indicates that there is an anomaly in a local image of the target item, it is determined that there is an anomaly in the target item in the image and that there is an anomaly in a local part of the target item; If the target text corresponding to each of the multiple image blocks indicates that there are no anomalies in the local image of the target item, then it is determined that there are no anomalies in the image of the target item.

9. The detection method according to claim 7 or 8, wherein determining the target text corresponding to each image patch using an anomaly detection model includes: For each image patch, the image features of the image patch are extracted using the anomaly detection model. The similarity between the image features and multiple preset text features is determined. From the multiple preset text features, a target text feature whose similarity meets preset requirements is determined. Each of the multiple preset text features corresponds to a preset text. The target text is the preset text corresponding to the target text feature.

10. The detection method according to claim 9, wherein acquiring multiple image patches comprises: The image is segmented using sliding windows of multiple sizes to obtain tile sets of each size, with each tile set containing multiple tiles of that size.

11. The detection method according to claim 10, wherein each image patch contains multiple pixels, and the step of determining the abnormal information of the target object image in the image based on the target text corresponding to the multiple image patches respectively includes: For each tile of each size, if the target text corresponding to the tile indicates that there is an anomaly in the local image of the target item in the tile, the similarity between the image features of the tile and the corresponding target text features is assigned to multiple pixels in the tile. For each pixel, an outlier value is obtained based on the similarity of the pixel across the multiple dimensions. The outlier value is used to indicate the probability that the pixel is abnormal. Based on the outlier values ​​of each pixel in the image, the abnormal information of the target object image in the image is determined. The abnormal information of the target object image includes at least one of the abnormal pixel positions and abnormal areas of the target object.

12. The detection method according to any one of claims 7 to 11, wherein the images are multiple, and the method further comprises: Based on the anomaly information of the target items in multiple images, multiple target images are identified among the multiple images, and the target item images in the target images do not have anomalies. Based on the multiple target images, non-abnormal features of the target item image are determined, wherein the non-abnormal features are the features of a target item that does not have any abnormalities; Anomaly detection is performed on images containing images of target items based on the aforementioned non-abnormal features.

13. The detection method according to claim 7, wherein the images are multiple, and the method further includes: Based on the target text corresponding to the multiple image tiles, multiple target image tiles are determined among the multiple image tiles, and the local image of the target item in the target image tile does not have any anomalies; Based on multiple target patches of each of the multiple images, non-abnormal features of the local image of the target item are determined. The non-abnormal features are the features of the local image of the target item where there are no abnormalities. Anomaly detection is performed on the tiles including the local image of the target item based on the non-abnormal features.

14. A training apparatus for an anomaly detection model, the apparatus comprising: The acquisition module is used to acquire multiple first sample images and generate multiple sets of sample pairs based on the multiple first sample images. Each first sample image contains a sample item image. Each set of sample pairs includes a sample patch and at least one sample text. The sample patch contains a sample item partial image of the sample item image, and the sample text is used to describe the abnormal situation of the sample item partial image. The extraction module is used to extract features from the sample patches in each sample pair using an anomaly detection model to obtain the image features of the sample patches, and to extract features from the sample text in the sample pair using the anomaly detection model to obtain the text features of the sample text, and to determine the similarity between the image features and the text features. The anomaly detection model is used to perform anomaly detection on the local image of the target item in the input patch. The training module is used to iteratively train the anomaly detection model based on the similarity of the multiple sets of samples and a preset similarity.

15. An anomaly detection device for an article, the device comprising: The acquisition module is used to acquire multiple tiles of an image, wherein the image contains an image of a target item, and each tile contains a partial image of the target item from the image of the target item; The determination module is used to determine the target text corresponding to each tile by means of an anomaly detection model. The anomaly detection model is obtained by the training method of any one of claims 1-6. The target text is used to describe the anomaly of the local image of the target item in the tile. The determining module is further configured to determine the abnormal information of the target item image in the image based on the target text corresponding to the plurality of image tiles respectively, and the abnormal information is used to describe the abnormal situation of the target item image.

16. A computer device comprising a processor and a memory, the memory being used to store at least one program, the at least one program being loaded by the processor and executed as a training method for an anomaly detection model according to any one of claims 1 to 6 or an anomaly detection method for an object according to any one of claims 7 to 13.

17. A computer-readable storage medium for storing at least one program for performing a training method for an anomaly detection model according to any one of claims 1 to 6 or an anomaly detection method for an article according to any one of claims 7 to 13.

18. A computer program product comprising at least one program segment stored in a computer-readable storage medium, wherein a processor of a computer device reads the at least one program segment from the computer-readable storage medium, and the processor executes the at least one program segment to cause the computer device to perform a training method for an anomaly detection model according to any one of claims 1 to 6 or an anomaly detection method for an item according to any one of claims 7 to 13.

Images