Anomaly detection model training method and apparatus, and item anomaly detection method and apparatus

Abstract

An anomaly detection model training method, which is executed by means of a computer device. The method comprises: a computer device acquiring a plurality of first sample images, and generating a plurality of sample pairs on the basis of the plurality of first sample images, wherein each first sample image includes a sample item image, and each sample pair comprises a sample image block and at least one piece of sample text, the sample image block including sample item local images of the sample item image, and the sample text being used for describing anomalies of the sample item local images (201); for each sample pair, the computer device performing feature extraction on the sample image block in the sample pair by means of an anomaly detection model, so as to obtain an image feature of the sample image block, performing feature extraction on the sample text in the sample pair by means of the anomaly detection model, so as to obtain a text feature of the sample text, and determining the similarity between the image feature and the text feature, wherein the anomaly detection model is used for performing anomaly detection on target item local images in an input image block (202); and on the basis of the respective similarities of the plurality of sample pairs and a preset similarity, the computer device iteratively training the anomaly detection model (203).

Description

Training method of anomaly detection model, and object anomaly detection method and device

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202410405801.X, filed on April 7, 2024, and entitled "Training method of anomaly detection model, and object anomaly detection method and device", the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present application relates to the field of artificial intelligence, and in particular to a training method of an anomaly detection model, and an object anomaly detection method and device. BACKGROUND

[0004] Industrial anomaly detection plays an important role in actual production. When performing anomaly detection on objects in industrial production, the distribution characteristics of normal objects are generally established first, and then the similarity between the characteristics of the objects to be detected and the distribution characteristics is judged to determine whether the objects are normal. Before the distribution characteristics of normal objects are established, a large number of normal objects need to be collected, that is, the abnormal objects in a large number of objects need to be detected and removed.

[0005] In related technologies, a clustering method is used to detect abnormal objects. The basic assumption of this method is that the proportion of abnormal objects in actual production is low. Therefore, this method clusters objects into clusters based on similarity. Since the proportion of abnormal objects is low, after clustering is completed, some abnormal objects are likely to form a small cluster, which is isolated, and thus it is highly possible that these abnormal objects are abnormal, and these abnormal objects can be removed. However, the proportion of abnormal objects in actual production may not be low, which leads to that some abnormal objects are clustered into a larger cluster after clustering, causing confusion with normal objects, and thus the detection method is not accurate. SUMMARY

[0006] Embodiments of the present application provide a training method of an anomaly detection model, an object anomaly detection method and device. The anomaly detection model trained based on the method can improve the accuracy of anomaly detection. The technical solution is as follows.

[0007] In one aspect, a training method of an anomaly detection model is provided, which is executed by a computer device, and the method comprises:

[0008] obtaining a plurality of first sample images, and generating a plurality of sample pairs according to the plurality of first sample images, each of the first sample images containing a sample article image, each of the sample pairs including a sample patch and at least one sample text, the sample patch containing a sample article local image of the sample article image, and the sample text being used to describe an abnormality of the sample article local image;

[0009] For each of the sample pairs, extracting a feature of the sample patch in the sample pair by using an anomaly detection model to obtain an image feature of the sample patch, and extracting a feature of the sample text in the sample pair by using the anomaly detection model to obtain a text feature of the sample text, determining a similarity between the image feature and the text feature, and the anomaly detection model being used for anomaly detection on a target article local image in an input patch; and

[0010] iteratively training the anomaly detection model based on the similarity of each of the plurality of sample pairs and a preset similarity.

[0011] In another aspect, a device for training an anomaly detection model is provided, and the device includes:

[0012] an obtaining module configured to obtain a plurality of first sample images, and generate a plurality of sample pairs according to the plurality of first sample images, each of the first sample images containing a sample article image, each of the sample pairs including a sample patch and at least one sample text, the sample patch containing a sample article local image of the sample article image, and the sample text being used to describe an abnormality of the sample article local image;

[0013] an extracting module configured to, for each of the sample pairs, extract a feature of the sample patch in the sample pair by using an anomaly detection model to obtain an image feature of the sample patch, and extract a feature of the sample text in the sample pair by using the anomaly detection model to obtain a text feature of the sample text, determine a similarity between the image feature and the text feature, and the anomaly detection model being used for anomaly detection on a target article local image in an input patch;

[0014] a training module configured to iteratively train the anomaly detection model based on the similarity of each of the plurality of sample pairs and a preset similarity.

[0015] In another aspect, an article anomaly detection method is provided, and the method includes:

[0016] obtaining a plurality of patches of an image, the image containing a target article image, and each of the patches containing a target article local image of the target article image;

[0017] For each patch, a target text corresponding to the patch is determined by an anomaly detection model, the anomaly detection model is obtained by the training method, and the target text is used to describe an anomaly of the target local image in the patch.

[0018] Based on the target texts corresponding to the plurality of patches, anomaly information of the target object image in the image is determined, and the anomaly information is used to describe an anomaly of the target object image.

[0019] In another aspect, an article anomaly detection apparatus is provided, and the apparatus comprises:

[0020] An acquisition module is configured to acquire a plurality of patches of an image, the image containing a target object image, and each patch containing a target local image of the target object image.

[0021] A determination module is configured to, for each patch, determine a target text corresponding to the patch by an anomaly detection model, the target text being used to describe an anomaly of the target local image in the patch.

[0022] The determination module is further configured to, based on the target texts corresponding to the plurality of patches, determine anomaly information of the target object image in the image, the anomaly information being used to describe an anomaly of the target object image.

[0023] In another aspect, a computer device is provided, and the computer device comprises a processor and a memory, the memory being configured to store at least one program, the at least one program being loaded and executed by the processor to implement the training method of the anomaly detection model or the article anomaly detection method in the embodiments of the present application.

[0024] In another aspect, a computer readable storage medium is provided, and the computer readable storage medium stores at least one program, the at least one program being loaded and executed by a processor to implement the training method of the anomaly detection model or the article anomaly detection method in the embodiments of the present application.

[0025] In another aspect, a computer program product is provided, and the computer program product comprises at least one program, the at least one program being stored in a computer readable storage medium, a processor of a computer device reading the at least one program from the computer readable storage medium, and the processor executing the at least one program to enable the computer device to perform the training method of the anomaly detection model or the article anomaly detection method according to any of the implementation manners.

[0026] The details of one or more embodiments of the present application are presented in the accompanying drawings and description below. Other features, objects, and advantages of the present application will become apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the disclosed drawings without any creative work.

[0028] FIG1 is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0029] FIG2 is a flow chart of a method for training an anomaly detection model provided in an embodiment of the present application;

[0030] FIG3 is a flowchart of another method for training an anomaly detection model provided in an embodiment of the present application;

[0031] FIG4 is a schematic diagram of an image calibration provided in an embodiment of the present application;

[0032] FIG5 is a flow chart of a method for detecting abnormal objects provided by an embodiment of the present application;

[0033] FIG6 is a flowchart of another method for detecting abnormal objects provided by an embodiment of the present application;

[0034] FIG7 is a schematic diagram of a process for industrial anomaly detection provided by an embodiment of the present application;

[0035] FIG8 is a flow chart of a training phase in industrial detection provided by an embodiment of the present application;

[0036] FIG9 is a flow chart of a test phase in an industrial inspection provided by an embodiment of the present application;

[0037] FIG10 is a block diagram of a training device for an anomaly detection model provided in an embodiment of the present application;

[0038] FIG11 is a block diagram of an object abnormality detection device provided in an embodiment of the present application;

[0039] FIG12 is a block diagram of a terminal provided in an embodiment of the present application;

[0040] FIG13 is a block diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] With reference to the drawings and the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0042] The terms "first", "second", and the like are used to distinguish between the same or similar items or items having substantially the same function, and it should be understood that there is no logical or chronological dependency between "first", "second", and "nth", and the quantity and execution order are not limited. The term "at least one" in the present application means one or more, and the meaning of "multiple" is two or more.

[0043] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions. For example, the sample images involved in the present application are obtained under sufficient authorization.

[0044] In the following, the implementation environment involved in the present application is introduced:

[0045] The training method of the anomaly detection model provided in the embodiments of the present application can be executed by a computer device, which can be provided as a server or a terminal. The implementation environment schematic diagram of the training method of the anomaly detection model provided in the embodiments of the present application is introduced as follows.

[0046] Referring to FIG. 1, FIG. 1 is a schematic diagram of an implementation environment of a training method of an anomaly detection model provided in an embodiment of the present application, which includes a terminal 101 and a server 102. The terminal 101 and the server 102 can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application. In some embodiments, the server 102 is used to train the anomaly detection model, and the trained anomaly detection model is used to determine the text corresponding to the image block, which is used to describe the abnormal situation of the local image of the sample object included in the image block. The target application is installed on the terminal 101, which is used to detect the anomaly of the target object. In some embodiments, the trained anomaly detection model is embedded on the terminal 101, and the terminal 101 detects the anomaly of the object through the anomaly detection model. In other embodiments, the terminal 101 detects the anomaly of the object through the anomaly detection model on the server 102.

[0047] In some embodiments, the terminal 101 can be a smartphone, a tablet computer, a notebook computer, a desktop computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, a flying vehicle, a VR (Virtual Reality) device, an AR (Augmented Reality) device, or the like, but is not limited thereto. In some embodiments, the server 102 can be a standalone server, or a server cluster composed of multiple servers or a distributed system, or a cloud server providing basic cloud computing services such as cloud service, cloud database, cloud computing, cloud function, cloud storage, network service, cloud communication, middleware service, domain name service, security service, CDN (Content Delivery Network), and big data and artificial intelligence platform. In some embodiments, the server 102 mainly undertakes the computing work, and the terminal 101 undertakes the secondary computing work; or the server 102 undertakes the secondary computing service, and the terminal 101 undertakes the main computing work; or the server 102 and the terminal 101 cooperatively compute in a distributed computing architecture.

[0048] Referring to FIG. 2, FIG. 2 is a flowchart of a method for training an anomaly detection model according to an embodiment of the present application. The method comprises the following steps.

[0049] 201, the computer device obtains a plurality of first sample images, and generates a plurality of sample pairs according to the plurality of first sample images. Each first sample image contains a sample object image. Each sample pair includes a sample patch and at least one sample text. The sample patch contains a sample object local image of the sample object image. The sample text is used to describe the abnormality of the sample object local image.

[0050] In the embodiments of the present application, the sample object image can be an image formed by imaging a target object. The target object can be any object to be detected for abnormality, such as various parts produced in industry. The plurality of sample patches are obtained by segmenting the first sample image. One first sample image can be segmented into a plurality of sample patches.

[0051] The plurality of first sample images each contain a sample object image. The sample object image is an image visually representing the shape of the sample object. The sample object image can also present the color or pattern on the sample object. The states of the sample objects in the first sample images can be the same or different, including different sizes, different angles of view, different brightness, etc. The plurality of first sample images include images in which the sample object has no abnormality and images in which the sample object has abnormality.

[0052] In the embodiments of the present application, the abnormal situation refers to the local existence of abnormality or non-existence of abnormality. If the sample article is a part, the local existence of abnormality refers to the local existence of defects of the part, such as surface roughness, surface scratches, insufficient size, etc. Further, for the local existence of abnormality in the sample block, the sample text is also used to describe the category of abnormality, such as surface roughness, surface scratches, insufficient size, etc.

[0053] Each set of sample pairs includes a sample block and at least one sample text. When there are multiple sample texts, the feature mean of the multiple sample texts is used as the text feature of the sample pair for training. For each set of sample pairs, if the sample local image of the sample article included in the sample block in the sample pair does not exist abnormality, then the sample text in the sample pair describes the local non-existence of abnormality in the sample block. If the sample local image of the sample article included in the sample block in the sample pair exists abnormality, then the sample text in the sample pair describes the local existence of abnormality in the sample block, or the sample text in the sample pair describes the category of abnormality existing in the target article local image of the sample block.

[0054] It should be noted that since the sample article has multiple different local parts, such as head and tail parts, middle parts, part head, part tail, etc., the sample text can also describe the sample article local image name included in the sample block, so that the trained abnormality detection model can also determine which local part the block includes.

[0055] 202. The computer device determines, for each set of sample pairs, the similarity between the image feature and the text feature by performing feature extraction on the sample block and the sample text in the sample pair through the abnormality detection model, to obtain the image feature of the sample block and the text feature of the sample text, and the abnormality detection model is used for abnormality detection on the target article local image in the input block.

[0056] In the embodiments of the present application, the abnormality detection model includes an image encoder and a text encoder. The image encoder is used for encoding the image to obtain the image feature. The text encoder is used for encoding the text to obtain the text feature.

[0057] In the embodiments of the present application, the image feature is a vector or a matrix, which is used to represent the sample block. The text feature is a vector or a matrix, which is used to represent the sample text. The similarity can be a cosine similarity.

[0058] 203. The computer device iteratively trains the abnormality detection model based on the respective similarities of the multiple sets of sample pairs and the preset similarity.

[0059] In the embodiments of the present application, the preset similarity can be set and changed as needed. The training target of the anomaly detection model is to make the feature similarity of the sample pair reach the preset similarity. In the embodiments of the present application, the computer device determines the loss value based on the similarity of the sample pair and the preset similarity, and iteratively adjusts the model parameters of the anomaly detection model based on the loss value. The model parameters of the anomaly detection model include the parameters of the image encoder and the parameters of the text encoder. The preset similarity can be determined through multiple experiments according to the distribution of the actual training data and the model performance requirements, and a suitable value is selected, such as a value in the range of [0.8, 0.9].

[0060] In the embodiments of the present application, the computer device iteratively trains the anomaly detection model based on the respective similarities of the multiple groups of sample pairs and the preset similarity, until the preset requirement is reached. The preset requirement can be that the loss value converges, or the loss value reaches a preset threshold, or the number of iterations reaches a preset number, or the similarity reaches a preset similarity, which is not limited herein.

[0061] In the embodiments of the present application, the trained anomaly detection model is used to extract the image features of the input image block, determine the target text feature from the multiple preset text features, and determine the similarity between the image features and the target text feature. The text corresponding to the target text feature is used as the text corresponding to the image block, and the text is used to describe the abnormal situation of the local image of the target object in the image block. Alternatively, the multiple preset text features are obtained by performing feature extraction on the multiple preset texts by the trained anomaly detection model. The target object can be the same or similar to the sample object.

[0062] The embodiment of the present application provides a training method of an anomaly detection model. The method trains the anomaly detection model based on sample blocks and sample texts used for describing abnormal conditions of local images of sample articles in the sample blocks. Since the anomaly detection model extracts image features and text features, and the pair of image features and text features should have high similarity, the anomaly detection model is trained based on the similarity between the image features and the text features and a preset similarity, so that the anomaly detection model learns the rule that the pair of image features and text features have high similarity, and the trained anomaly detection model can accurately extract features. Thus, for any block containing a local image of a target article, the text feature with high similarity to the image feature of the block can be determined based on the extracted image feature, and the text corresponding to the text feature is the text used for describing the abnormal condition of the local image of the target article in the block. Therefore, the anomaly detection model trained based on the method has high accuracy in anomaly detection of the local image of the target article in the block. Moreover, the method is trained based on the block, and the block contains the local image of the sample article, so that the anomaly detection model can specifically detect which local image of the target article image has an abnormality, thereby improving the accuracy of anomaly detection. Therefore, the anomaly detection model trained based on the method can improve the accuracy and precision of anomaly detection.

[0063] The above Figure 2 is a basic flow of the training method of the anomaly detection model. The training method of the anomaly detection model is further introduced based on Figure 3. Referring to Figure 3, Figure 3 is a flowchart of the training method of the anomaly detection model provided by the embodiment of the present application. The method comprises the following steps.

[0064] 301, the computer device obtains a plurality of second sample images, each second sample image containing a sample article image. For each second sample image, the sample article image is segmented from the second sample image to obtain a plurality of first sample images, and the first sample image contains the sample article image.

[0065] In the embodiment of the present application, the second sample image contains not only the sample article image but also a background, and the background includes other interference elements. Therefore, the sample article image is segmented from the second sample image to realize the differentiation between the sample article image and the background in the second sample image, and then the background in the second sample image is removed so that the second sample image only retains the sample article image, that is, the first sample image is obtained.

[0066] Optionally, the computer device performs image segmentation by a SAM (Segment Anything Model) model. The computer device inputs the second sample image into the SAM model, segments the sample item in the second sample image by the model, and distinguishes the foreground from the background, so as to remove the background and eliminate the noise interference caused by the background. Further, the computer device sets the pixel value of the segmented background to 0, that is, sets the background to black, and accordingly, the sample item in the first sample image is the foreground, and the background other than the sample item in the first sample image is black.

[0067] In this embodiment, the sample item is segmented from the second sample image to remove the background in the second sample image, so as to avoid the interference caused by the background and facilitate subsequent image processing.

[0068] In some embodiments, the computer device further calibrates the position of the sample item in the sample image. The computer device obtains a plurality of third sample images, and each of the plurality of third sample images contains a sample item image. For each third sample image, the position of the sample item image in the third sample image is calibrated based on the position of the sample item image in the template image, to obtain a first sample image, and the position of the sample item image in the first sample image matches the position of the sample item image in the template image.

[0069] The position of the sample item image in the first sample image matching the position of the sample item image in the template image means that the two positions are the same or the coordinate difference between the two positions is within a preset range.

[0070] Optionally, the computer device inputs the third sample image into an image calibration module, and the image calibration module calibrates the position of the sample item image in the third sample image based on the position of the sample item image in the template image to obtain the first sample image. The image calibration module is configured to calibrate the input image, so as to accurately extract the features of the region where the sample item image is located in the image subsequently.

[0071] Optionally, the image calibration module uses a least error iteration method to calibrate. The method constructs an error function, defines the parameters to be estimated, and optimizes the parameters by an iterative algorithm based on the current estimated value to gradually reduce the error function. The image calibration module first performs edge detection on the sample item in the input image and the template image to obtain the contour maps of the two. Then, the contour maps are subjected to a non-maximum suppression (NMS) operation to convert the contour maps into a two-dimensional contour point set. Alternatively, the contour maps are subjected to partition processing to obtain a two-dimensional contour point set of each region in the image, so as to partition the image. The average error of the two-dimensional contour point sets between the input image and the template image is the error function. The error function is shown in the following formula (1).

[0072] wherein F represents an error function, i.e. the average error, K represents the number of contour points in the template image, P K represents the coordinate of the kth contour point in the input image, represents the coordinate of the kth contour point in the template image, and ∑ represents the accumulation.

[0073] After setting the error function, the input image is calibrated by using an iterative optimization method. First, the contour point set to be registered is registered by using the current estimated transformation matrix, i.e. the initial transformation matrix. Then, the position matching relationship between the input image and the contour point set of the template image is established by using the nearest neighbor search method, and the matching pairs with too large errors, i.e. the contour points with positions difficult to match, are removed. Finally, the transformation matrix is estimated again by using the RANSAC algorithm, and returns to the first step until the error function converges, and the target transformation matrix is obtained. Based on the target transformation matrix, the coordinates of each point in the input image are transformed to obtain the calibrated image.

[0074] The nearest neighbor search method is an algorithm for establishing the position matching relationship between point sets. In image calibration, it is used to establish the position matching relationship between the input image and the contour point set of the template image, and the matching pairs with too large errors, i.e. the contour points with positions difficult to match, are removed.

[0075] The RANSAC algorithm, i.e. the random sample consensus algorithm, is used to estimate the transformation matrix again in image calibration. First, the contour point set to be registered is registered by using the current estimated transformation matrix, then the position matching relationship between the input image and the contour point set of the template image is established by using the nearest neighbor search method, the matching pairs with too large errors are removed, and finally the transformation matrix is estimated again by using the algorithm until the error function converges, and the target transformation matrix is obtained.

[0076] In other embodiments, the computer device can also use a photometric loss function for unsupervised image alignment learning to calibrate the image, which is not specifically limited herein. The photometric loss function is a function for unsupervised image alignment learning, which can be used for the position calibration of the sample article image in the sample image in the present application.

[0077] In this embodiment, the position of the sample object image in the sample image is calibrated, so as to facilitate subsequent extraction of the region of the sample object image in the sample image for processing, such as facilitating segmentation of the sample object image from the image, facilitating determination of the region of the sample object image in the image, and facilitating extraction of the features of the sample object image in the image.

[0078] For example, referring to FIG. 4, which is a schematic diagram of image calibration provided in an embodiment of the present application. First, the target transformation matrix between the template image and the input image is determined, and then the input image is processed based on the target transformation matrix to minimize the position difference between the input image and the sample object image in the template image, thereby obtaining the calibrated image. The orientation and angle of the sample object image in the calibrated image are the same as those of the sample object image in the template image.

[0079] In some embodiments, the computer device first calibrates the position of the sample object image before segmenting the second sample image, thereby facilitating image segmentation. Alternatively, the computer device calibrates the position of the sample object image after segmenting the second sample image, which facilitates calibration of the position of the sample object image due to the elimination of the interference of the background.

[0080] 302. The computer device segments each first sample image using a plurality of sizes of sliding windows respectively to obtain a plurality of sets of patches of the respective sizes, wherein each set of patches of a size contains a plurality of sample patches of the size, and each sample patch contains a sample object partial image of a sample object image.

[0081] A sliding window refers to a window with a certain size used for local processing of an image in the process of image segmentation. The window slides on the image according to a certain rule, and the region passed by the window is operated to obtain different local information of the image. In the present application, a plurality of sizes of sliding windows are used to segment the first sample image or the image containing the target object image to obtain sets of patches of different sizes.

[0082] In the embodiments of the present application, the size of each sliding window can be set and changed as needed, which is not specifically limited herein. For example, the resolution of the first sample image x is h x w, and a given encoder f is used for image segmentation. The plurality of sizes of sliding windows are represented as {w ij}, and each sliding window {w ij}∈{0,1} h×w is a binary mask. It is used to represent the local activity of the k x k kernel around the pixel point (i, j). i, j, h, w, and k are non-negative integers. The sample patch corresponding to each sliding window is represented as The effective area of the first sample image, i.e., a sample tile, is defined according to a plurality of sliding windows of different sizes The kernel size k corresponds to the number of surrounding contexts of each position in the calculation image, which controls the balance between local details and global information in the segmentation. The kernel size corresponds to the number of surrounding contexts of each position in the calculation image, which controls the balance between local details and global information in the segmentation when the image is segmented using a sliding window.

[0083] In some embodiments, the image encoder in the anomaly detection model can automatically segment the input image. In other embodiments, the anomaly detection model cannot segment the input image, and the computer device segments the image before inputting it into the anomaly detection model.

[0084] In some embodiments, the computer device performs image segmentation by the ViT patch method, which includes a plurality of sliding windows of different sizes. For example, a smaller sliding window of 2x2 corresponds to an image of 32x32 pixel resolution, and a medium-sized sliding window of 3x3 corresponds to an image of 48x48 pixel resolution. In other embodiments, the computer device performs image segmentation by the ViT token method to capture the context of the image.

[0085] In the embodiments of the present application, the first sample images are segmented by a plurality of sliding windows of different sizes, so that each first sample image corresponds to a plurality of sample tiles of different sizes, and the sample tiles of different sizes include local parts of the sample items from small to large. Therefore, by using a plurality of sample tiles of different sizes, the defects of the sample item images in the sample images from small to large can be captured, the diversity of the samples is improved, and the generalization ability and accuracy of the anomaly detection model can be improved by training the anomaly detection model based on these samples.

[0086] 303、The computer device obtains a plurality of groups of sample pairs based on the sample tiles in the tile set of each of the plurality of first sample images and the sample texts corresponding to each sample tile, each group of sample pairs including a sample tile and at least one sample text, and the sample text is used to describe the abnormal situation of the local image of the sample item.

[0087] In the embodiments of the present application, each first sample image is segmented into a plurality of sample tiles, and correspondingly, each first sample image corresponds to a plurality of groups of sample pairs.

[0088] In some embodiments, the same abnormal situation can be described by different texts, and thus one sample block can correspond to multiple sample texts, and the multiple sample texts respectively describe the abnormal situation of the local image of the sample item in the sample block by different texts, and thus one sample block can form multiple sample pairs, and the sample blocks in the multiple sample pairs are the same, and the sample texts are different.

[0089] In some embodiments, each sample block corresponds to multiple sample texts, and the multiple sample texts respectively describe the abnormal situation of the local image of the sample item in the sample block by different texts. The process of extracting the text feature of the sample text in the sample pair by the computer device through the abnormality detection model includes the following steps: the computer device extracts the features of the multiple sample texts in the sample pair respectively through the abnormality detection model to obtain the initial text features corresponding to the multiple sample texts respectively, determines the mean of the multiple initial text features to obtain the text feature.

[0090] In the embodiments of the present application, the multiple initial text features are multiple vectors with the same dimension, and the text feature is also a vector, and the dimension of the text feature is the same as that of the initial text feature. Then, determining the mean of the multiple initial text features means determining the mean of the multiple vectors to obtain the text feature. Alternatively, the multiple initial text features are multiple matrices with the same dimension, and the text feature is also a matrix, and the dimension of the text feature is the same as that of the initial text feature. Then, determining the mean of the multiple initial text features means determining the mean of the multiple matrices to obtain the text feature.

[0091] In the embodiments of the present application, the multiple sample texts respectively describe the abnormal situation of the local image of the sample item in the sample block by different texts. The multiple sample texts include but are not limited to at least one of the following cases: each sample text includes at least one word different from the words in other sample texts; the number of words included in the multiple sample texts is different; and the order of the multiple words in the multiple sample texts is different.

[0092] For example, the sample text used to describe that there is no abnormality in the local can be “perfect”, and can also be “defect-free”. The sample text used to describe that there is an abnormality in the local can be “damaged”, and can also be “defective”.

[0093] In the embodiments of the present application, since the same abnormal situation can be described by different texts, averaging the text features of the multiple texts describing the same abnormal situation makes the obtained text feature more accurate and more effective.

[0094] In some embodiments, the computer device obtains the plurality of sample texts by the following steps. In the steps, the computer device fills the plurality of text templates based on the sample item local image in the sample image block and the abnormal information of the sample item local image to obtain the plurality of sample texts, the abnormal information is used to describe the abnormal situation of the sample item local image, and the plurality of text templates are different.

[0095] The text template is a preset general format used to generate a text describing the abnormal situation of the sample item local image. The plurality of text templates are different, for example, each text template includes at least one word different from the words in other text templates, the number of words included in the plurality of text templates is different, or the order of the plurality of words in the plurality of text templates is different. By filling the text template based on the sample item local image and the abnormal information thereof, the plurality of sample texts can be obtained.

[0096] For example, the text template can be ‘[c]’s cropped photo’ or ‘[c]’s photo’, c is at least one of the abnormal information of the sample item local image to be filled and the name of the sample item local image; it can also be ‘perfect [o]’, ‘flawless [o]’, ‘damaged [o]’, or ‘defective [o]’, where o represents the name of the sample item local image to be filled or the name of the sample item image.

[0097] In the embodiments of the present application, the plurality of text templates are preset, and the plurality of text templates include but are not limited to at least one of the following cases: each text template includes at least one word different from the words in other text templates; the number of words included in the plurality of text templates is different; and the order of the plurality of words in the plurality of text templates is different.

[0098] The text template can be filled based on the sample item local image and the abnormal information of the sample item local image, or the text template can be filled based on one of the local image and the abnormal information. For example, the text template can be ‘[c]’s cropped photo’ or ‘[c]’s photo’, c is at least one of the abnormal information of the sample item local image to be filled and the name of the sample item local image.

[0099] In this embodiment, the text template is a general text template for multiple abnormal situations, and the plurality of sample texts are obtained based on the plurality of text templates. The text template only needs to be filled based on the abnormal situation, which improves the convenience of obtaining the sample texts.

[0100] In some embodiments, a text template is provided for describing different abnormal situations, and thus only the name of the sample local image needs to be filled in the text template to obtain the sample text. For example, the text template can be "perfect [o]", "defect-free [o]", and can also be "damaged [o]", "defective [o]", where o represents the name of the sample local image to be filled, such as "screw head". Alternatively, o can also represent the name of the sample image to be filled, such as "screw".

[0101] In the embodiments of the present application, the process of generating a plurality of sample pairs from a plurality of first sample images is implemented through steps 302-303. In this embodiment, the first sample image is segmented by a plurality of sizes of sliding windows to obtain a plurality of patches of the first sample image of a plurality of sizes, which not only improves the diversity of the sample, but also can capture defects of the sample object from small to large, so that the anomaly detection model can identify defects of the sample object from small to large, and the accuracy of the anomaly detection model is improved.

[0102] It should be noted that steps 302-303 are only one optional implementation of the process, and the computer device can also implement the process through other optional implementations; for example, the first sample image is segmented by a sliding window of one size to obtain a plurality of patches of the first sample image, and then the sample pairs are obtained based on the sample patches and the sample texts.

[0103] 304、The computer device extracts features of the sample patch and the sample text in the sample pair respectively through the anomaly detection model to obtain image features of the sample patch and text features of the sample text, determines the similarity between the image features and the text features, and the anomaly detection model is used for anomaly detection of a target local image in an input patch.

[0104] In the embodiments of the present application, feature extraction of the sample patch and the sample text can obtain more expressive representations of the sample patch and the sample text, which facilitates subsequent model anomaly judgment.

[0105] In some embodiments, a trained feature extraction network is used for preliminary feature extraction, such as a ResNet50 network. The feature extraction network is pre-trained on a large-scale dataset, which can better extract semantic features of images and texts. In the subsequent learning process, the parameters of the network remain unchanged, that is, the network is only responsible for feature extraction, and the parameters are not updated in the subsequent training process. Alternatively, at least one fully connected layer is connected to the back of the network, and the fully connected layer is used to extract features again from the features output by the network to obtain corresponding image features or text features. That is, only the parameters of the fully connected layer need to be adjusted in the training process, which can improve the training efficiency.

[0106] The pre-trained model is referred to as a pre-training model (Pre-training model), also known as a cornerstone model, a large model, and refers to a deep neural network (Deep Neural Network, DNN) with a large number of parameters. The pre-training model is trained on a large amount of unlabeled data, and the function approximation capability of the large parameter DNN (Deep Neural Network) is used to extract common features from the data. Through fine tuning, parameter-efficient fine tuning (PEFT, Parameter-Efficient Fine-Tuning), prompt-tuning (rapid adjustment), and other technologies, the pre-training model is suitable for downstream tasks. Therefore, the pre-training model can achieve ideal results in a few-shot or zero-shot scenario. The PTM can be divided into language models (ELMO, BERT, GPT), visual models (swin-transformer, ViT, V-MOE), speech models (VALL-E), and multi-modal models (ViBERT, CLIP, Flamingo, Gato) according to the data modalities processed, wherein the multi-modal model refers to a model that establishes a feature representation of two or more data modalities. The pre-training model is an important tool for outputting artificial intelligence generated content (AIGC), and can also be used as a general interface connecting multiple specific task models.

[0107] Zero-shot transfer ability refers to the ability of a model to apply knowledge learned from large-scale, multi-modal data sets to new tasks without training samples for specific tasks. In this application, large models such as CLIP have good zero-shot transfer ability by pre-training on large-scale, multi-modal data sets, and can be used for tasks such as abnormal sample detection and rejection.

[0108] 305、The computer device iteratively trains the anomaly detection model based on the similarity of each sample pair and the preset similarity.

[0109] In the embodiments of the present application, the similarity of each sample pair refers to the similarity between the features of the sample patches and the sample text in the sample pair.

[0110] In each iteration process, the model parameters of the anomaly detection model can be adjusted based on the similarity of the at least one set of sample pairs and the preset similarity. For example, a loss value between the similarity of the at least one set of sample pairs and the preset similarity is determined, and the model parameters are adjusted based on the mean of the plurality of loss values. In the next iteration process, the next batch of at least one set of sample pairs is input into the adjusted anomaly detection model to obtain the similarity of each of the next batch of at least one set of sample pairs, and then the model parameters of the anomaly detection model are adjusted based on the similarity of the at least one set of sample pairs and the preset similarity. The above iteration process is repeated until a preset requirement is reached.

[0111] In some other embodiments, the above-mentioned sample pairs are positive sample pairs, and the computer device can further construct negative sample pairs, and train in combination of the positive sample pairs and the negative sample pairs. The negative sample pairs include a sample patch and a negative sample text, and the negative sample text does not describe the abnormal condition of the local image of the sample item in the sample patch. Further, the negative sample text is any sample text in the positive sample pairs other than the positive sample pair in which the sample patch is located, thereby improving the efficiency of obtaining sample texts.

[0112] In the embodiments of the present application, in the training process of the anomaly detection model, the image features and the text features are respectively extracted by the image encoder and the text encoder in the anomaly detection model, the paired image-text is pulled closer in the feature space by using the loss function of contrast learning, and the unpaired image-text is pushed further apart in the feature space, so as to realize the pre-training of the image-text pair. Therefore, in the use stage of the anomaly detection model, different preset texts such as “a defective item” and “an item without defects” are used to respectively describe the abnormal condition of the sample item image. Then, the text features of different texts and the image features of the input image are extracted by the anomaly detection model, the similarity between the extracted image features and each preset text feature is compared, and it is determined which text the image is closer to, so as to realize the detection of anomaly.

[0113] Contrastive learning is a machine learning method. In the training process of the anomaly detection model of the present application, the loss function of contrast learning is used to pull the paired image-text closer in the feature space, and push the unpaired image-text further apart in the feature space, so as to realize the pre-training of the image-text pair, and make the model learn the rule that the paired image features and text features have a high similarity.

[0114] The loss function of contrast learning can have the following form:

[0115] wherein sim(I, T) represents the similarity of the image feature I and the text feature T, τ is a temperature parameter, and N is the number of samples. The temperature parameter is a parameter in the loss function of the contrast learning, which is used to adjust the weight of the similarity in the loss function and affects the learning degree of the model to different samples in the training process.

[0116] The embodiment of the present application provides a training method of an anomaly detection model. The method trains the anomaly detection model based on sample patches and sample texts used for describing abnormal conditions of local images of sample items in the sample patches. Since the anomaly detection model extracts image features and text features, and the pair of image features and text features should have a high similarity, the anomaly detection model is trained based on the similarity between the image features and the text features and a preset similarity, so that the anomaly detection model learns the rule that the pair of image features and text features have a high similarity, and then the trained anomaly detection model can accurately extract features. Thus, for any patch containing a local image of a target item, the text feature with a high similarity to the image feature of the patch can be determined based on the extracted image feature, and then the text corresponding to the text feature is the text used for describing the abnormal condition of the local image of the target item in the patch. Therefore, the accuracy of the anomaly detection model trained based on the method in detecting the abnormal condition of the local image of the target item in the patch is high. Moreover, since the method is trained based on the patch, and the patch includes the local image of the sample item, the anomaly detection model can specifically detect which local image of the target item image has an abnormality, that is, the precision of the anomaly detection is improved. Therefore, the anomaly detection model trained based on the method can improve the accuracy and precision of the anomaly detection.

[0117] The anomaly detection model is trained through the above embodiments of FIG. 2 and FIG. 3. Next, the anomaly detection is performed based on the anomaly detection model trained in FIG. 2 or FIG. 3. Referring to FIG. 5, FIG. 5 is an item anomaly detection method provided by an embodiment of the present application. The method includes the following steps.

[0118] 501, the computer device acquires a plurality of patches of an image, the image containing a target item image, and each patch containing a local image of a target item of the target item image.

[0119] In the embodiment of the present application, the target item can be any item to be detected for anomaly, such as various parts produced in industry.

[0120] 502, the computer device determines, for each patch, a target text corresponding to the patch through the anomaly detection model, the target text being used for describing an abnormal condition of the local image of the target item in the patch.

[0121] In the embodiment of the present application, the abnormality detection model is used for abnormality detection on the target object local image in the input tile. Wherein, the computer device inputs the tile into the abnormality detection model, extracts the image feature of the tile through the abnormality detection model, determines the text feature with which the similarity degree meets the preset requirement, and then takes the text corresponding to the text feature as the target text.

[0122] 503、The computer device determines the abnormality information of the target object image in the image based on the target texts corresponding to the plurality of tiles respectively, and the abnormality information is used for describing the abnormality of the target object image.

[0123] In some embodiments, the abnormality information of the target object image includes the target texts corresponding to the plurality of tiles respectively, so as to obtain the abnormality of each target object local image of the target object; or the abnormality information is used for describing whether the target object as a whole is abnormal. For example, the abnormality information can be "the target object is abnormal" or "the target object is not abnormal".

[0124] The embodiment of the present application provides an object abnormality detection method, which is detected based on an abnormality detection model. The abnormality detection model is trained based on sample tiles and sample texts used for describing the abnormality of sample object local images in the sample tiles. Since the abnormality detection model extracts image features and text features, and the pair of image features and text features should have a high similarity, the abnormality detection model is trained based on the similarity between the image features and the text features and the preset similarity, so that the abnormality detection model learns the rule that the pair of image features and text features have a high similarity, and then the trained abnormality detection model can accurately extract features. Thus, for any tile containing a target object local image, based on the extracted image features, a text feature with a high similarity can be determined, and then the text corresponding to the text feature is the text used for describing the abnormality of the target object local image in the tile. Therefore, the accuracy of the abnormality detection model trained based on the method for detecting the abnormality of the target object local image in the tile is high, and then based on the abnormality of each tile in the image, the overall abnormality of the target object image in the image can be obtained, thereby improving the convenience and accuracy of the abnormality detection.

[0125] The embodiment of FIG. 5 is a basic process of object abnormality detection based on an abnormality detection model. The object abnormality detection method is further introduced based on the embodiment of FIG. 6. Referring to FIG. 6, FIG. 6 is a flowchart of an object abnormality detection method provided by an embodiment of the present application. The method includes the following steps.

[0126] 601、The computer device obtains a plurality of tiles of an image, and the image contains a target object image. Each tile contains a target object local image of the target object image.

[0127] In some embodiments, the computer device obtains an initial image, the initial image containing an overall image of the target object image; the computer device segments the target object from the initial image to obtain the image. Alternatively, the computer device calibrates the position of the target object image in the initial image based on the position of the target object image in the template image to obtain the image. Alternatively, the computer device segments the target object from the initial image to obtain a first image, and then calibrates the position of the target object image in the first image based on the position of the target object image in the template image to obtain the image. Alternatively, the computer device calibrates the position of the target object image in the initial image based on the position of the target object image in the template image to obtain a second image, and then segments the target object from the second image to obtain the image.

[0128] The specific process of the computer device segmenting and calibrating the image is the same as that of step 301, and will not be repeated here.

[0129] In some embodiments, the process of the computer device obtaining the plurality of tiles of the image includes the following steps: the computer device segments the image with a plurality of sizes of sliding windows to obtain a plurality of sets of tiles of the respective sizes, each set of tiles of a size including a plurality of tiles of the size.

[0130] The process of the computer device segmenting the image is the same as that of segmenting the first sample image in step 302, and will not be repeated here.

[0131] 602、The computer device extracts image features of the tile through the anomaly detection model, determines similarities between the image features and a plurality of preset text features respectively, determines a target text feature that satisfies a preset requirement from the plurality of preset text features, obtains a target text based on the target text feature, and the plurality of preset text features respectively correspond to preset texts, the target text is a preset text corresponding to the target text feature, and the target text is used to describe an abnormal situation of a local image of the target object in the tile.

[0132] The plurality of preset text features respectively correspond to preset texts; the computer device inputs the plurality of preset texts into the anomaly detection model, respectively extracts features of the plurality of preset texts through the anomaly detection model, and obtains preset text features of the plurality of preset texts respectively. In this embodiment, the plurality of preset text features are determined in advance through the anomaly detection model, and then the plurality of preset text features can be repeatedly used for detection of a large number of tiles, improving the efficiency and saving resources.

[0133] In the embodiment of the present application, the similarity between the target text feature and the image feature meets the preset requirement, that is, the similarity between the target text feature and the image feature reaches the target similarity, or the similarity between the target text feature and the image feature is the maximum among the similarities between the plurality of preset text features and the image feature.

[0134] In some embodiments, the preset text is two, that is, normal text and abnormal text, and the corresponding two preset text features are normal text feature and abnormal text feature. The normal text is used to describe that the local in the patch does not exist abnormality, and the abnormal text is used to describe that the local in the patch exists abnormality. If the similarity between the image feature and the normal text feature is higher, the local in the patch does not exist abnormality, and the patch is a normal patch, that is, the target patch. If the similarity between the image feature and the abnormal text feature is higher, the local in the image exists abnormality, and the patch is an abnormal patch.

[0135] In the embodiment of the present application, the process of determining the target text corresponding to the patch by the abnormality detection model for each patch is realized by the above step 602. In this embodiment, since the trained abnormality detection model can accurately extract the image feature of the patch and can accurately extract the text feature of the text, the similarity between the text feature of the text used to describe the abnormality of the local image of the target object in the patch and the image feature is high, and then a plurality of text features are preset, the similarity between the plurality of text features and the image feature is compared respectively, and the text corresponding to the text feature whose similarity meets the preset requirement is the text used to describe the abnormality of the local image of the target object in the patch, that is, by the abnormality detection model, the fast and accurate detection of the abnormal local image of the target object in the image of the target object in the patch is realized.

[0136] It should be noted that step 602 is only one optional implementation manner to realize the process, and the computer device can also realize the process by other optional implementation manners, which will not be described here.

[0137] 603. The computer device determines the abnormal information of the image of the target object in the image based on the target text corresponding to the plurality of patches respectively, and the abnormal information is used to describe the abnormality of the image of the target object.

[0138] In some embodiments, the process of determining the abnormal information of the image of the target object in the image based on the target text corresponding to the plurality of patches respectively by the computer device includes the following steps: in the case that the target text corresponding to at least one patch in the plurality of patches indicates that the local image of the target object exists abnormality, the computer device determines that the target object in the image exists abnormality and determines the local where the target object exists abnormality; in the case that the target text corresponding to each patch in the plurality of patches indicates that the local image of the target object does not exist abnormality, the computer device determines that the image of the target object in the image does not exist abnormality.

[0139] If the text corresponding to at least one tile indicates that the local image of the target object exists abnormally, it is determined that at least one local image of the target object in the target object image exists abnormally, and further that the target object exists abnormally. On this basis, since the plurality of tiles respectively contain local images of the target object, the local image of the target object in which the target object exists abnormally is determined based on the at least one tile.

[0140] In some embodiments, the target text also indicates the kind of abnormality, and the kind of abnormality of the local image of the target object in which the target object exists abnormally can also be determined based on the target text. Accordingly, the abnormality information includes the local image of the target object in which the target object exists abnormally and the kind of abnormality of the local image of the target object. In some embodiments, based on the proportion of the at least one tile in the plurality of tiles, the area in which the target object exists abnormally can also be determined. Accordingly, the abnormality information includes the area in which the target object image exists abnormally.

[0141] In other embodiments, the abnormality information includes the target text corresponding to each of the plurality of tiles, and thus, through the target text of each of the plurality of tiles in the abnormality information, it can be known which local images of the target object in which tiles exist abnormally or which local images of the target object in which tiles do not exist abnormally.

[0142] In the embodiments of the present application, the overall abnormality of the target object image is determined based on the target text of each of the plurality of tiles, which realizes the summarization of the abnormality information of the plurality of tiles, improves the readability of the abnormality detection result, and further improves the user experience.

[0143] In some embodiments, the computer device is segmented by a plurality of sliding windows of different sizes to obtain a plurality of tiles of different sizes, each tile containing a plurality of pixel points. The process of determining the abnormality information of the target object image in the image based on the target text corresponding to each of the plurality of tiles includes the following steps: for each tile of each size, if the target text corresponding to the tile indicates that the local image of the target object in the tile exists abnormally, the similarity between the image features of the tile and the target text features corresponding to the tile is assigned to the plurality of pixel points in the tile; for each pixel point, based on the similarity of the pixel point under a plurality of sizes, an abnormality value of the pixel point is obtained, the abnormality value being used to indicate the probability of the pixel point existing abnormally; based on the abnormality value of each pixel point in the image, the abnormality information of the target object image in the image is determined, the abnormality information of the target object image including at least one of the position of the pixel point in which the target object image exists abnormally and the abnormal area.

[0144] Optionally, the computer device obtains the abnormal value of the pixel point based on the similarity of the pixel point in multiple sizes, and the process comprises the following steps: the computer device determines the mean value of the similarity of the pixel point in multiple sizes to obtain the abnormal value of the pixel point, and the mean value can be a harmonic mean or an arithmetic mean. Alternatively, the computer device determines the sum of the similarity of the pixel point in multiple sizes to obtain the abnormal value of the pixel point. The harmonic mean is a statistical average value, and for a set of data x1, x2, …, xN, the harmonic mean H is calculated as follows: n The calculation formula of the harmonic mean H is as follows: In this application, the mean value of the similarity of the pixel point in multiple sizes can be used to calculate the abnormal value of the pixel point.

[0145] Optionally, the computer device determines the abnormal information of the target object image in the image based on the abnormal value of each pixel point in the image, and the process comprises the following steps: the computer device determines a plurality of target pixel points with abnormal values greater than an abnormal threshold value from a plurality of pixel points in the image, determines the pixel points on the target object in the plurality of target pixel points to obtain the pixel point positions with abnormalities on the target object. Further, the computer device obtains the pixel points with abnormalities on the target object, determines the ratio between the number of the pixel points and the number of the plurality of pixel points on the target object to obtain the abnormal area of the target object image.

[0146] In other embodiments, the computer device determines the mean value of the similarity between the image feature of the tile and a plurality of preset text features, and assigns the mean value to a plurality of pixel points in the tile. Alternatively, the computer device determines a plurality of first text features in the plurality of preset text features for describing local abnormalities, determines the mean value of the similarity between the image feature of the tile and the plurality of first text features, and assigns the mean value to a plurality of pixel points in the tile. Alternatively, the computer device can also determine a plurality of second text features in the plurality of preset text features with a similarity greater than a similarity threshold value with the image feature, determine the mean value of the similarity between the image feature of the tile and the plurality of second text features, and assign the mean value to a plurality of pixel points in the tile.

[0147] In this embodiment, for each pixel point, the abnormal value of each pixel point is more accurate by fusing the similarity of the pixel point in multiple sizes, that is, the probability of the abnormal value indicating that the pixel point has an abnormality is more accurate, and then the abnormal information of the target object image in the image is determined by the abnormal values of a plurality of pixel points, so that the abnormal information is more accurate.

[0148] In some embodiments, the abnormality detection model provided in the embodiments of the application is used to screen abnormal samples, and the remaining normal samples are used as training data to construct an abnormality detection network, and the abnormality detection network is used to detect abnormalities of the object.

[0149] The anomaly detection network is a network for detecting anomalies of an article. The network constructs a non-anomalous feature of a sample article image or a sample article local image based on normal samples, and detects an article or a local part with a large difference from the non-anomalous feature by comparing a feature of a to-be-detected image or a to-be-detected patch with the non-anomalous feature, and judges that the article or the local part is an article or a local part with an anomaly.

[0150] In the embodiment, the target article is a sample article. The abnormal sample refers to a sample image in which the included sample article has an anomaly, and the normal sample refers to a sample image in which the included sample article has no anomaly. The abnormal sample is detected from a large number of sample images by an anomaly detection model, and the anomaly detection network is constructed based on the remaining normal samples. Specifically, a non-anomalous feature of a sample article image is constructed based on the normal samples, and the non-anomalous feature is a feature distribution of the sample article image in the normal sample. Then, the anomaly detection network detects an anomalous article based on the non-anomalous feature, that is, an article with a large difference from the non-anomalous feature is detected, and such an article is an article with an anomaly. The construction and use process of the anomaly detection network are described in the following steps, which are optional implementation steps.

[0151] 604. The computer device determines a plurality of target images in the plurality of images based on the respective anomaly information of the target articles in the plurality of images, and the target article image in the target image has no anomaly.

[0152] In the embodiment of the present application, the anomaly information is used to describe the anomaly of the target article image, and therefore, based on the anomaly information, it can be determined whether the target article in the image has an anomaly. The image in which the anomaly information indicates that the target article image has no anomaly is taken as the target image.

[0153] 605. The computer device determines a non-anomalous feature of the target article image based on the plurality of target images, and the non-anomalous feature is a feature possessed by the target article with no anomaly.

[0154] The computer device respectively extracts features of the plurality of image features to obtain respective image features of the plurality of target images, and obtains the non-anomalous feature of the target article image based on the respective image features of the plurality of target images.

[0155] Optionally, the computer device extracts features by the anomaly detection model, or the computer device extracts features by other feature extraction networks, such as a ResNet50 network.

[0156] The non-abnormal feature includes image features of the plurality of target images. Alternatively, the computer device clusters the image features of the plurality of target images to obtain a plurality of clusters, takes the mean of the image features in each cluster as the image feature corresponding to the cluster, and the non-abnormal feature includes the image features corresponding to the plurality of clusters. Alternatively, the computer device determines the mean of the image features of the plurality of target images to obtain the non-abnormal feature.

[0157] Therefore, the non-abnormal feature can represent the feature distribution of the target object image without abnormality. Optionally, the non-abnormal feature is stored in a memory, which directly and discretely represents the feature distribution of the target object image without abnormality.

[0158] 606、The computer device performs abnormality detection on the image containing the target object image based on the non-abnormal feature.

[0159] In some embodiments, the computer device determines that the target object in the image has abnormality when the distance between the image feature of the image containing the target object image and the non-abnormal feature is greater than a distance threshold; and determines that the target object image in the image does not have abnormality when the distance between the image feature of the image containing the target object image and the non-abnormal feature is not greater than the distance threshold. The distance can be a cosine distance.

[0160] In some embodiments, the non-abnormal feature includes the image features of the plurality of target images or the image features corresponding to the plurality of clusters, and the target image feature closest to the image feature of the image is determined from the plurality of image features included in the non-abnormal feature, and the target object image in the image is determined to not have abnormality when the distance is not greater than a distance threshold; and the target object in the image is determined to have abnormality when the distance is greater than the distance threshold.

[0161] The distance threshold can be set and changed as needed. The distance threshold can be determined according to the yield rate requirement of the product in actual production. Alternatively, a small amount of verification data (images of target object images without abnormality) can be collected, input into the abnormality detection network, and the abnormality score (i.e., distance) of each verification data is calculated. Then the abnormality scores are sorted in ascending order. Assuming that the yield rate in actual demand is m%, the abnormality score at the m% position in the sorted abnormality scores is taken as the distance threshold. In the abnormality detection, the distance (such as the cosine distance) between the image feature of the image to be detected and the non-abnormal feature can be used as the abnormality score. By collecting a small amount of verification data, inputting the verification data into the abnormality detection network, calculating the abnormality score of each verification data, and then sorting the abnormality scores in ascending order, the distance threshold can be determined according to the yield rate in actual demand.

[0162] In the embodiment of the present application, the abnormal information of the image is determined by the abnormality detection model, and the abnormality detection model can accurately detect the image in which the target object exists abnormally. Therefore, the abnormality detection model can detect the image in which the target object exists abnormally from the multiple images, and retain the normal image in which the target object image does not exist abnormally, thereby improving the accuracy of abnormality detection. Furthermore, the accuracy of the non-abnormal features of the normal object established based on these normal images is high. Accordingly, abnormality detection based on the non-abnormal features can detect images that do not match the non-abnormal features, and the target objects included in these images are abnormal objects, thereby further improving the accuracy of abnormality detection.

[0163] The above embodiment takes the image including the entire target object image as the detection object of the abnormality detection network. In other embodiments, the abnormality detection network takes a tile containing a local image of the target object as the detection object, and the abnormal sample refers to a tile sample including a local abnormal tile, and the normal sample refers to a tile sample including a local normal tile. Specifically, the abnormality detection model screens the abnormal sample from a large number of tiles, and the remaining normal samples are used to construct the abnormality detection network. Specifically, the non-abnormal features of the sample object image are constructed based on the normal samples, and the non-abnormal features are the feature distribution of the local image of the sample object in the normal sample. Furthermore, the abnormality detection network detects the abnormal target object local image based on the non-abnormal features of the target object image, that is, detects the local image that is far away from the non-abnormal features, and such local image is an abnormal local image. After detecting the abnormal local image, the abnormal target object is also detected.

[0164] In the embodiment of the present application, the abnormality detection model is used to determine the abnormal information of the image, and the abnormality detection model can accurately detect the image in which the target object exists abnormally. Therefore, the abnormality detection model can detect the image in which the target object exists abnormally from the multiple images, and retain the normal image in which the target object image does not exist abnormally, thereby improving the accuracy of abnormality detection. Furthermore, the accuracy of the non-abnormal features of the normal object established based on these normal images is high. Accordingly, abnormality detection based on the non-abnormal features can detect images that do not match the non-abnormal features, and the target objects included in these images are abnormal objects, thereby further improving the accuracy of abnormality detection.

[0165] In the embodiment of the present application, the abnormality detection model is used to determine the abnormal information of the image, and the abnormality detection model can accurately detect the image in which the target object exists abnormally. Therefore, the abnormality detection model can detect the image in which the target object exists abnormally from the multiple images, and retain the normal image in which the target object image does not exist abnormally, thereby improving the accuracy of abnormality detection. Furthermore, the accuracy of the non-abnormal features of the normal object established based on these normal images is high. Accordingly, abnormality detection based on the non-abnormal features can detect images that do not match the non-abnormal features, and the target objects included in these images are abnormal objects, thereby further improving the accuracy of abnormality detection.

[0166] Optionally, the computer device respectively performs feature extraction on the plurality of target patches to obtain image features of the plurality of target patches respectively, and obtains the non-anomalous feature of the target item local image based on the image features of the plurality of target patches respectively. In a case where the computer device performs feature extraction through the anomaly detection model, the image features extracted in step 602 can be directly obtained since the image features of the target patches have been extracted when the target text is determined. Alternatively, the computer device performs feature extraction through other feature extraction networks, such as a ResNet50 network.

[0167] The non-anomalous feature of the target item local image includes the image features of the plurality of target patches. Alternatively, the computer device clusters the image features of the plurality of target patches to obtain a plurality of clusters, takes the mean of the plurality of image features in each cluster as the image feature corresponding to the cluster, and the non-anomalous feature includes the image features corresponding to the plurality of clusters respectively. Alternatively, the computer device determines the mean of the patch features of the plurality of target patches to obtain the non-anomalous feature.

[0168] In some embodiments, the computer device uniformly processes the plurality of patches containing different target item local images of the target item image to obtain the non-anomalous feature of the target item local image. In other embodiments, for each local image of the target item image, the computer device determines the non-anomalous feature of the target item local image based on the plurality of target patches including the target item local image, and further obtains the non-anomalous features of the plurality of local images of the target item image respectively. Accordingly, the computer device performs anomaly detection on the patches containing the target item local image based on the non-anomalous feature of each target item local image.

[0169] The process of performing anomaly detection on the patches containing the target item local image based on the non-anomalous feature of the target item local image is the same as the process of performing anomaly detection on the image containing the target item image based on the non-anomalous feature of the target item image, which will not be described herein.

[0170] In the embodiments of the present application, the anomaly information of the image is determined through the anomaly detection model, and the anomaly detection model can accurately detect the target item local image to have an anomalous patch, and then the anomaly detection model can detect the patch having a local anomaly from the plurality of patches, and retain the normal patch having no local anomaly, thereby achieving accurate detection of the anomalous patch, and further achieving high accuracy of the non-anomalous feature of the normal target item local image established based on the normal patch. Accordingly, the anomaly detection based on the non-anomalous feature can accurately detect the patch that does not match the non-anomalous feature, and further the patch includes a local having an anomaly, i.e., achieving accuracy of anomaly detection.

[0171] When the anomaly detection network constructs a non-anomalous feature of a target object image, the training data used by default are normal samples, which refer to images of target objects that do not include any anomalies. Therefore, for the calibrated image, denoising is performed first, that is, the abnormal samples are detected and removed, and then the remaining samples are used as training data to train the anomaly detection network. In the embodiments of the present application, the method provided in the embodiments of the present application can accurately detect abnormal samples, so that the training data used by the anomaly detection network are all normal samples, thereby improving the accuracy of the trained anomaly detection network, and the accuracy of anomaly detection based on the anomaly detection network is high.

[0172] In the present application, denoising refers to a process of detecting abnormal samples from a large number of sample images or tiles by a specific model (such as an anomaly detection model or a large model) and removing them, so as to ensure that the samples used for training the anomaly detection network are all normal samples, thereby improving the accuracy of the trained anomaly detection network.

[0173] In the embodiments of the present application, anomaly detection is performed by using the anomaly detection model obtained by the training method as an example, and in other embodiments, the computer device can also directly use a basic model that has been trained on a large-scale data set, such as a CLIP (Contrastive Language-Image Pre-Training) model, and models improved based on CLIP, such as an APRIL-GAN model and an Anomaly CLIP model, without specific limitation. Such a large model has good zero-shot transfer ability by being pre-trained on a large-scale, multi-modal data set, and thus denoising (that is, detecting and removing abnormal samples) using such a model can avoid a series of problems caused by clustering. Therefore, the method provided in the embodiments of the present application realizes a fully automatic anomaly detection framework based on a basic large model, which utilizes the general ability of the large model and does not rely on specific assumptions and pre-training of a large number of normal items, realizes a more reliable denoising process, effectively improves the denoising performance of the overall framework, and thus realizes a more efficient, stable and universal fully automatic industrial anomaly detection process.

[0174] A normal item refers to an item in actual production, the characteristics of which conform to the distribution characteristics of normal items, and there is no anomaly such as a rough surface, scratches, insufficient size, etc. Correspondingly, an abnormal item refers to an item in actual production, the characteristics of which are different from the distribution characteristics of normal items. In an industrial production scenario, items with a rough surface, scratches, and sizes that do not meet the requirements are all abnormal items.

[0175] The method provided by the embodiments of the present application does not need to update parameters when constructing the anomaly detection network, so that the time cost of the training stage is extremely low, and the efficiency is improved. In other embodiments, the computer device can also construct the anomaly detection network by using PatchCore (an image processing algorithm based on a convolutional neural network), UniAD (an anomaly detection algorithm), SimpleNet (an anomaly detection algorithm), and the like, which will not be described herein.

[0176] The method provided by the embodiments of the present application can be applied to automatic industrial quality detection, can accurately detect abnormal defects in industrial parts, greatly reduces the labor cost of industrial quality inspection, and is based on the anomaly detection module for denoising, and is more efficient, stable and universal.

[0177] For example, referring to FIG. 7, FIG. 7 is a flowchart of industrial anomaly detection provided by the embodiments of the present application. The method provided by the embodiments of the present application is applied to the process of full-automatic industrial anomaly detection, which includes a training stage and a test stage. In the training stage, the anomaly detection model is trained based on the parts produced by the factory. First, the sample image including the part is aligned with the template image, which means that the position of the sample part image in the sample image is calibrated based on the position of the sample part image in the template image. Then, the sample images are trained to obtain the anomaly detection model. The anomaly detection model is then used for denoising, and then an anomaly detection network is constructed based on the remaining normal samples, which includes non-anomalous features of the parts. In the test stage and the actual anomaly detection process, the parts produced by the factory are detected by the anomaly detection network to detect abnormal parts that are greatly different from the non-anomalous features.

[0178] For example, referring to FIG. 8, FIG. 8 is a flowchart of the training stage in industrial detection provided by the embodiments of the present application. The computer device obtains the input automatically collected image, performs image alignment, and then performs denoising based on the anomaly detection model, that is, removes abnormal samples, and then constructs an anomaly detection network based on the remaining normal samples. For another example, referring to FIG. 9, FIG. 9 is a flowchart of the test stage in industrial detection provided by the embodiments of the present application. The computer device obtains the input test image, performs image alignment, and then performs anomaly detection by using the anomaly detection network, and outputs the result of the anomaly detection.

[0179] It should be noted that the above embodiment takes the actual anomaly detection by the anomaly detection network as an example for description, and the computer device can also directly perform the actual anomaly detection based on the anomaly detection model. Alternatively, the actual anomaly detection is performed by the anomaly detection network and the anomaly detection model respectively, and the detection results of the two are integrated to obtain the target detection result. For example, any one of the two is detected as an abnormal article; or the article detected by both is an abnormal article.

[0180] The embodiment of the present application provides an article anomaly detection method, which is based on an anomaly detection model for detection. The anomaly detection model is trained based on sample patches and sample texts used to describe the abnormality of sample article local images in the sample patches. Since the anomaly detection model extracts image features and text features, and the pair of image features and text features should have high similarity, the anomaly detection model is trained based on the similarity between the image features and the text features and the preset similarity, so that the anomaly detection model learns the rule that the pair of image features and text features has high similarity, and then the trained anomaly detection model can accurately extract features. Since the trained anomaly detection model can accurately extract image features of the patch and accurately extract text features of the text, the similarity between the text features of the text used to describe the abnormality of the target article local image in the patch and the image features is high, and then a plurality of text features are preset, and the similarity between the plurality of text features and the image features is compared respectively. The text feature corresponding to the text that meets the preset requirement is the text used to describe the abnormality of the target article local image in the patch, that is, through the anomaly detection model, the target article local image of the abnormality of the target article image in the patch is quickly and accurately detected, and on this basis, the overall abnormality of the target article image in the image can be obtained based on the abnormality of each patch corresponding to the image, thereby improving the convenience and accuracy of anomaly detection.

[0181] FIG. 10 is a block diagram of an anomaly detection model training device according to an embodiment of the present application. Referring to FIG. 10, the device includes:

[0182] The acquisition module 1001 is configured to acquire a plurality of first sample images, generate a plurality of sample pairs from the plurality of first sample images, each first sample image containing a sample article image, each sample pair including a sample patch and at least one sample text, the sample patch containing a sample article local image of the sample article image, and the sample text being used to describe the abnormality of the sample article local image in the sample patch.

[0183] The extraction module 1002 is configured to, for each sample pair, perform feature extraction on the sample patches and the sample text in the sample pair respectively by using the anomaly detection model to obtain image features of the sample patches and text features of the sample text, and determine the similarity between the image features and the text features.

[0184] The training module 1003 is configured to train the anomaly detection model iteratively based on the similarity of each sample pair and a preset similarity.

[0185] In some embodiments, the acquisition module 1001 is configured to:

[0186] For each first sample image, the first sample image is segmented by using a plurality of sliding windows of different sizes to obtain a plurality of patch sets of different sizes, and each patch set of a size includes a plurality of sample patches of the size.

[0187] Based on the sample patches in the patch set of each first sample image and the sample text corresponding to each sample patch, a plurality of sample pairs are obtained.

[0188] In some embodiments, each sample patch corresponds to a plurality of sample texts, and the plurality of sample texts respectively describe the abnormality of the local image of the target object in the sample patch by using different text descriptions; the extraction module 1002 is configured to:

[0189] The plurality of sample texts in the sample pair are respectively subjected to feature extraction by using the anomaly detection model to obtain initial text features corresponding to the plurality of sample texts respectively, and the mean of the plurality of initial text features is determined to obtain the text features.

[0190] In some embodiments, the device further comprises:

[0191] The filling module is configured to, for each sample patch, fill a plurality of text templates based on the local image of the target object in the sample patch and abnormal information of the local image of the target object to obtain a plurality of sample texts, the abnormal information is used to describe the abnormality of the local image of the target object, and the plurality of text templates are different.

[0192] In some embodiments, the acquisition module 1001 is further configured to acquire a plurality of second sample images, and each second sample image includes a sample object image.

[0193] The device further comprises a segmentation module configured to, for each second sample image, segment the sample object image from the second sample image to obtain a first sample image.

[0194] In some embodiments, the acquisition module 1001 is further configured to acquire a plurality of third sample images, and each third sample image includes a sample object image.

[0195] The device further comprises a calibration module configured to, for each third sample image, calibrate a position of a sample item image in the third sample image based on a position of the sample item image in the template image, to obtain a first sample image, wherein the position of the sample item image in the first sample image matches the position of the sample item image in the template image.

[0196] The embodiment of the present application provides a training device of an anomaly detection model. The device trains the anomaly detection model based on a sample image block and a sample text used for describing an anomaly of a sample item local image in the sample image block. Since the anomaly detection model extracts image features and text features, and the pair of image features and text features should have high similarity, the anomaly detection model is trained based on the similarity between the image features and the text features and a preset similarity, so that the anomaly detection model learns the rule that the pair of image features and text features have high similarity, and the trained anomaly detection model can accurately extract features. Thus, for any image block containing a target item local image, the text feature with high similarity to the image features of the image block can be determined based on the extracted image features, and the text corresponding to the text feature is the text used for describing the anomaly of the target item local image in the image block. Therefore, the accuracy of the anomaly detection model trained by the device in detecting the anomaly of the target item local image in the image block is high. Moreover, since the device is trained based on the image block containing the target item local image, the anomaly detection model can specifically detect which target item local image has an anomaly, that is, the accuracy of anomaly detection is improved. Therefore, anomaly detection based on the trained anomaly detection model can improve the accuracy and precision of anomaly detection.

[0197] FIG. 11 is a block diagram of an article anomaly detection device according to an embodiment of the present application. Referring to FIG. 11, the device comprises:

[0198] The acquisition module 1101 is configured to acquire a plurality of image blocks of an image, wherein the image contains a target item image, and each image block contains a target item local image of the target item image.

[0199] The determination module 1102 is configured to, for each image block, determine a target text corresponding to the image block by using an anomaly detection model, wherein the anomaly detection model is obtained by using the training method, and the target text is used for describing an anomaly of the target item local image in the image block.

[0200] The determination module 1102 is further configured to determine anomaly information of the target item image in the image based on the target texts corresponding to the plurality of image blocks respectively, wherein the anomaly information is used for describing an anomaly of the target item image.

[0201] In some embodiments, the determination module 1102 is configured to:

[0202] In a case where the target text corresponding to at least one of the plurality of tiles indicates that the local image of the target object exists abnormally, it is determined that the target object in the image exists abnormally and the local of the target object where the abnormality exists is determined.

[0203] In a case where the target text corresponding to each of the plurality of tiles indicates that the local image of the target object does not exist abnormally, it is determined that the image of the target object in the image does not exist abnormally.

[0204] In some embodiments, the determining module 1102 is configured to:

[0205] For each tile, the image feature of the tile is extracted by using the abnormality detection model, the similarity between the image feature and each of the plurality of preset text features is determined, the target text feature satisfying a preset requirement is determined from the plurality of preset text features, the plurality of preset text features respectively correspond to preset texts, and the target text is the preset text corresponding to the target text feature.

[0206] In some embodiments, the obtaining module 1101 is configured to:

[0207] The image is segmented by using a plurality of sliding windows of different sizes respectively to obtain a plurality of tile sets of different sizes, and each size of the tile set has a plurality of tiles of the size.

[0208] In some embodiments, each tile includes a plurality of pixel points, and the determining module 1102 is configured to:

[0209] For each tile of each size, in a case where the target text corresponding to the tile indicates that the local image of the target object in the tile exists abnormally, the similarity between the image feature of the tile and the corresponding target text feature is assigned to the plurality of pixel points in the tile.

[0210] For each pixel point, based on the similarity of the pixel point under the plurality of sizes, an abnormal value of the pixel point is obtained, and the abnormal value is used to indicate the probability that the pixel point exists abnormally.

[0211] Based on the abnormal value of each pixel point in the image, abnormal information of the image of the target object in the image is determined, and the abnormal information of the image of the target object includes at least one of the position of the pixel point where the image of the target object exists abnormally and the abnormal area.

[0212] In some embodiments, the image is multiple, and the determining module 1102 is further configured to determine a plurality of target images in the plurality of images based on the respective abnormal information of the target objects in the plurality of images, the image of the target object in the target image does not exist abnormally, determine a non-abnormal feature of the image of the target object based on the plurality of target images, and the non-abnormal feature is a feature possessed by the target object that does not exist abnormally.

[0213] The device further includes a first detection module configured to perform abnormality detection on the image containing the target object image based on the non-abnormal feature.

[0214] In some embodiments, the image is a plurality of images, and the determination module 1102 is further configured to determine a plurality of target patches in the plurality of patches based on the target text corresponding to each of the plurality of patches, the target object partial image in the target patch being non-abnormal; and determine the non-abnormal feature of the target object partial image based on the plurality of target patches in the plurality of images, the non-abnormal feature being a feature possessed by the target object partial image in the target object image that is non-abnormal.

[0215] The device further includes a second detection module configured to perform abnormality detection on the patch containing the target object partial image based on the non-abnormal feature.

[0216] The embodiments of the present application provide an object abnormality detection device. The device performs detection based on an abnormality detection model. The abnormality detection model is trained based on sample patches and sample texts describing abnormality of sample object partial images in the sample patches. Since the abnormality detection model extracts image features and text features, and the pair of image features and text features should have a high similarity, the abnormality detection model is trained based on the similarity between the image features and the text features and a preset similarity, so that the abnormality detection model learns the rule that the pair of image features and text features have a high similarity, and the trained abnormality detection model can accurately extract features. Thus, for any patch containing a target object partial image, a text feature with a high similarity to the patch can be determined based on the extracted image features of the patch, and the text corresponding to the text feature is the text describing the abnormality of the target object partial image in the patch. Therefore, the accuracy of the abnormality detection model trained by the device in performing abnormality detection on the target object partial image in the patch is high, and the abnormality of the target object image in the image can be obtained based on the abnormality of each patch in the image, thereby improving the convenience and accuracy of abnormality detection.

[0217] In the embodiments of the present application, the computer device can be a terminal or a server. When the computer device is a terminal, the terminal serves as an execution subject to implement the technical solutions provided by the embodiments of the present application. When the computer device is a server, the server serves as an execution subject to implement the technical solutions provided by the embodiments of the present application. Alternatively, the technical solutions provided by the present application are implemented through interaction between the terminal and the server, which is not limited in the embodiments of the present application.

[0218] In summary, the training method and device of the above abnormality detection model, the computer equipment, the computer readable storage medium and the computer program product, the computer equipment obtains a plurality of first sample images, and the images contain sample object images. Based on the first sample images, a plurality of sample pairs are generated, each sample pair contains a sample patch and at least one sample text, the sample patch contains a local image of the sample object image, and the sample text describes the abnormality of the local image. Then, for each sample pair, the image features of the sample patch and the text features of the sample text are extracted by the abnormality detection model respectively, and the similarity between the two is determined. Finally, according to the similarity of the plurality of sample pairs and the preset similarity, the abnormality detection model is iteratively trained. This method uses the feature similarity between the sample patch and the sample text to train the model, so that the model learns the similarity between the two, improves the accuracy and precision of the abnormality detection of the target object local image in the patch.

[0219] When generating sample pairs, for each first sample image, a plurality of sliding windows of different sizes are used to segment respectively, obtaining a plurality of patch sets of different sizes, each patch set containing a plurality of sample patches of the corresponding size. Then, based on the sample patches and the respective sample texts, a plurality of sample pairs are obtained. In this way, the diversity of the samples can be improved, the defects of different sizes of the sample objects can be captured, and the generalization ability and accuracy of the abnormality detection model can be improved.

[0220] When each sample patch corresponds to a plurality of sample texts, and the sample texts describe the abnormality of the local image of the sample object in different texts, the initial text features of the plurality of sample texts are extracted by the abnormality detection model respectively, and the text features are obtained by taking the mean value. This makes the text features more accurate and effective, and further optimizes the model training.

[0221] For each sample patch, based on the local image of the sample object and its abnormal information, a plurality of different text templates are filled to obtain a plurality of sample texts. This method uses a general text template, and the sample texts can be obtained by filling based on the abnormality, improving the convenience of obtaining the sample texts.

[0222] A plurality of second sample images are obtained, which contain sample object images and backgrounds. The sample object images are segmented from each second sample image to obtain the first sample images, avoiding background interference and facilitating subsequent image processing.

[0223] A plurality of third sample images are obtained, and based on the position of the sample object image in the template image, the position of the sample object image in the third sample image is calibrated to obtain the first sample image, so that the position of the sample object image in the first sample image matches the template image, facilitating accurate extraction of the region features of the sample object image.

[0224] The above abnormality detection method, device, computer equipment, computer readable storage medium and computer program product, the computer equipment obtains a plurality of tiles of an image containing a target object image, each tile containing a local image of the target object. The abnormality detection model obtained by the above training method is used to determine the target text corresponding to each tile, which describes the abnormality of the local image of the target object in the tile. Finally, based on the target texts corresponding to the plurality of tiles, the abnormality information of the target object image is determined. This method determines the abnormality of the target object image based on the trained abnormality detection model through the text corresponding to the tile, improving the convenience and accuracy of abnormality detection.

[0225] If the target text corresponding to at least one of the plurality of tiles indicates that the local image of the target object exists abnormality, it is determined that the target object in the image exists abnormality and abnormality part; if the target text corresponding to each tile indicates that the local image of the target object does not exist abnormality, it is determined that the target object image does not exist abnormality. In this way, whether the target object as a whole is abnormal and the abnormal position can be accurately determined.

[0226] For each tile, the image features are extracted by the abnormality detection model, the similarity of the features and a plurality of preset text features is determined, the target text feature meeting the preset requirement is determined from the similarity, and then the target text is obtained. This way uses the comparison between the preset text feature and the image feature to quickly and accurately determine the target text corresponding to the tile.

[0227] The image containing the target object image is segmented by using a plurality of size sliding windows to obtain a plurality of tile sets of different sizes, each tile set containing a plurality of tiles of a corresponding size. In this way, the tiles obtained can contain different local information of the target object, which is beneficial to more comprehensive detection of abnormality.

[0228] When the tile contains a plurality of pixel points, if the target text corresponding to the tile indicates that the local image of the target object exists abnormality, the similarity of the tile image features and the target text features is assigned to the pixel points of the tile. Based on the similarity of the pixel points under a plurality of sizes, an abnormal value is obtained to determine the abnormality information of the target object image, including the positions of abnormal pixel points and the abnormal area, etc., so that the abnormality information is more accurate.

[0229] For a plurality of images, based on the abnormality information, target images in which the target object image does not exist abnormality are determined, and then based on these target images, non-abnormal features of the target object image are determined, and finally the image containing the target object image is subjected to abnormality detection according to the non-abnormal features. By screening abnormal samples, a more accurate abnormality detection network is constructed, and the accuracy of abnormality detection is improved.

[0230] Based on the target text corresponding to the plurality of tiles, a target tile without an abnormality of the target local image of the target object is determined, and then a non-abnormal feature of the target local image of the target object is determined based on the target tile of the plurality of images, and finally, the tiles containing the target local image of the target object are detected for abnormalities according to the non-abnormal feature. The abnormality detection network is further optimized from the tile level, and the detection accuracy is improved.

[0231] FIG. 12 shows a structural block diagram of a terminal 1200 according to an example embodiment of the present application.

[0232] Generally, the terminal 1200 includes a processor 1201 and a memory 1202.

[0233] The processor 1201 can include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 1201 can be implemented in at least one of a hardware form of a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), and a PLA (Programmable Logic Array). The processor 1201 can also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a CPU (Central Processing Unit). The coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 can be integrated with a GPU (Graphics Processing Unit) for rendering and drawing content to be displayed on a display screen. In some embodiments, the processor 1201 can also include an AI (Artificial Intelligence) processor for processing machine learning-related computing operations.

[0234] The memory 1202 can include one or more computer-readable storage media, which can be non-transitory. The memory 1202 can also include a high-speed random access memory and a non-volatile memory such as one or more disk storage devices, flash storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one program code for being executed by the processor 1201 to implement the training method of the abnormality detection model or the object abnormality detection method provided by the method embodiments of the present application.

[0235] In some embodiments, the terminal 1200 can further optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 can be connected through a bus or a signal line. Each peripheral device can be connected to the peripheral device interface 1203 through a bus, a signal line or a circuit board. Specifically, the peripheral device includes at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207 and a power supply 1208.

[0236] The peripheral device interface 1203 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202 and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202 and the peripheral device interface 1203 can be implemented on a separate chip or circuit board, and the present embodiments are not limited thereto.

[0237] The radio frequency circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with a communication network and other communication devices through electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a subscriber identity module card, and the like. The radio frequency circuit 1204 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to the World Wide Web, a metropolitan area network, an intranet, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a WiFi (Wireless Fidelity) network. In some embodiments, the radio frequency circuit 1204 can further include NFC (Near Field Communication) related circuitry, and the present application is not limited thereto.

[0238] The display screen 1205 is configured to display a UI (User Interface). The UI can include graphics, text, icons, video, and any combination thereof. When the display screen 1205 is a touch display screen, the display screen 1205 is further configured to capture touch signals on or above the surface of the display screen 1205. The touch signals can be input to the processor 1201 as control signals for processing. In this case, the display screen 1205 can also be configured to provide virtual buttons and / or virtual keyboard, also known as soft buttons and / or soft keyboard. In some embodiments, the display screen 1205 can be one, disposed on the front panel of the terminal 1200; in other embodiments, the display screen 1205 can be at least two, respectively disposed on different surfaces of the terminal 1200 or in a folding design; in other embodiments, the display screen 1205 can be a flexible display screen, disposed on a curved surface or a folding surface of the terminal 1200. Even, the display screen 1205 can also be disposed in an irregular shape, i.e., a special-shaped screen. The display screen 1205 can be made of LCD (Liquid Crystal Display), OLED (Organic Light-Emitting Diode), etc.

[0239] The camera assembly 1206 is configured to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, the rear camera is at least two, which are any one of a main camera, a depth-of-field camera, a wide-angle camera, and a telephoto camera, to realize the background blur function by fusing the main camera and the depth-of-field camera, the panoramic shooting and VR (Virtual Reality) shooting function by fusing the main camera and the wide-angle camera, or other fusion shooting functions. In some embodiments, the camera assembly 1206 can further include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.

[0240] The audio circuit 1207 can include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into an electrical signal input to the processor 1201 for processing, or input to the radio frequency circuit 1204 to realize voice communication. For the purpose of stereo sound collection or noise reduction, the microphone can be multiple, respectively arranged at different parts of the terminal 1200. The microphone can also be an array microphone or an omnidirectional collection type microphone. The speaker is used to convert the electrical signal from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker can be a traditional diaphragm speaker, or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, not only can it convert electrical signals into sound waves that humans can hear, but it can also convert electrical signals into sound waves that humans cannot hear for ranging purposes. In some embodiments, the audio circuit 1207 can also include a headphone jack.

[0241] The power supply 1208 is used to supply power to each component in the terminal 1200. The power supply 1208 can be alternating current, direct current, disposable battery or rechargeable battery. When the power supply 1208 includes a rechargeable battery, the rechargeable battery can be a wired charging battery or a wireless charging battery. The wired charging battery is a battery charged through a wired line, and the wireless charging battery is a battery charged through a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0242] In some embodiments, the terminal 1200 further includes one or more sensors 1209. The one or more sensors 1209 include, but are not limited to, an acceleration sensor 1210, a gyroscope sensor 1211, a pressure sensor 1212, an optical sensor 1213, and a proximity sensor 1214.

[0243] The acceleration sensor 1210 can detect the acceleration magnitude in three coordinate axes of the coordinate system established by the terminal 1200. For example, the acceleration sensor 1210 can be used to detect the components of the gravitational acceleration in three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a landscape view or a portrait view according to the gravitational acceleration signal collected by the acceleration sensor 1210. The acceleration sensor 1210 can also be used for game or user motion data collection.

[0244] The gyroscope sensor 1211 can detect the body direction and rotation angle of the terminal 1200, and the gyroscope sensor 1211 can collect 3D actions of the user on the terminal 1200 in cooperation with the acceleration sensor 1210. The processor 1201 can realize the following functions according to the data collected by the gyroscope sensor 1211: motion sensing (such as changing the UI according to the user's tilt operation), image stabilization when shooting, game control, and inertial navigation.

[0245] The pressure sensor 1212 can be disposed at the side frame of the terminal 1200 and / or the lower layer of the display screen 1205. When the pressure sensor 1212 is disposed at the side frame of the terminal 1200, the holding signal of the user to the terminal 1200 can be detected, and the left-hand or right-hand recognition or the shortcut operation can be performed by the processor 1201 according to the holding signal collected by the pressure sensor 1212. When the pressure sensor 1212 is disposed at the lower layer of the display screen 1205, the operability control on the UI interface can be controlled by the processor 1201 according to the pressure operation of the user to the display screen 1205. The operability control includes at least one of the button control, the scroll bar control, the icon control, and the menu control.

[0246] The optical sensor 1213 is used to collect the ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 according to the ambient light intensity collected by the optical sensor 1213. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is decreased. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameter of the camera assembly 1206 according to the ambient light intensity collected by the optical sensor 1213.

[0247] The proximity sensor 1214, also called the distance sensor, is usually disposed at the front panel of the terminal 1200. The proximity sensor 1214 is used to collect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1214 detects that the distance between the user and the front of the terminal 1200 gradually decreases, the display screen 1205 is switched from the bright screen state to the off-screen state by the processor 1201; when the proximity sensor 1214 detects that the distance between the user and the front of the terminal 1200 gradually increases, the display screen 1205 is switched from the off-screen state to the bright screen state by the processor 1201.

[0248] Those skilled in the art can understand that the structure shown in FIG. 12 does not constitute a limitation on the terminal 1200, and can include more or fewer components than those shown, or combine certain components, or adopt different component arrangements.

[0249] FIG. 13 is a structural schematic diagram of a server according to an embodiment of the present application. The server 1300 can have a large difference in configuration or performance, and can include one or more processors (Central Processing Units, CPUs) 1301 and one or more memories 1302. The memory 1302 is configured to store executable program codes, and the processor 1301 is configured to execute the executable program codes to implement the training method of the anomaly detection model or the method for detecting an abnormal article according to any of the above-mentioned method embodiments. Of course, the server can also have a wired or wireless network interface, a keyboard, an input and output interface, and other components for implementing the functions of the device, and the details are not described herein.

[0250] The embodiments of the present application further provide a computer readable storage medium, and at least one program is stored in the computer readable storage medium. The at least one program is loaded and executed by a processor to implement the training method of the anomaly detection model or the method for detecting an abnormal article according to any of the above-mentioned implementation manners.

[0251] The embodiments of the present application further provide a computer program product, and the computer program product includes at least one program. The at least one program is stored in a computer readable storage medium. A processor of a computer device reads the at least one program from the computer readable storage medium. The processor executes the at least one program, so that the computer device executes the training method of the anomaly detection model or the method for detecting an abnormal article according to any of the above-mentioned implementation manners.

[0252] In some embodiments, the computer program product related to the embodiments of the present application can be deployed on one computer device for execution, or on multiple computer devices located in one place for execution, or on multiple computer devices distributed in multiple places and interconnected through a communication network for execution. The multiple computer devices distributed in multiple places and interconnected through a communication network can constitute a blockchain system.

[0253] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present disclosure.

[0254] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be noted that for those skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for training an anomaly detection model, performed by a computer device, comprising: Acquire a plurality of first sample images, and generate a plurality of sample pairs based on the plurality of first sample images, wherein each of the first sample images includes a sample item image, and each sample pair includes a sample image block and at least one sample text, wherein the sample image block includes a sample item partial image of the sample item image, and the sample text is used to describe an abnormality of the sample item partial image; For each sample pair, extract features from the sample image blocks in the sample pair using an anomaly detection model to obtain image features of the sample image blocks, and extract features from the sample text in the sample pair using the anomaly detection model to obtain text features of the sample text, and determine the similarity between the image features and the text features. The anomaly detection model is used to perform anomaly detection on the local image of the target object in the input image block; and The anomaly detection model is iteratively trained based on the similarities of the multiple groups of sample pairs and a preset similarity.

2. The method according to claim 1, generating a plurality of sample pairs according to the plurality of first sample images comprises: For each first sample image, segment the first sample image using sliding windows of multiple sizes to obtain tile sets of the multiple sizes, where a tile set of one size contains multiple sample tiles of the same size; The multiple groups of sample pairs are obtained based on the sample image blocks in the image block sets of the respective multiple first sample images and the sample text corresponding to each sample image block.

3. The method according to claim 1 or 2, wherein each sample image block corresponds to a plurality of sample texts, and the plurality of sample texts respectively describe abnormal conditions of the local image of the sample object in the sample image block in different texts; Extracting features from the sample text in the sample pair using the anomaly detection model to obtain text features of the sample text includes: The anomaly detection model is used to extract features of the multiple sample texts in the sample pair to obtain initial text features corresponding to the multiple sample texts, and the mean of the multiple initial text features is determined to obtain the text feature.

4. The method according to claim 3, further comprising: For each sample image block, multiple text templates are filled in based on the sample item local image in the sample image block and the abnormal information of the sample item local image to obtain the multiple sample texts. The abnormal information is used to describe the abnormal situation of the sample item local image, and the multiple text templates are different.

5. The method according to any one of claims 1 to 4, further comprising: Acquire a plurality of second sample images, each of the second sample images comprising the sample item image; For each second sample image, the sample item image is segmented from the second sample image to obtain the first sample image.

6. The method according to any one of claims 1 to 5, further comprising: Acquire a plurality of third sample images, each of the third sample images including the sample item image; For each third sample image, the position of the sample item image in the third sample image is calibrated based on the position of the sample item image in the template image to obtain the first sample image, and the position of the sample item image in the first sample image matches the position of the sample item image in the template image.

7. A method for detecting anomalies in an object, the method comprising: Acquire a plurality of image tiles, wherein the image includes a target object image, and each image tile includes a partial image of the target object in the target object image; For each image block, determining a target text corresponding to the image block using an anomaly detection model, wherein the anomaly detection model is obtained using the training method of any one of claims 1 to 6, and the target text is used to describe an abnormality of the local image of the target object in the image block; Based on the target texts respectively corresponding to the multiple image blocks, abnormal information of the target object image in the image is determined, where the abnormal information is used to describe the abnormal situation of the target object image.

8. The detection method according to claim 7, wherein determining abnormal information of the target object image in the image based on the target texts corresponding to the plurality of image blocks comprises: When the target text corresponding to at least one of the plurality of image blocks indicates that an abnormality exists in a partial image of the target object, determining that an abnormality exists in the target object in the image and determining the part of the target object where the abnormality exists; In a case where the target text corresponding to each of the plurality of image blocks indicates that there is no abnormality in the partial image of the target object, it is determined that there is no abnormality in the image of the target object in the image.

9. The detection method according to claim 7 or 8, wherein for each image block, determining the target text corresponding to the image block by using an anomaly detection model comprises: For each image block, the image features of the image block are extracted through the anomaly detection model, and the similarities between the image features and multiple preset text features are determined. A target text feature whose similarity meets preset requirements is determined from the multiple preset text features, and the multiple preset text features respectively correspond to preset texts, and the target text is the preset text corresponding to the target text feature.

10. The detection method according to claim 9, wherein the acquiring of the plurality of image blocks comprises: The image is segmented using sliding windows of multiple sizes to obtain tile sets of the multiple sizes, where the tile sets of each size include multiple tiles of the size.

11. The detection method according to claim 10, wherein each image block comprises a plurality of pixels, and determining abnormal information of the target object image in the image based on the target texts corresponding to the plurality of image blocks comprises: For each image block of each size, when the target text corresponding to the image block indicates that there is an abnormality in the local image of the target object in the image block, assigning similarities between the image features of the image block and the corresponding target text features to multiple pixels in the image block; For each pixel, based on the similarities of the pixel at the multiple sizes, obtaining an outlier value of the pixel, where the outlier value is used to indicate a probability that the pixel is outlier; Based on the abnormal value of each pixel point in the image, abnormal information of the target object image in the image is determined, where the abnormal information of the target object image includes at least one of a pixel point position where the target object has an abnormality and an abnormal area.

12. The detection method according to any one of claims 7 to 11, wherein the image is multiple, and the method further comprises: determining, based on the abnormality information of each target object in the plurality of images, a plurality of target images in the plurality of images, wherein the target object images in the target images do not have abnormalities; determining a non-abnormal feature of the target object image based on the multiple target images, wherein the non-abnormal feature is a feature possessed by the target object without any abnormality; Anomaly detection is performed on the image containing the target object image based on the non-abnormal features.

13. The detection method according to claim 7, wherein the image is multiple, and the method further comprises: Determining, based on the target texts respectively corresponding to the multiple image blocks, multiple target image blocks among the multiple image blocks, wherein the local images of the target objects in the target image blocks do not have any abnormalities; determining, based on the plurality of target image blocks in each of the plurality of images, a non-abnormal feature of the partial image of the target object, the non-abnormal feature being a feature of the partial image of the target object when the target object image does not have any abnormality; Anomaly detection is performed on the image block including the partial image of the target object based on the non-abnormal features.

14. A training device for an anomaly detection model, the device comprising: an acquisition module, configured to acquire a plurality of first sample images and generate a plurality of sample pairs based on the plurality of first sample images, wherein each first sample image includes a sample item image, and each sample pair includes a sample image block and at least one sample text, wherein the sample image block includes a sample item partial image of the sample item image, and the sample text is used to describe an abnormality of the sample item partial image; an extraction module configured to, for each sample pair, extract features from a sample image block in the sample pair using an anomaly detection model to obtain image features of the sample image block, extract features from a sample text in the sample pair using the anomaly detection model to obtain text features of the sample text, and determine a similarity between the image features and the text features. The anomaly detection model is configured to perform anomaly detection on a local image of a target object in an input image block; The training module is used to iteratively train the anomaly detection model based on the similarities of the multiple groups of sample pairs and the preset similarities.

15. An object abnormality detection device, comprising: an acquisition module, configured to acquire a plurality of image blocks, wherein the image includes an image of a target object, and each image block includes a partial image of the target object in the image of the target object; a determination module configured to determine, for each image block, a target text corresponding to the image block using an anomaly detection model obtained by the training method of any one of claims 1 to 6, the target text being used to describe an abnormality of the partial image of the target object in the image block; The determination module is further configured to determine abnormal information of the target object image in the image based on the target texts respectively corresponding to the multiple image blocks, where the abnormal information is used to describe an abnormal situation of the target object image.

16. A computer device comprising a processor and a memory, wherein the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the training method of the anomaly detection model described in any one of claims 1 to 6 or the object anomaly detection method described in any one of claims 7 to 13.

17. A computer-readable storage medium, wherein the computer-readable storage medium is used to store at least one program, wherein the at least one program is used to execute the training method of the anomaly detection model described in any one of claims 1 to 6 or the object anomaly detection method described in any one of claims 7 to 13.

18. A computer program product, comprising at least one program segment, wherein the at least one program segment is stored in a computer-readable storage medium, wherein a processor of a computer device reads the at least one program segment from the computer-readable storage medium, and the processor executes the at least one program segment, so that the computer device executes the anomaly detection model training method described in any one of claims 1 to 6 or the object anomaly detection method described in any one of claims 7 to 13.

Images