Product anomaly detection method and device, electronic equipment, storage medium and product

Abstract

The application provides an abnormality detection method and device of a product, an electronic device, a computer readable storage medium and a computer program product, including: obtaining a product image corresponding to a target product, the product image including at least two image regions, and the image contents of the at least two image regions being spliced to form the product image; taking the product image as a reference, respectively determining complementary image regions having a complementary relationship with each image region; for each image region, based on the image content of the corresponding complementary image region, reconstructing the image content of the image region to obtain the reconstructed image content corresponding to the image region; combining the reconstructed image content corresponding to each image region to obtain a combined image; obtaining the difference between the combined image and the product image, and determining an abnormality detection result corresponding to the target product based on the difference. In this way, the abnormality detection efficiency and accuracy of the product can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting anomalies in a product. Background Technology

[0002] In related technologies, anomaly detection for industrial products mostly relies on image reconstruction methods. These methods reconstruct the input image or input features and calculate the difference between the original input and the reconstructed result for anomaly detection. Alternatively, they utilize pre-trained feature and metric learning methods, extracting features through a fixed network and then performing distance measurements at the feature level for anomaly detection. However, image reconstruction methods require progressive upsampling to obtain the final reconstruction result, leading to a somewhat blurry result and limited anomaly detection accuracy. Pre-trained feature and metric learning methods require training a separate model for each type of industrial product and storing a large number of features, consuming significant memory during inference and resulting in low anomaly detection efficiency. Therefore, the anomaly detection methods for industrial products in these related technologies generally result in low efficiency and accuracy in anomaly detection. Summary of the Invention

[0003] This application provides a method, apparatus, electronic device, computer-readable storage medium, and computer program product for detecting product anomalies, which can improve the efficiency and accuracy of product anomaly detection.

[0004] The technical solution of this application embodiment is implemented as follows: This application provides an anomaly detection method for a product, the method comprising: Obtain a product image corresponding to the target product, wherein the product image includes at least two image regions, and the image contents of the at least two image regions are stitched together to form the product image; Using the product image as a reference, complementary image regions that have a complementary relationship with each of the image regions are determined respectively; For each of the image regions, based on the image content of the corresponding complementary image regions, the image content of the image region is reconstructed to obtain the reconstructed image content corresponding to the image region. The reconstructed image contents corresponding to each of the image regions are combined to obtain a combined image; The difference between the combined image and the product image is obtained, and the anomaly detection result corresponding to the target product is determined based on the difference.

[0005] This application provides an anomaly detection device for a product, the device comprising: The acquisition module is used to acquire a product image corresponding to the target product. The product image includes at least two image regions, and the image contents of the at least two image regions are stitched together to form the product image. The first determining module is used to determine, with reference to the product image, complementary image regions that have a complementary relationship with each of the image regions; The reconstruction module is used to reconstruct the image content of each image region based on the image content of the corresponding complementary image region, so as to obtain the reconstructed image content corresponding to the image region. The combination module is used to combine the reconstructed image content corresponding to each of the image regions to obtain a combined image; The second determining module is used to obtain the difference between the combined image and the product image, and to determine the anomaly detection result corresponding to the target product based on the difference.

[0006] This application provides an electronic device, including: Memory, used to store executable instructions; The processor, when executing executable instructions stored in the memory, implements the anomaly detection method of the product provided in the embodiments of this application.

[0007] This application provides a computer-readable storage medium storing computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor will execute the anomaly detection method of the product provided in this application.

[0008] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the anomaly detection method provided in this application.

[0009] The embodiments of this application have the following beneficial effects: First, based on at least two image regions included in the product image corresponding to the target product, complementary image regions with complementary relationships to each image region are identified. Then, based on the image content of the complementary image regions, the image content of the corresponding image regions is reconstructed. The reconstructed image content of each image region is then combined to obtain a combined image. Based on the differences between the combined image and the product image, the anomaly detection result corresponding to the target product is determined. In this way, compared to the process of directly reconstructing the image based on the overall features of the product image in related technologies, predicting the image content of image regions in the product image based on the image content of complementary image regions reduces the possibility of reconstructing abnormal locations during the reconstruction process, thus improving the accuracy of anomaly detection. Simultaneously, based on the image content of the complementary image regions of the product image, the corresponding canonical image (i.e., the combined image) of the product image is predicted. Therefore, based on the differences between the canonical image and the product image, the anomaly detection result corresponding to the target product is determined. This eliminates the need to train a separate model for each product category and store a large number of features, significantly reducing the cost of the anomaly detection process and improving the efficiency of anomaly detection. Attached Figure Description

[0010] Figure 1 This is a schematic diagram of the architecture of the anomaly detection system 100 provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a flowchart illustrating the anomaly detection method for the product provided in the embodiments of this application; Figure 4 This is a schematic diagram of a product image corresponding to a target product captured by a camera according to an embodiment of this application; Figure 5 This is a schematic diagram of a product image including at least two image regions provided in an embodiment of this application; Figure 6 This is a schematic diagram of the masking image provided in an embodiment of this application; Figure 7 This is a schematic diagram of a segmented image provided in an embodiment of this application; Figure 8 This is a flowchart of the training process of the image reconstruction model provided in the embodiments of this application; Figure 9 This is a schematic diagram of the reconstructed image provided in an embodiment of this application; Figure 10 This is a schematic diagram of the target stitched image provided in an embodiment of this application; Figure 11 This is a schematic diagram of the combined image, product image, and corresponding difference diagram provided in the embodiments of this application; Figure 12This is a schematic diagram of a product image, a mask for the product image, and a complementary mask provided in an embodiment of this application; Figure 13 This is a schematic diagram of the structure of the image reconstruction model provided in the embodiments of this application; Figure 14 This is a schematic diagram illustrating the final reconstructed image obtained through resampling, as provided in an embodiment of this application. Figure 15 This is a schematic diagram of the technical architecture of the industrial anomaly detection method based on diffusion model image restoration provided in the embodiments of this application. Detailed Implementation

[0011] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0012] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0013] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0014] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0015] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0016] 1) Client, also known as user terminal, refers to the program that provides local services to users in contrast to the server. Except for some applications that can only run locally, it is generally installed on ordinary client machines and needs to cooperate with the server to run. That is, there needs to be a corresponding server and service program in the network to provide the corresponding services. Thus, a specific communication connection needs to be established between the client and the server to ensure the normal operation of the application.

[0017] 2) The Diffusion Model is an image generation model that learns the information decay caused by noise and then uses the learned pattern to generate images. This concept also applies to latent variables because it attempts to learn the noise distribution rather than the data distribution. The noise distribution is modeled using the concept of a Markov chain, making it a probabilistic model. The diffusion model consists of two phases: a forward / diffusion process from the original image to the noise and a reverse process from the noise back to the original image.

[0018] 3) User interface (UI) is the medium through which the system and the user interact and exchange information, and is used to convert information from its internal form to a form that is acceptable to humans.

[0019] 4) Image restoration (Inpainting): Image restoration refers to the process of reconstructing lost or damaged parts of images and videos. For example, in museums, this work is often performed by experienced museum curators or art restorers. In the digital world, image restoration is also known as image interpolation or video interpolation, which refers to using complex algorithms to replace lost or damaged image data, mainly replacing small areas and flaws.

[0020] 5) KL divergence (Kullback-Leibler Divergence) is used to measure the distance between two probability distribution functions.

[0021] 6) Markov process, a stochastic process with Markov property. Markov property refers to the characteristic that, given the state of a process (or system) at time t0 is known, the conditional distribution of the state of the process at time t>t0 is independent of the state of the process at time t0. That is, the conditional probability of a Markov process is only related to the current state of the system, and is independent and unrelated to its past history or future state.

[0022] See Figure 1 , Figure 1This is a schematic diagram of the architecture of the product anomaly detection system 100 provided in this application embodiment. To realize the application scenario of product anomaly detection (for example, the application scenario of product anomaly detection can be that when performing defect detection on industrial products, a product image including at least two image regions corresponding to the industrial product to be detected is obtained, then complementary image regions with complementary relationships with each image region are determined, and then based on the image content of the corresponding complementary image regions, the standard product image corresponding to the industrial product to be detected is predicted, thereby determining the defect detection result used to indicate whether there is a defect in the industrial product and the location of the defect based on the difference between the standard product image and the product image), the terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two. The terminal 400 is used for users to use the client 401 and display on the display interface (display interface 401-1 is shown as an example). The terminal 400 and the server 200 are connected to each other through a wired or wireless network.

[0023] The server 200 is used to: acquire a product image corresponding to the target product, wherein the product image includes at least two image regions, and the image content of the at least two image regions is stitched together to form the product image; using the product image as a reference, acquire and determine complementary image regions that have a complementary relationship with each image region; for each image region, reconstruct the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region; combine the reconstructed image content corresponding to each image region to obtain a combined image; acquire the difference between the combined image and the product image, and determine the anomaly detection result corresponding to the target product based on the difference; and send the anomaly detection result corresponding to the target product to the terminal 400. Terminal 400 is also used to display the anomaly detection results corresponding to the target product based on the display interface.

[0024] In some embodiments, server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, set-top box, smart voice interaction device, smart home appliance, vehicle terminal, aircraft, and mobile devices (e.g., mobile phones, portable music players, personal digital assistants, dedicated messaging devices, portable gaming devices, smart speakers, and smartwatches), but is not limited to these. Terminal devices and servers can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0025] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. In practical applications, the electronic device can be... Figure 1 The server 200 or terminal 400 shown is an electronic device. Figure 1 Taking the terminal shown as an example, Figure 2 The illustrated electronic device includes at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in terminal 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general labeled all buses as Bus System 440.

[0026] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0027] User interface 430 includes one or more output devices 431 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0028] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 410.

[0029] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0030] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0031] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 453 is configured to enable the presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 associated with user interface 430 (e.g., a display screen, a speaker, etc.). The input processing module 454 is used to detect and translate one or more user inputs or interactions from one or more input devices 432.

[0032] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 An anomaly detection device 455 for a product stored in memory 450 is shown. This device can be software in the form of programs and plug-ins, and includes the following software modules: an acquisition module 4551, a first determination module 4552, a reconstruction module 4553, a combination module 4554, and a second determination module 4555. These modules are logically connected and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0033] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the anomaly detection apparatus of the product provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the anomaly detection method of the product provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0034] In some embodiments, the terminal or server can implement the anomaly detection method for the product provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), that is, a program that needs to be installed in the operating system to run, such as an instant messaging APP or a web browser APP; it can also be a mini-program, that is, a program that only needs to be downloaded into a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0035] Based on the above description of the anomaly detection system and electronic device provided in the embodiments of this application, the anomaly detection method of the product provided in the embodiments of this application is described below. In actual implementation, the anomaly detection method of the product provided in the embodiments of this application can be implemented by the terminal or the server alone, or by the terminal and the server working together, so that... Figure 1 The following description uses the example of server 200 executing the anomaly detection method of the product provided in this application embodiment alone. See also... Figure 3 , Figure 3 This is a flowchart illustrating the anomaly detection method for the product provided in this application embodiment. Below, it will be discussed in conjunction with... Figure 3 The steps shown are explained.

[0036] Step 101: The server obtains the product image corresponding to the target product. The product image includes at least two image regions, and the image content of the at least two image regions is stitched together to form the product image.

[0037] In practice, the product image corresponding to the target product can be pre-stored locally, obtained from external sources (such as the Internet), or it can be an image obtained by capturing images of the target product through an image acquisition device.

[0038] As an example, the target product could be an industrial product such as a nail or hammer, while the image acquisition device could be a camera. The camera could be a monocular camera, a binocular camera, a depth camera, or a 3D (3-Dimensional) camera. In practical implementation, the camera is activated to enter scanning mode, scanning the target product within the camera's field of view in real time and generating images at a specified frame rate, such as... Figure 4 As shown, Figure 4 This is a schematic diagram of a product image corresponding to a target product captured by a camera according to an embodiment of this application, based on... Figure 4 The target product here is Figure 4 The screw shown.

[0039] As another example, the image acquisition device can also be a radar device such as lidar or millimeter-wave radar. Lidar is a radar device that detects the position, velocity, attitude, shape, and other characteristic data of a target product by emitting a laser beam. Millimeter-wave radar is a radar device that detects in the millimeter-wave band. The radar device transmits detection signals to the target product in real time and receives the echo signals reflected back from the target product. Based on the differences between the detection signal and the echo signal, it determines the characteristic data of the target product. The radar device uses multiple transmitters and receivers, thereby acquiring a three-dimensional point cloud image.

[0040] It should be noted that the image acquisition device can be located on a mobile device, thereby enabling image acquisition based on the mobile device and anomaly detection of the product image corresponding to the acquired target product. In this way, the flexibility and low cost of the mobile device can greatly improve the usability and convenience of anomaly detection of the target product.

[0041] In practical implementation, the image region encompassed by the product image corresponding to the target product is used to indicate a portion of the product image itself. Each image region represents a different part of the product image, and the image content of each region is stitched together to form the product image. For example, see... Figure 5 , Figure 5 This is a schematic diagram of a product image including at least two image regions provided in an embodiment of this application, based on... Figure 5When a product image includes four image regions, these four image regions are A, B, C, and D, and the image content of these four image regions is stitched together to form the product image. When a product image includes two image regions, these two image regions can be regions formed by A and B and regions formed by C and D, or regions formed by A and C and regions formed by B and D, or regions formed by A and D and regions formed by B and C.

[0042] Step 102: Using the product image as a reference, obtain and determine the complementary image regions that have a complementary relationship with each image region.

[0043] It should be noted that the complementary relationship is used to indicate the combination relationship that can be spliced ​​to form a product image. That is, for image regions with a complementary relationship and complementary image regions, the image content of the image region and the image content of the complementary image region are spliced ​​to form a corresponding product image.

[0044] For example, see [link to previous article] Figure 5 When the product image includes four image regions, for image region A, the complementary image regions that have a complementary relationship are the regions formed by B, C, and D; for image region B, the complementary image regions that have a complementary relationship are the regions formed by A, C, and D; for image region C, the complementary image regions that have a complementary relationship are the regions formed by A, B, and D; and for image region D, the complementary image regions that have a complementary relationship are the regions formed by A, B, and C. When the product image includes two image regions, and these two image regions are respectively formed by A and B and C and D, the complementary image regions that have a complementary relationship are the regions formed by C and D. Complementary image regions are the regions formed by A and B. When a product image includes two image regions, namely regions formed by A and C and regions formed by B and D respectively, the complementary image regions that have a complementary relationship with the image regions formed by A and C are the regions formed by B and D, and the complementary image regions that have a complementary relationship with the image regions formed by B and D are the regions formed by A and C.

[0045] Step 103: For each image region, based on the image content of the corresponding complementary image region, reconstruct the image content of the image region to obtain the reconstructed image content corresponding to the image region.

[0046] In practical implementation, after identifying complementary image regions that are complementary to each image region, the process involves reconstructing the image content of each image region based on the image content of the corresponding complementary image regions. There are various ways to determine the image content of the corresponding complementary image regions, such as masking the product image or segmenting the product image. Therefore, the process of reconstructing the image content of each image region based on the image content of the corresponding complementary image regions also varies. The following explanation will use two methods of determining the image content of the corresponding complementary image regions as examples to illustrate the process of reconstructing the image content of each image region based on the image content of the corresponding complementary image regions.

[0047] In some embodiments, after determining complementary image regions that are complementary to each image region, the product image can be copied to obtain the same number of product images as the image regions; each product image is masked to obtain a masked image; wherein the masked region in the masked image corresponds to the image region, and different masked images correspond to different image regions; thus, the process of reconstructing the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region can be, based on the masked image, reconstructing the image content of the masked region to obtain the reconstructed image content corresponding to the image region.

[0048] It should be noted that the masking process here can involve masking or blurring corresponding areas in each product image. Furthermore, the masked or blurred areas will differ for different product images. For example, see [link to example]. Figure 6 , Figure 6 This is a schematic diagram of the masking image provided in an embodiment of this application. First, the product image is copied to obtain a product image with the same number of image areas, such as... Figure 6As shown, the product image here includes four image regions. Therefore, after copying the product image, four product images are obtained. For the product image including image region A and the complementary image regions formed by B, C, and D, the product image is masked to obtain the masked image shown in Figure 601. Here, the masked area in the masked image corresponds to image region A. For the product image including image region B and the complementary image regions formed by A, C, and D, the product image is masked to obtain the masked image shown in Figure 602. Here, the masked area in the masked image corresponds to image region B. For the product image including image region C and the complementary image regions formed by A, B, and D, the product image is masked to obtain the masked image shown in Figure 603. Here, the masked area in the masked image corresponds to image region C. For the product image including image region D and the complementary image regions formed by A, B, and C, the product image is masked to obtain the masked image shown in Figure 604. Here, the masked area in the masked image corresponds to image region D.

[0049] In practical implementation, the process of reconstructing the image content of the masked region based on the masked image to obtain the reconstructed image content corresponding to the image region specifically includes: obtaining an image reconstruction model; wherein the number of image reconstruction models is the same as the number of image regions, and there is a one-to-one correspondence between the image reconstruction models and the image regions; and reconstructing the image content of the masked region using the image reconstruction model to obtain the reconstructed image content corresponding to the image region. For example, see [link to example]. Figure 6 ,like Figure 6 As shown, the product image here includes 4 image regions. Therefore, the obtained image reconstruction model can also be 4. Thus, the image content of different masked regions can be reconstructed through the 4 image reconstruction models to obtain the corresponding reconstructed image content of image regions A, B, C, and D.

[0050] In other embodiments, after determining complementary image regions that are complementary to each image region, the product image is copied to obtain the same number of product images as the image regions; for each product image, the product image is segmented to obtain segmented images corresponding to the complementary image regions of the product image; wherein, different segmented images correspond to different image regions; thus, the process of reconstructing the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region can be, based on the segmented image, reconstructing the image content of the image region corresponding to the segmented image to obtain the reconstructed image content corresponding to the image region.

[0051] It should be noted that the segmented images obtained from segmentation will differ depending on the product image. For example, see [link to example]. Figure 7 , Figure 7This is a schematic diagram of a segmented image provided in an embodiment of this application. First, the product image is copied to obtain a product image with the same number of image regions, such as... Figure 7 As shown, the product image here includes four image regions. Therefore, after copying the product image, four product images are obtained. For the product image including image region A and the complementary image regions formed by B, C, and D, the product image is segmented to obtain the segmented image corresponding to the complementary image regions formed by B, C, and D of the product image, as shown in Figure 701. Here, the image region corresponding to the segmented image is A. For the product image including image region B and the complementary image regions formed by A, C, and D, the product image is segmented to obtain the complementary image regions formed by A, C, and D, as shown in Figure 702. The corresponding segmented image is shown in Figure 703, where the image region corresponding to the segmented image is B. For the product image including image region C and the complementary image region formed by A, B, and D, the product image is segmented to obtain the segmented image corresponding to the complementary image region formed by A, B, and D, as shown in Figure 704. The image region corresponding to the segmented image is D.

[0052] In practical implementation, the process of reconstructing the image content of the masked region based on the masked image to obtain the reconstructed image content corresponding to the image region specifically includes: obtaining an image reconstruction model; wherein the number of image reconstruction models is the same as the number of image regions, and there is a one-to-one correspondence between the image reconstruction models and the image regions; and reconstructing the image content of the image region corresponding to the segmented image using the image reconstruction model to obtain the reconstructed image content corresponding to the image region. For example, see [link to example]. Figure 7 ,like Figure 7 As shown, the product image here includes 4 image regions. Therefore, the obtained image reconstruction model can also be 4. Thus, the image content of the image regions corresponding to different segmented images can be reconstructed through the 4 image reconstruction models, and the corresponding reconstructed image content of image regions A, B, C, and D can be obtained.

[0053] In some embodiments, the image reconstruction model can be trained before obtaining the image reconstruction model, see [link to relevant documentation]. Figure 8 , Figure 8 This is a flowchart of the training process of the image reconstruction model provided in the embodiments of this application, based on Figure 8 Before step 101, the following steps may also be performed.

[0054] Step 201: Obtain the initial image reconstruction model and the product image sample corresponding to the normal product. The product image sample includes at least two training image regions, and the image content of at least two training image regions is stitched together to form the product image training sample.

[0055] It should be noted that "normal products" here refers to products without defects or abnormalities, such as resistors and screws without defects or abnormalities.

[0056] Step 202: Using the product image training samples as a reference, determine the complementary training image regions that have a complementary relationship with each training image region.

[0057] Step 203: Using the initial image reconstruction model, for each training image region, based on the image content of the corresponding complementary training image region, reconstruct the image content of the training image region to obtain the reconstructed image content corresponding to the training image region.

[0058] Step 204: Obtain the content of the reconstructed image and the difference between the content of the image and the corresponding training image region, and update the model parameters of the initial image reconstruction model based on the difference to obtain the image reconstruction model.

[0059] It should be noted that the image reconstruction model here can be the U-Net model, or any model with consistent input and output, such as an autoencoder, Transformer model, etc.

[0060] In some embodiments, the process of performing masking processing on each product image to obtain a masked image can also be based on virtual masks such as masks, where each product image is masked to obtain a masked image. Specifically, virtual masks for corresponding image regions are generated based on each image region of the product image and its corresponding complementary image region; and masking processing is performed on the product image based on each virtual mask to obtain a masked image.

[0061] In actual implementation, the process of generating virtual masks for corresponding image regions can also be as follows: determine the first target number of initial virtual masks for the corresponding product image, then divide each initial virtual mask to obtain a second target number of rectangular regions, and then mask the third target number of rectangular regions in the second target number of rectangular regions for each initial virtual mask to obtain at least one virtual mask; wherein, there is no overlap between the third target number of rectangular regions masked in different virtual masks.

[0062] It should be noted that the first target quantity is a natural number greater than 1, the second target quantity can be the ratio between the product of the height and width of the product image and the product of the height and width of the rectangular region, and the third target quantity can be the ratio of the second target quantity to the first target quantity. Here, the height and width of the rectangular region and the first target quantity of the product image can be preset. The first target quantity corresponds to the number of image reconstruction models, and all image reconstruction models have uniform model parameters. In addition, the virtual mask can be replaced with a mask of other shapes. For example, a more efficient virtual mask can be designed based on known defects and the shape of the object, or three or more complementary virtual masks can be generated. This application does not limit the scope of the embodiments.

[0063] In practice, the process of masking a product image based on virtual masks to obtain a masked image can be achieved by fusing the virtual masks with the product image to obtain a masked image. For example, for a virtual mask, the pixel values ​​of the masked portion (i.e., the invisible portion) are 0, and the pixel values ​​of the unmasked portion (i.e., the visible portion) are 1. Thus, the product image is fused with the virtual mask, such as by multiplying the pixels. In this way, the pixel values ​​of the unmasked portion (i.e., the visible portion) of the product image remain the original RGB values, while the RGB pixel values ​​of the masked portion (i.e., the obscured portion) are [0, 0, 0], meaning the information is discarded.

[0064] Step 104: Combine the reconstructed image content corresponding to each image region to obtain a combined image.

[0065] In practical implementation, the process involves reconstructing the image content of an image region based on its complementary image content, thus obtaining the reconstructed image content corresponding to that region. Here, the reconstructed image is the product image corresponding to the complementary image region. The reconstructed image content corresponding to each image region is then determined from the reconstructed image; that is, each reconstructed image content corresponds to one reconstructed image. The process of combining the reconstructed image content corresponding to each image region to obtain a combined image specifically includes: cropping the reconstructed image content from each reconstructed image based on its position; stitching the cropped images together to obtain a target stitched image; and finally, determining the target stitched image as the combined image. This combined image is also the predicted product image of a normal product corresponding to the target product, i.e., the product image when the target product is without abnormalities.

[0066] For example, see Figure 9 , Figure 9 This is a schematic diagram of the reconstructed image provided in the embodiments of this application, based on Figure 9When the image region is A and the corresponding complementary image regions are B, C, and D, the reconstructed image content of image region A is obtained based on the image content of complementary image regions B, C, and D. At this time, the reconstructed image of the corresponding image region A is shown in 901. Then, based on the position of the reconstructed image content of image region A in the reconstructed image shown in 901, the reconstructed image content of image region A is cropped from this reconstructed image to obtain the cropped image of image region A. Similarly, when the image region is B and the corresponding complementary image regions are A, C, and D, the reconstructed image content of image region B is obtained based on the image content of complementary image regions A, C, and D. At this time, the reconstructed image of the corresponding image region B is shown in 902. Then, based on the position of the reconstructed image content of image region B in the reconstructed image shown in 902, the reconstructed image content of image region B is cropped from this reconstructed image to obtain the cropped image of image region B. When the image region is C and the corresponding complementary image regions are A, B, and D, the reconstructed image content of image region C is obtained based on the image content of complementary image regions A, B, and D. At this time, the reconstructed image of the corresponding image region C is shown in 903. Then, based on the position of the reconstructed image content of image region C in the reconstructed image shown in 903, the reconstructed image content of image region C is cropped from the reconstructed image to obtain the cropped image of image region C. When the image region is D and the corresponding complementary image regions are A, B, and C, the reconstructed image content of image region D is obtained based on the image content of complementary image regions A, B, and C. At this time, the reconstructed image of the corresponding image region D is shown in 904. Then, based on the position of the reconstructed image content of image region D in the reconstructed image shown in 904, the reconstructed image content of image region D is cropped from the reconstructed image to obtain the cropped image of image region D.

[0067] In practical implementation, the process of stitching together the cropped images to obtain a target stitched image and then determining the target stitched image as the composite image involves, after obtaining each cropped image, stitching together multiple cropped images based on their positions in the reconstructed image to obtain the target stitched image, thereby determining the target stitched image as the composite image. For example, see [link to example]. Figure 9 See also Figure 10 , Figure 10 This is a schematic diagram of the target stitched image provided in the embodiments of this application, based on Figure 9 After obtaining cropped images of corresponding image regions A, B, C, and D, the cropped images are stitched together based on their positions in the corresponding reconstructed images to obtain the following result: Figure 10 The target stitched image is shown.

[0068] Step 105: Obtain the differences between the combined image and the product image, and determine the anomaly detection result corresponding to the target product based on the differences.

[0069] In practice, the process of obtaining the difference between the combined image and the product image specifically includes: obtaining the pixel difference between each first pixel in the combined image and the second pixel at the same position in the product image; determining the difference map between the combined image and the product image based on the pixel difference; and using the difference map as the difference between the combined image and the product image.

[0070] It should be noted that the process of obtaining the pixel difference between each first pixel in the composite image and the second pixel at the same position in the product image can be as follows: First, obtain the first color value of each first pixel in the red-green-blue channels and the second color value of each second pixel in the red-green-blue channels. For each of the red-green-blue channels, subtract the first color value from the second color value to obtain the first color value difference for the corresponding single red channel, the second color value difference for the corresponding single green channel, and the third color value difference for the corresponding single blue channel. Then, based on the first color value difference, the second color value difference, and the third color value difference between each first pixel and the corresponding second pixel in the red-green-blue channels, determine the pixel difference between each first pixel in the composite image and the second pixel at the same position in the product image. Therefore, the process of determining the difference map between the combined image and the product image based on the differences of each pixel can be as follows: Regularization is performed on the differences of each pixel to obtain the difference map between the combined image and the product image. Specifically, for each pixel difference, the sum of the squares of the differences in the first, second, and third color values ​​of the corresponding pixel is obtained, and then the square root of the sum of the squares is taken to obtain the pixel value (such as grayscale value) of each pixel. Thus, the difference map between the combined image and the product image is determined based on the pixel values ​​of each pixel. ...Formula (1); Where S is the difference diagram between the combined image and the product image. For product images, This is a composite image corresponding to the product image.

[0071] For example, see Figure 11 , Figure 11 This is a schematic diagram of the combined image, product image, and corresponding difference image provided in the embodiments of this application, based on... Figure 11 The image indicated by 1101 is a product image, the image indicated by 1102 is a combined image corresponding to the product image, and the image indicated by 1103 is a difference map between the product image and the combined image, thereby using the difference map as a representation of the difference between the combined image and the product image.

[0072] In actual implementation, the process of determining the anomaly detection result corresponding to the target product based on the difference specifically includes: when the difference is a difference map between the combined image and the product image, determining the anomaly detection result corresponding to the target product based on the difference map; wherein, the anomaly detection result includes at least one of a first anomaly detection result and a second anomaly detection result, the first anomaly detection result is used to indicate whether there is an anomaly in the target product, and the second anomaly detection result is used to indicate the location of the anomaly on the target product.

[0073] It should be noted that the first anomaly detection result, used to indicate whether a target product has an anomaly, is an image-level detection result, meaning it indicates whether an image contains an abnormal region. The second anomaly detection result, used to indicate the location of the anomaly on the target product, is a pixel-level detection result, meaning it locates the abnormal pixels. Because the first and second anomaly detection results indicate different things, the process of determining the corresponding first and second anomaly detection results for the target product based on the difference map also differs. Next, based on the differences between the first and second anomaly detection results, the process of determining the corresponding anomaly detection result for the target product based on the difference map will be explained.

[0074] In some embodiments, the anomaly detection result includes a first anomaly detection result. The process of determining the anomaly detection result corresponding to the target product based on the difference map may include: dividing the difference map into regions to obtain at least two sub-image regions; for each sub-image region, averaging the pixel values ​​of multiple pixels in the sub-image region to obtain an average pixel value for the sub-image region; selecting the maximum average pixel value from the average pixel values ​​of the multiple sub-image regions based on the magnitude of the average pixel value; and determining the first anomaly detection result based on the maximum average pixel value.

[0075] In practical implementation, the process of averaging the pixel values ​​of multiple pixels in each sub-image region to obtain the average pixel value for that sub-image region can be as follows: For each sub-image region, obtain the sum of the pixel values ​​(i.e., the sum of grayscale values) of multiple pixels in that sub-image region, then determine the number of pixels in that sub-image region, and obtain the ratio of the sum of pixel values ​​to the number of pixels. This ratio is then used as the average pixel value for that sub-image region. The process of determining the first anomaly detection result based on the maximum average pixel value can be as follows: Compare the maximum average pixel value with a pixel value threshold for the pixel region to obtain a comparison result. When the comparison result indicates that the maximum average pixel value is greater than the pixel value threshold for the pixel region, a first anomaly detection result indicating that the target product has an anomaly is determined. When the comparison result indicates that the maximum average pixel value is not greater than the pixel value threshold for the pixel region, a first anomaly detection result indicating that the target product does not have an anomaly is determined.

[0076] In other embodiments, the anomaly detection result includes a second anomaly detection result. The process of determining the anomaly detection result corresponding to the target product based on the difference map may be as follows: based on the difference map, obtain the pixel value of each pixel in the difference map; based on the magnitude of each pixel value, select at least one target pixel from multiple pixels in the difference map; and based on the position of the target pixel, determine the second anomaly detection result.

[0077] In practical implementation, the process of selecting at least one target pixel from multiple pixels in the difference map based on the size of each pixel value specifically includes performing the following processing for each pixel in the difference map: comparing the pixel value of the pixel with a pixel value threshold to obtain a comparison result; when the comparison result indicates that the pixel value of the pixel is greater than the pixel value threshold, the pixel is determined as a target pixel; and the process of determining a second anomaly detection result based on the position of the target pixel can be that the position of the target pixel is determined as the position of the anomaly on the target product, and a second anomaly detection result is determined based on this position to indicate the position of the anomaly on the target product.

[0078] In some embodiments, when the image reconstruction model is a diffusion model such as the U-Net model, the process of reconstructing the image content of an image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region can also be as follows: based on each image region and the corresponding complementary image region, a mask for the corresponding product image is determined, and a complementary mask with a complementary relationship to the mask is obtained using the product image as a reference; wherein, the mask is used to cover the image content of the image region in the product image, and the complementary mask is used to cover the image content of the complementary image region in the product image; thus, the process of reconstructing the image content of an image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region can be as follows: combining the mask, the complementary mask, and the image content of the corresponding complementary image region to reconstruct the image content of the image region to obtain the reconstructed image content corresponding to the image region.

[0079] It should be noted that the mask here can refer to a masking image corresponding to the corresponding product image. That is, the process of determining the mask for the corresponding product image based on each image region and its corresponding complementary image region, i.e., the process of masking the corresponding product image to obtain the masking image; and then, by combining the mask, the complementary mask, and the image content of the corresponding complementary image region, the image content of the image region is reconstructed to obtain the reconstructed image content corresponding to the image region. Here, the image content of the corresponding complementary image region is used to indicate the image content of the uncovered area. For example, see [link to example]. Figure 12 , Figure 12 This is a schematic diagram of a product image, a mask for the product image, and a complementary mask provided in an embodiment of this application, based on... Figure 12 Image 1201 indicates a product image, image 1202 indicates a mask corresponding to the product image, and image 1203 indicates a complementary mask that is complementary to the mask corresponding to the product image. Therefore, by combining the mask shown in 1202, the complementary mask shown in 1203, and the image content of the corresponding complementary image region, the image content of the image region is reconstructed, resulting in the reconstructed image content corresponding to the image region. Here, the image content of the corresponding complementary image region is the unpainted area in the mask shown in 1202.

[0080] In practical implementation, the process of reconstructing the image content of an image region by combining the image content of the mask, complementary mask, and corresponding complementary image regions, and obtaining the reconstructed image content corresponding to the image region, specifically includes: adding noise to the product image at least once to obtain a first noisy image, and obtaining a second noisy image corresponding to the product image, wherein the second noisy image is the image to be processed before the last noise addition process is performed on the product image; performing denoising processing on the first noisy image to obtain a denoised image; fusing the second noisy image with the mask to obtain a first fused image, and fusing the denoised image with the complementary mask to obtain a second fused image; stitching the first fused image and the second fused image to obtain a reconstructed image; and determining the reconstructed image content corresponding to the image region based on the reconstructed image and the image content of the corresponding complementary image regions.

[0081] It should be noted that the process of stitching the first fused image and the second fused image to obtain the reconstructed image specifically involves: stitching the first fused image and the second fused image to obtain the first stitched image; denoising the first stitched image to obtain the second denoised image; then obtaining the third noise image corresponding to the product image, which is the image to be processed before the penultimate noise addition process on the product image; fusing the third noise image with a mask to obtain the third fused image; and fusing the second denoised image with a complementary mask to obtain the fourth fused image; and finally, performing image stitching on the third fused image and the fourth fused image... The images are stitched together to obtain a second stitched image; then the second stitched image is denoised to obtain a third denoised image, and then a fourth noise image corresponding to the product image is obtained. The fourth noise image is the image to be processed before the third-to-last noise addition process is performed on the product image; the fourth noise image is fused with a mask to obtain a fifth fused image, and the third denoised image is fused with a complementary mask to obtain a sixth fused image; the fifth fused image and the sixth fused image are stitched together to obtain a third stitched image, and the above process is repeated until the number of denoising times is the same as the number of noise addition times, and the last stitched image is used as the reconstructed image.

[0082] For example, see Figure 13 , Figure 13 This is a schematic diagram of the structure of the image reconstruction model provided in the embodiments of this application, based on Figure 13Here, noise is added four times. The first noise addition to the product image results in a fourth noisy image. A second noise addition to the fourth noise image results in a third noise image. A third noise addition to the third noise image results in a second noise image. Finally, a fourth noise addition to the second noise image results in a first noise image. The first noise image is then denoised to obtain a denoised image. This denoised image is then fused with a complementary mask to obtain a second fused image. Simultaneously, the second noise image is fused with the mask to obtain a first fused image. The first and second fused images are then stitched together to obtain a first stitched image. The first stitched image is then denoised a second time to obtain a second denoised image. This second denoised image is then fused with a complementary mask to obtain a fourth fused image. Finally, the third noise image is fused with the mask to obtain a third fused image. The process involves several steps: first, denoising the third and fourth fused images to obtain a second fused image; then, denoising the second fused image a third time to obtain a third denoised image, which is then fused with a complementary mask to obtain a sixth fused image; simultaneously, denoising the fourth denoised image with the mask to obtain a fifth fused image, which is then fused with the sixth fused image to obtain a third fused image; finally, denoising the third fused image a fourth time to obtain a final denoised image, which is then fused with a complementary mask to obtain an eighth fused image; simultaneously, fusion the product image with the mask to obtain a seventh fused image, which is then fused with the seventh and eighth fused images to obtain the final fused image. Since the number of denoising steps is the same as the number of noise addition steps, this final fused image is also the reconstructed image.

[0083] In practice, the process of determining the reconstructed image content corresponding to an image region based on the reconstructed image and the image content of the corresponding complementary image region can be as follows: the position of the image region is determined based on the position of the complementary image region in the reconstructed image, and then the reconstructed image content corresponding to the image region is determined in the reconstructed image based on the position of the image region.

[0084] In some embodiments, the final reconstructed image can also be obtained through resampling. Specifically, during the denoising process, denoising is not performed along a monotonically decreasing sequence, i.e., continuous denoising. Instead, after every J denoising iterations, a diffusion process of J / 2 iterations is performed. That is, after J denoising iterations, J / 2 additional noise is added. This allows for better fusion of uncovered and covered areas in the reconstructed image. For example, see [link to example]. Figure 14 , Figure 14 This is a schematic diagram illustrating the process of obtaining the final reconstructed image through resampling, as provided in an embodiment of this application. Figure 14 If noise is added to the product image 5 times and J is 20 times, then after 20 iterations of the above denoising, image fusion, and image stitching processes to obtain the corresponding stitched image, a 21st denoising step will not be performed according to the above process. Instead, 10 more noise steps will be added to the corresponding stitched image to obtain a first target noise image. Then, based on the first target noise image, the above denoising, image fusion, and image stitching processes will be performed 20 more times to obtain the corresponding stitched image. Next, 10 more noise steps will be added to the corresponding stitched image to obtain a second target noise image. Then, based on the second target noise image, the above denoising, image fusion, and image stitching processes will be performed 20 more times. This process continues until the number of denoising steps is the same as the number of noise steps added. The final stitched image is then used as the reconstructed image.

[0085] Applying the above embodiments of this application, firstly, based on at least two image regions included in the product image corresponding to the target product, complementary image regions with complementary relationships to each image region are determined. Then, based on the image content of the complementary image regions, the image content of the corresponding image regions is reconstructed. The reconstructed image content of each image region is then combined to obtain a combined image. Based on the differences between the combined image and the product image, the anomaly detection result corresponding to the target product is determined. Thus, compared to the process of directly reconstructing images based on the overall features of the product image in related technologies, predicting the image content of image regions in the product image based on the image content of complementary image regions reduces the possibility of reconstructing abnormal locations during the reconstruction process, improving the accuracy of product anomaly detection. Simultaneously, based on the image content of the complementary image regions of the product image, the corresponding canonical image, i.e., the combined image, is predicted. Based on the differences between the canonical image and the product image, the anomaly detection result corresponding to the target product is determined. This eliminates the need to train a separate model for each product category and store a large number of features, significantly reducing the cost of the product anomaly detection process and improving the efficiency of product anomaly detection.

[0086] The following will describe an exemplary application of the embodiments of this application in a real-world application scenario.

[0087] The inventors discovered that in industrial manufacturing scenarios, emerging technologies such as artificial intelligence and deep learning are commonly used for workpiece quality inspection of product appearance to ensure product consistency, yield rate, and safety, achieving full automation of production line quality inspection. However, mature production lines have high yield rates, making it difficult to obtain a large number of defective products. Furthermore, the types of defects are diverse, and manual labeling is costly. Therefore, it is difficult to obtain a large number of usable defective samples, and the imbalance between positive and negative samples limits the performance of supervised classification methods. Based on this, most anomaly detection processes in industrial products are based on image reconstruction methods, which reconstruct the input image or input features and calculate the difference between the original input and the reconstructed result for anomaly detection; or they are based on pre-trained feature and metric learning methods, which extract features through a fixed network and then perform distance measurements at the feature level for anomaly detection. However, for image reconstruction methods, the common use of autoencoders or generative adversarial networks requires the model to progressively upsample to obtain the final reconstruction result, resulting in a relatively blurry reconstruction and limiting anomaly detection accuracy. In addition, due to the large number of parameters and strong fitting ability of deep networks, the model can often reconstruct abnormal inputs, leading to a discrepancy between input and output that is not as expected, which also significantly affects the final anomaly detection performance. For methods based on pre-trained features and metric learning, a large number of features need to be stored, which requires a lot of memory space during inference. In addition, each product needs to be trained separately, making it difficult to handle the joint judgment of multiple categories. Moreover, since a feature may be a normal feature in one category and an abnormal feature in another category, this method often leads to poor performance in multi-semantic category scenarios.

[0088] Based on this, the present application provides an industrial anomaly detection method based on diffusion model image inpainting. Compared with the process of image reconstruction based directly on the overall features of product images in related technologies, this application only requires training on normal samples and can adaptively detect various types of abnormal samples. At the same time, it can locate abnormal regions. Therefore, since the trained model does not contain abnormal images but only normal samples, it can only reconstruct normal samples of the same semantic category, reducing the possibility of reconstructing abnormal locations during the reconstruction process and improving the accuracy of product anomaly detection. At the same time, normal workpieces of all different semantic categories are added to the training to train a unified model, so that the model can obtain generation results of different prediction categories. Therefore, it is not necessary to explicitly classify samples or train multiple different anomaly detection models, which greatly reduces the cost of model deployment and improves efficiency.

[0089] In practice, two complementary masking masks (virtual masks) are generated, and the product image corresponding to the target product is masked based on the two masks respectively. Then, using the existing unmasked information (image content of the complementary image regions), a diffusion model is used to generate a predicted masked region image (the reconstructed image content corresponding to the image region). The two predicted masked regions are then stitched together to obtain a complete predicted image (combined image). Since the diffusion model is trained on a fully normal training set, the final predicted image is also a normal image. At this point, the square root of the sum of the squares of the differences between pixels at the same position in the input image (product image) and the predicted image (the sum of the squares of the differences in the first, second, and third color values ​​of the corresponding pixels) is taken to obtain the change between the input and the prediction. This image of the changes is the anomaly segmentation map (difference map), and the anomaly detection result is determined based on the anomaly segmentation map.

[0090] Next, the technical solution of this application will be described from the product side. The industrial anomaly detection method based on diffusion model image inpainting is mainly used in the field of industrial parts quality inspection. Here, when training the model, only normal samples (product image samples corresponding to normal products) are needed, without distinguishing between types. Normal samples of different products (such as metal parts and glass parts) can be input simultaneously for training. Thus, during inference, the normal sample image (combined image) that is closest to the approximate image of the current sample can be adaptively reconstructed, thereby accurately giving the difference between the current input sample and the normal sample, and determining the pixel positions with large differences as anomaly areas.

[0091] Next, the technical solution of this application will be described from a technical perspective. (See also...) Figure 15 , Figure 15 This is a schematic diagram of the technical architecture of the industrial anomaly detection method based on diffusion model image inpainting provided in this application embodiment. The industrial anomaly detection method based on diffusion model image inpainting can be implemented based on an anomaly detection model, which includes a mask generation module, a diffusion module, a reconstruction module, and an output module. Figure 15The mask generation module generates two complementary mask images (virtual masks), where the masked portion is exactly the visible portion of the other image. Here, the pixel values ​​of the masked portion are 0, and the pixel values ​​of the visible portion are 1. The input image is then multiplied by the mask image. Thus, the visible pixel values ​​in the input image remain the original RGB values, while the RGB pixel values ​​of the masked portion are [0, 0, 0], meaning the information is discarded. Next, for the diffusion module, i.e., the diffusion model, the trained diffusion model uses mathematical methods and the U-Net model to progressively backdifflate from Gaussian noise into the "original image" (reconstructed image) based on the existing RGB values. This "original image" is the predicted value. Since the visible portion of the final "original image" is directly copied from the input without processing, only the masked portion of the "original image" is generated by the model based on existing information. Therefore, by combining the predicted parts of two different masks through the reconstruction module, the final reconstructed image is obtained (based on the position of the reconstructed image content in each reconstructed image, the reconstructed image content is cropped from each reconstructed image to obtain a cropped image; the cropped images are then stitched together to obtain a target stitched image; the target stitched image is determined as the combined image); finally, this reconstructed image (combined image) is compared with the input image (product image) to obtain the final anomaly segmentation map.

[0092] Next, we will explain the diffusion model. First, for an image, if a little noise is added to each pixel each time, after a sufficient number of iterations, the entire image will appear to have lost almost all its information. It would be extremely difficult to directly reconstruct the original image from the resulting noisy image. However, if we only reconstruct a little bit each time, that is, attempt to reconstruct the result from one iteration ago, the diffusion model has two processes: the diffusion process and the de-diffusion process.

[0093] The diffusion process, which is essentially the gradual addition of noise to an image, involves adding noise t times to the input image X0, resulting in X. t The process, at the same time, X t It is for X t-1 Obtained by adding noise, and only affected by X t-1 Therefore, the diffusion process is a Markov process, influenced by variables. Furthermore, the step size t of each diffusion step is affected by the variable... The effect of X0. X0 represents an image sampled from the real dataset. Adding T noises to X0 gradually blurs the image. When t is large enough, X... t It follows a standard normal distribution. During training, the noise added each time is known, i.e. It is known that, based on the properties of Markov processes, it can be obtained recursively. ,Right now It is known.

[0094] And for certain and The process, given X t-1 Under the condition that X t Follow the mean The variance is It follows a normal distribution, therefore... It can be written as Among them, the reparameterization technique represents X t ,make ,make ,but ...Formula (2); Among them, let ,but ...Formula (3); Let random variable for ...Formula (4); but Expected value E and variance D are ...Formula (5); ...Formula (6); therefore ...Formula (7); ...Formula (8); Based on this, according to and The diffusion state (Xt) at step t can be directly calculated using the original image (X0). t ), or the image at step t-1 (X) can be used t-1 ), calculate the diffusion state (X) at step t. t ).

[0095] The inverse diffusion process, also known as the gradual denoising of an image, is used to represent the gradual reconstruction of an image from noise. When noise is added t times, if it is possible to recover an image from a given X... t Knowing X under the given conditions t-1 The distribution, that is, if we can know Therefore, we can obtain an image from any noisy image through repeated sampling, thus achieving the goal of image generation. Obviously, it's difficult to know... Therefore, here we use To approximate , that is This refers to the network that needs to be trained, such as the U-Net model. Furthermore, although it is unknown... ,but It is usable and Indicated, that is It is knowable, therefore it can be based on To come The training process. Here That is to say ...Formula (9); Since the diffusion process is a Markov process, it only depends on the previous moment, that is... ...Formula (10); so ...Formula (11); because ...Formula (12); ...Formula (13); ...Formula (14); Here, exp indicates an exponential function with the natural constant e as its base, so ...Formula (15); ...Formula (16); That is to say ...Formula (17); in, It does not involve X t-1 The mean and variance are functions of the Gaussian density function. According to the standard Gaussian density function, the mean and variance can be parameterized as follows: ...Formula (18); ...Formula (19); so ...Formula (20); because ...Formula (21); so ...Formula (22); Based on this, it was determined that .

[0096] It should be noted that the only requirement for the network model in the diffusion model is that the input and output data dimensions must be the same (both are the dimensions of an image). Therefore, as... Figure 13 As shown, the overall structure of the diffusion model is similar to an encoder-decoder. The first half of the network (encoder) extracts features and saves the extraction results. Then, the extracted features are concatenated to the second half of the network to assist in the prediction of pixel-by-pixel noise (used in the inverse diffusion process).

[0097] In practical implementation, for the process of determining the reconstructed image, i.e., the image restoration, based on the diffusion model, here, it is assumed that it is based on... Determine the known portion (unmasked portion) in the image, based on Identify the unknown portions (masked areas) in the image. For the known portions, the image (product image) after adding noise can be directly determined from step 0 through the diffusion process to step t-1, i.e. For the unknown parts, we can use the U-Net network model to progressively sample back from the Gaussian noise, from step t through the reverse diffusion process to the denoised image at step t-1, i.e. Then, the two are concatenated to obtain the image prediction at step t-1, i.e. ...Formula (23); Among them, such as Figure 12 As shown, m is the mask of the corresponding product image determined based on each image region and the corresponding complementary image region, which is the image indicated by 1202. 1-m is used to indicate the complementary mask that has a complementary relationship with the mask obtained with the product image as a reference, which is the image indicated by 1203.

[0098] Then, by gradually sampling (denoising) from step t back to step 0, the final image prediction result, i.e., the reconstructed image, can be obtained.

[0099] In some embodiments, a resampling mechanism is introduced to facilitate image inpainting. Specifically, when sampling back from step t to step 0, sampling is not performed along a monotonically decreasing sequence. Instead, a diffusion process of J / 2 steps is performed every J steps. This allows for better integration of known and unknown regions in the reconstructed image. For example, when t=50 and J=20, a resampling process is performed as follows: Figure 14 The diffusion process and the reverse diffusion process are shown.

[0100] Next, the processing procedure of the mask generation module will be explained. First, the input image is copied to obtain n (the first target number) input images. Each input image is then processed into N small rectangular blocks of size K×K (the second target number of rectangular regions). N / n (the third target number of) small rectangular blocks are masked, and this process is repeated n times to obtain n masked "incomplete images" (masked images). The intersection of the masked small rectangular blocks in these n "incomplete images" is empty, ensuring that each small rectangular block is masked and then reconstructed in subsequent processing. Here, the height and width of the input image are H and W, respectively, N is H / K×W / K, and n is a hyperparameter, such as 2, used to indicate the number of incomplete images to be generated.

[0101] It should be noted that for the training process of the anomaly detection model, only the diffusion model needs to be trained here; the processing procedures of the other modules are deterministic and do not contain learnable parameters. Specifically, firstly, all normal product samples of various categories to be detected are obtained and randomly input into the network. Since the trained diffusion model is "unconditional," there is no need to label its semantic categories, i.e., there is no need to distinguish between different products such as resistors and screws. Then, L... t The loss function is used to supervise the network. Here, determining the loss function is difficult because it's hard to integrate over the noise space, making direct optimization challenging. Therefore, we will instead optimize the variational upper bound of the loss function. ,Right now ...Formula (24); ...Formula (25); ...Formula (26); ...Formula (27); ...Formula (28); ...Formula (29); ...Formula (30); ...Formula (31); …Formula (32); Organized ...Formula (33); in ...Formula (34); ...Formula (35); ...Formula (36); Based on this, by L t It can be determined that, The training process is also known as minimization. and The KL divergence.

[0102] Next, the processing procedure of the output module will be explained. Here, the output consists of image-level anomaly results (first anomaly detection results) and pixel-level anomaly results (second anomaly detection results). For pixel-level anomaly results, an anomaly value (pixel value) is assigned to each pixel in the output image. Specifically, as shown in formula (1), the pixel-level anomaly result is determined based on the anomaly segmentation map. For image-level anomaly results, they are used to indicate whether the target product in the detected image contains an anomaly region. Specifically, the anomaly segmentation map is average-pooled, and then the maximum value after average pooling is determined. Based on this maximum value, the image-level anomaly result is determined.

[0103] Thus, the anomaly detection model has a clear network structure, with each module exhibiting good generalization ability. It can train on multiple artifacts with different semantic categories at once and adaptively perform image inpainting during inference. Moreover, explicit semantic category information is not required during model training and inference, reducing the cost of anomaly detection and improving its efficiency. It can not only directly output image-level anomaly detection results but also generate accurate pixel-level anomaly detection results. Furthermore, each stage is highly interpretable, and the results are clear, facilitating error correction when the algorithm encounters anomalies.

[0104] Applying the above embodiments of this application, firstly, based on at least two image regions included in the product image corresponding to the target product, complementary image regions with complementary relationships to each image region are determined. Then, based on the image content of the complementary image regions, the image content of the corresponding image regions is reconstructed. The reconstructed image content of each image region is then combined to obtain a combined image. Based on the differences between the combined image and the product image, the anomaly detection result corresponding to the target product is determined. Thus, compared to the process of directly reconstructing images based on the overall features of the product image in related technologies, predicting the image content of image regions in the product image based on the image content of complementary image regions reduces the possibility of reconstructing abnormal locations during the reconstruction process, improving the accuracy of product anomaly detection. Simultaneously, based on the image content of the complementary image regions of the product image, the corresponding canonical image, i.e., the combined image, is predicted. Based on the differences between the canonical image and the product image, the anomaly detection result corresponding to the target product is determined. This eliminates the need to train a separate model for each product category and store a large number of features, significantly reducing the cost of the product anomaly detection process and improving the efficiency of product anomaly detection.

[0105] The following description continues to illustrate the exemplary structure of the anomaly detection device 455 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software module in the product anomaly detection device 455 stored in the memory 450 may include: The acquisition module 4551 is used to acquire a product image corresponding to the target product. The product image includes at least two image regions, and the image contents of the at least two image regions are stitched together to form the product image. The first determining module 4552 is used to determine, with reference to the product image, complementary image regions that have a complementary relationship with each of the image regions; The reconstruction module 4553 is used to reconstruct the image content of each image region based on the image content of the corresponding complementary image region, so as to obtain the reconstructed image content corresponding to the image region. The combination module 4554 is used to combine the reconstructed image contents corresponding to each of the image regions to obtain a combined image; The second determining module 4555 is used to obtain the difference between the combined image and the product image, and determine the anomaly detection result corresponding to the target product based on the difference.

[0106] In some embodiments, the apparatus further includes a masking module, which is configured to copy the product image to obtain product images of the same number as the image regions; and to perform masking processing on each of the product images to obtain masked images; wherein the masked regions in the masked images correspond to the image regions, and different masked images correspond to different image regions; the reconstruction module 4553 is further configured to reconstruct the image content of the masked regions based on the masked images to obtain the reconstructed image content corresponding to the image regions.

[0107] In some embodiments, the reconstruction module 4553 is further configured to obtain an image reconstruction model; wherein the number of image reconstruction models is the same as the number of image regions, and there is a one-to-one correspondence between the image reconstruction models and the image regions; the image content of the masked region is reconstructed using the image reconstruction model to obtain the reconstructed image content corresponding to the image region.

[0108] In some embodiments, the apparatus further includes a training module, which is configured to acquire an initial image reconstruction model and product image samples corresponding to normal products, wherein the product image samples include at least two training image regions, and the image content of the at least two training image regions is stitched together to form the product image training sample; using the product image training sample as a reference, complementary training image regions that have a complementary relationship with each of the training image regions are determined; using the initial image reconstruction model, for each of the training image regions, the image content of the training image region is reconstructed based on the image content of the corresponding complementary training image region to obtain the reconstructed image content corresponding to the training image region; the difference between the reconstructed image content and the image content of the corresponding training image region is acquired, and the model parameters of the initial image reconstruction model are updated based on the difference to obtain the image reconstruction model.

[0109] In some embodiments, the apparatus further includes a segmentation module, which is configured to copy the product image to obtain product images of the same number as the image regions; segment each product image to obtain segmented images corresponding to the complementary image regions of the product images; wherein different segmented images correspond to different image regions; the reconstruction module 4553 is further configured to reconstruct the image content of the image region corresponding to the segmented image based on the segmented image to obtain the reconstructed image content corresponding to the image region.

[0110] In some embodiments, each reconstructed image content corresponds to a reconstructed image. The combination module 4554 is further configured to crop the reconstructed image content from each reconstructed image based on the position of the reconstructed image content in each reconstructed image to obtain a cropped image; perform image stitching on each cropped image to obtain a target stitched image; and determine the target stitched image as the combined image.

[0111] In some embodiments, the apparatus further includes a third determining module, which is configured to determine a mask corresponding to the product image based on each of the image regions and the corresponding complementary image regions, and to obtain a complementary mask that is complementary to the mask, with the product image as a reference; wherein the mask is used to cover the image content of the image region in the product image, and the complementary mask is used to cover the image content of the complementary image region in the product image; the reconstruction module 4553 is further configured to combine the mask, the complementary mask, and the image content of the corresponding complementary image region to reconstruct the image content of the image region, thereby obtaining the reconstructed image content corresponding to the image region.

[0112] In some embodiments, the reconstruction module 4553 is further configured to add noise to the product image at least once to obtain a first noise image, and obtain a second noise image corresponding to the product image, wherein the second noise image is the image to be processed before the last noise addition processing is performed on the product image; perform denoising processing on the first noise image to obtain a denoised image; perform image fusion of the second noise image and the mask to obtain a first fused image, and perform image fusion of the denoised image and the complementary mask to obtain a second fused image; perform image stitching of the first fused image and the second fused image to obtain the reconstructed image; and determine the reconstructed image content corresponding to the image region based on the reconstructed image and the image content of the corresponding complementary image region.

[0113] In some embodiments, the second determining module 4555 is further configured to obtain pixel differences between each first pixel in the combined image and a second pixel at the same position in the product image; determine a difference map between the combined image and the product image based on each pixel difference; and use the difference map as the difference between the combined image and the product image.

[0114] In some embodiments, the second determining module 4555 is further configured to determine an anomaly detection result corresponding to the target product based on the difference map when the difference is a difference map between the combined image and the product image; wherein the anomaly detection result includes at least one of a first anomaly detection result and a second anomaly detection result, the first anomaly detection result being used to indicate whether there is an anomaly in the target product, and the second anomaly detection result being used to indicate the location of the anomaly on the target product.

[0115] In some embodiments, the anomaly detection result includes a first anomaly detection result. The second determining module 4555 is further configured to divide the difference map into regions to obtain at least two sub-image regions; for each sub-image region, average the pixel values ​​of multiple pixels in the sub-image region to obtain an average pixel value for the sub-image region; select the maximum average pixel value from the average pixel values ​​of the multiple sub-image regions based on the magnitude of the average pixel value; and determine the first anomaly detection result based on the maximum average pixel value.

[0116] In some embodiments, the anomaly detection result includes a second anomaly detection result. The second determining module 4555 is further configured to: obtain the pixel value of each pixel in the difference map based on the difference map; select at least one target pixel from the plurality of pixels in the difference map based on the magnitude of each pixel value; and determine the second anomaly detection result based on the position of the target pixel.

[0117] In some embodiments, the second determining module 4555 is further configured to perform the following processing for each pixel in the difference map: compare the pixel value of the pixel with a pixel value threshold to obtain a comparison result; when the comparison result indicates that the pixel value of the pixel is greater than the pixel value threshold, determine the pixel as a target pixel.

[0118] This application provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the anomaly detection method of the product described in this application embodiment, for example, as... Figure 3 The method for detecting anomalies in the product is shown.

[0119] This application provides a computer-readable storage medium storing executable instructions. When these executable instructions are executed by a processor, they cause the processor to execute the anomaly detection method provided in this application. For example, ... Figure 3 The method for detecting anomalies in the product is shown.

[0120] In some embodiments, the computer-readable storage medium may be a read-only memory (ROM), random access memory (RAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or CD-ROM, etc.; or it may be a device that includes one or any combination of the above-mentioned memories.

[0121] In some embodiments, executable instructions may take the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a standalone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0122] As an example, executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple collaborating files (e.g., a file that stores one or more modules, subroutines, or code sections).

[0123] As an example, executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0124] In summary, the embodiments of this application have the following beneficial effects: (1) Compared with the process of image reconstruction based on the overall features of the product image in related technologies, the image content of the image region in the product image is predicted based on the image content of the complementary image region in the product image, which reduces the possibility of reconstructing abnormal positions in the reconstruction process and improves the accuracy of product anomaly detection. At the same time, the corresponding product image is predicted based on the image content of the complementary image region of the product image, i.e., the combined image. Based on the difference between the standardized image and the product image, the anomaly detection result corresponding to the target product is determined. There is no need to train a separate model for each type of product and save a lot of features, which greatly reduces the cost of the product anomaly detection process and improves the product anomaly detection efficiency.

[0125] (2) By setting the image acquisition device as a mobile terminal, image acquisition is performed based on the mobile terminal device, and anomaly detection is performed on the product image corresponding to the acquired target product. In this way, the flexibility and low cost of the mobile terminal device can greatly improve the availability and convenience of anomaly detection of the target product.

[0126] It should be noted that in the embodiments of this application, data such as product images are involved. When the embodiments of this application are applied to specific products or technologies, it is necessary to obtain the corresponding license or consent, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0127] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A method for detecting anomalies in a product, characterized in that, The method includes: Obtain a product image corresponding to the target product, wherein the product image includes at least two image regions, and the image contents of the at least two image regions are stitched together to form the product image; Using the product image as a reference, complementary image regions that have a complementary relationship with each of the image regions are determined respectively; For each of the image regions, based on the image content of the corresponding complementary image regions, the image content of the image region is reconstructed to obtain the reconstructed image content corresponding to the image region. The reconstructed image contents corresponding to each of the image regions are combined to obtain a combined image; Obtain the pixel difference between each first pixel in the combined image and the second pixel at the same position in the product image; Based on the differences between each pixel, a difference map between the combined image and the product image is determined; Based on the difference map, an anomaly detection result corresponding to the target product is determined; wherein, the anomaly detection result includes at least one of a first anomaly detection result and a second anomaly detection result, the first anomaly detection result is used to indicate whether there is an anomaly in the target product, and the second anomaly detection result is used to indicate the location of the anomaly on the target product.

2. The method as described in claim 1, characterized in that, After determining complementary image regions that are complementary to each of the image regions, using the product image as a reference, the method further includes: The product image is copied to obtain product images with the same number of regions as the image area; Each of the product images is masked to obtain a masked image; The masked area in the masked image corresponds to the image area, and different masked images correspond to different image areas; The step of reconstructing the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region includes: Based on the masked image, the image content of the masked region is reconstructed to obtain the reconstructed image content corresponding to the image region.

3. The method as described in claim 2, characterized in that, The step of reconstructing the image content of the masked region based on the masked image to obtain the reconstructed image content corresponding to the image region includes: Obtain an image reconstruction model; wherein the number of image reconstruction models is the same as the number of image regions, and there is a one-to-one correspondence between the image reconstruction models and the image regions; The image content of the masked area is reconstructed using an image reconstruction model, resulting in the reconstructed image content corresponding to the image area.

4. The method as described in claim 3, characterized in that, Before obtaining the product image corresponding to the target product, the method further includes: An initial image reconstruction model and product image samples corresponding to normal products are obtained. The product image samples include at least two training image regions. The image contents of the at least two training image regions are stitched together to form the product image training samples. Using the product image training samples as a reference, complementary training image regions that have a complementary relationship with each of the training image regions are determined respectively; Using the initial image reconstruction model, for each of the training image regions, the image content of the training image region is reconstructed based on the image content of the corresponding complementary training image region, thus obtaining the reconstructed image content corresponding to the training image region; The differences between the reconstructed image content and the image content of the corresponding training image region are obtained, and the model parameters of the initial image reconstruction model are updated based on the differences to obtain the image reconstruction model.

5. The method as described in claim 1, characterized in that, After determining complementary image regions that are complementary to each of the image regions, using the product image as a reference, the method further includes: The product image is copied to obtain product images with the same number of regions as the image area; For each of the product images, the product images are segmented to obtain segmented images corresponding to the complementary image regions of the product images; Different segmented images correspond to different image regions; The step of reconstructing the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region includes: Based on the segmented image, the image content of the image region corresponding to the segmented image is reconstructed to obtain the reconstructed image content corresponding to the image region.

6. The method as described in claim 1, characterized in that, Each reconstructed image content corresponds to a reconstructed image. The step of combining the reconstructed image content corresponding to each image region to obtain a combined image includes: Based on the position of the reconstructed image content in each of the reconstructed images, the reconstructed image content is cropped from each of the reconstructed images to obtain a cropped image; The cropped images are stitched together to obtain the target stitched image; The target stitched image is determined as the combined image.

7. The method as described in claim 6, characterized in that, The method further includes: Based on each of the image regions and the corresponding complementary image regions, a mask corresponding to the product image is determined, and with the product image as a reference, a complementary mask that has a complementary relationship with the mask is obtained; Wherein, the mask is used to cover the image content of the image region in the product image, and the complementary mask is used to cover the image content of the complementary image region in the product image; The step of reconstructing the image content of the image region based on the image content of the corresponding complementary image region to obtain the reconstructed image content corresponding to the image region includes: By combining the mask, the complementary mask, and the image content of the corresponding complementary image region, the image content of the image region is reconstructed to obtain the reconstructed image content corresponding to the image region.

8. The method as described in claim 7, characterized in that, The process of combining the mask, the complementary mask, and the image content of the corresponding complementary image region to reconstruct the image content of the image region, thereby obtaining the reconstructed image content corresponding to the image region, includes: Add noise to the product image at least once to obtain a first noisy image, and obtain a second noisy image corresponding to the product image. The second noisy image is the image to be processed before the last noise addition process is performed on the product image. The first noisy image is denoised to obtain a denoised image. The second noisy image is fused with the mask to obtain a first fused image, and the denoised image is fused with the complementary mask to obtain a second fused image. The first fused image and the second fused image are stitched together to obtain the reconstructed image; Based on the reconstructed image and the image content of the corresponding complementary image region, the reconstructed image content corresponding to the image region is determined.

9. The method as described in claim 1, characterized in that, The anomaly detection results include a first anomaly detection result. Determining the anomaly detection result corresponding to the target product based on the difference map includes: The difference map is divided into regions to obtain at least two sub-image regions; For each of the sub-image regions, the average pixel value of multiple pixels in the sub-image region is taken to obtain the average pixel value for the sub-image region; Based on the magnitude of the average pixel value, the maximum average pixel value is selected from the average pixel values ​​of the plurality of sub-image regions; The first anomaly detection result is determined based on the maximum average pixel value.

10. The method as described in claim 1, characterized in that, The anomaly detection result includes a second anomaly detection result. Determining the anomaly detection result corresponding to the target product based on the difference map includes: Based on the difference map, obtain the pixel value of each pixel in the difference map; Based on the magnitude of each pixel value, at least one target pixel is selected from the plurality of pixels in the difference map; The second anomaly detection result is determined based on the position of the target pixel.

11. The method as described in claim 10, characterized in that, The step of selecting at least one target pixel from a plurality of pixels in the difference map based on the magnitude of each pixel value includes: For each pixel in the difference map, the following processing is performed: The pixel value of the pixel is compared with the pixel value threshold to obtain the comparison result; When the comparison result indicates that the pixel value of the pixel is greater than the pixel value threshold, the pixel is determined as the target pixel.

12. An anomaly detection device for a product, characterized in that, The device includes: The acquisition module is used to acquire a product image corresponding to the target product. The product image includes at least two image regions, and the image contents of the at least two image regions are stitched together to form the product image. The first determining module is used to determine, with reference to the product image, complementary image regions that have a complementary relationship with each of the image regions; The reconstruction module is used to reconstruct the image content of each image region based on the image content of the corresponding complementary image region, so as to obtain the reconstructed image content corresponding to the image region. The combination module is used to combine the reconstructed image content corresponding to each of the image regions to obtain a combined image; The second determining module is used to obtain the pixel difference between each first pixel in the combined image and a second pixel at the same position in the product image; based on each pixel difference, determine a difference map between the combined image and the product image; based on the difference map, determine the anomaly detection result corresponding to the target product; wherein the anomaly detection result includes at least one of a first anomaly detection result and a second anomaly detection result, the first anomaly detection result being used to indicate whether there is an anomaly in the target product, and the second anomaly detection result being used to indicate the location of the anomaly on the target product.

13. The apparatus according to claim 12, characterized in that, The masking module is used to copy the product image after determining complementary image regions that are complementary to each of the image regions, with the product image as a reference, to obtain a product image with the same number of image regions. Each of the product images is masked to obtain a masked image; The masked area in the masked image corresponds to the image area, and different masked images correspond to different image areas; The reconstruction module is further configured to reconstruct the image content of the masked region based on the masked image, thereby obtaining the reconstructed image content corresponding to the image region.

14. The apparatus according to claim 13, characterized in that, The reconstruction module is further configured to acquire image reconstruction models; wherein the number of image reconstruction models is the same as the number of image regions, and there is a one-to-one correspondence between the image reconstruction models and the image regions; The image content of the masked area is reconstructed using an image reconstruction model, resulting in the reconstructed image content corresponding to the image area.

15. The apparatus according to claim 14, characterized in that, The training module is used to acquire an initial image reconstruction model and product image samples corresponding to normal products before acquiring the product image corresponding to the target product. The product image samples include at least two training image regions, and the image contents of the at least two training image regions are stitched together to form the product image training samples. Using the product image training samples as a reference, complementary training image regions that have a complementary relationship with each of the training image regions are determined respectively; Using the initial image reconstruction model, for each of the training image regions, the image content of the training image region is reconstructed based on the image content of the corresponding complementary training image region, thus obtaining the reconstructed image content corresponding to the training image region; The differences between the reconstructed image content and the image content of the corresponding training image region are obtained, and the model parameters of the initial image reconstruction model are updated based on the differences to obtain the image reconstruction model.

16. The apparatus according to claim 12, characterized in that, The segmentation module is used to copy the product image after determining complementary image regions that are complementary to each of the image regions, with the product image as a reference, to obtain a product image with the same number of image regions. For each of the product images, the product images are segmented to obtain segmented images corresponding to the complementary image regions of the product images; Different segmented images correspond to different image regions; The reconstruction module is further configured to reconstruct the image content of the image region corresponding to the segmented image based on the segmented image, thereby obtaining the reconstructed image content corresponding to the image region.

17. The apparatus according to claim 12, characterized in that, Each piece of reconstructed image content corresponds to one reconstructed image. The combination module is further configured to crop out the reconstructed image content from each of the reconstructed images based on the position of the reconstructed image content in each of the reconstructed images, thereby obtaining a cropped image; The cropped images are stitched together to obtain the target stitched image; The target stitched image is determined as the combined image.

18. The apparatus according to claim 17, characterized in that, The third determining module is used to determine the mask corresponding to the product image based on each of the image regions and the corresponding complementary image regions, and to obtain a complementary mask that has a complementary relationship with the mask using the product image as a reference. Wherein, the mask is used to cover the image content of the image region in the product image, and the complementary mask is used to cover the image content of the complementary image region in the product image; The reconstruction module is further configured to combine the mask, the complementary mask, and the image content of the corresponding complementary image region to reconstruct the image content of the image region, thereby obtaining the reconstructed image content corresponding to the image region.

19. The apparatus according to claim 18, characterized in that, The reconstruction module is further configured to add noise to the product image at least once to obtain a first noise image, and obtain a second noise image corresponding to the product image, wherein the second noise image is the image to be processed before the last noise addition process is performed on the product image; The first noisy image is denoised to obtain a denoised image. The second noisy image is fused with the mask to obtain a first fused image, and the denoised image is fused with the complementary mask to obtain a second fused image. The first fused image and the second fused image are stitched together to obtain the reconstructed image; Based on the reconstructed image and the image content of the corresponding complementary image region, the reconstructed image content corresponding to the image region is determined.

20. The apparatus according to claim 12, characterized in that, The anomaly detection results include the first anomaly detection result. The second determining module is further configured to divide the difference map into regions to obtain at least two sub-image regions; For each of the sub-image regions, the average pixel value of multiple pixels in the sub-image region is taken to obtain the average pixel value for the sub-image region; Based on the magnitude of the average pixel value, the maximum average pixel value is selected from the average pixel values ​​of the plurality of sub-image regions; The first anomaly detection result is determined based on the maximum average pixel value.

21. The apparatus according to claim 12, characterized in that, The anomaly detection results include a second anomaly detection result. The second determining module is further configured to obtain the pixel value of each pixel in the difference map based on the difference map; Based on the magnitude of each pixel value, at least one target pixel is selected from the plurality of pixels in the difference map; The second anomaly detection result is determined based on the position of the target pixel.

22. The apparatus according to claim 21, characterized in that, The second determining module is further configured to perform the following processing for each pixel in the difference map: The pixel value of the pixel is compared with the pixel value threshold to obtain the comparison result; When the comparison result indicates that the pixel value of the pixel is greater than the pixel value threshold, the pixel is determined as the target pixel.

23. An electronic device, characterized in that, include: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the anomaly detection method of the product according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, It stores executable instructions for causing the processor to execute, thereby implementing the anomaly detection method of the product according to any one of claims 1 to 11.

25. A computer program product comprising a computer program or computer-executable instructions, characterized in that, When the computer program or computer-executable instructions are executed by the processor, the abnormality detection method of the product according to any one of claims 1 to 11 is implemented.

Images