Method and system for visual inspection of industrial products

Abstract

The present application relates to a kind of industrial product visual detection method and system, belong to industrial product detection field, by the image to be detected is carried out overlapping segmentation processing and obtains several sub-images, on the one hand, reduce the influence of image size on the accuracy of detection result, another sub-image has sequence correlation feature;By fusing the classification features of the current sub-image features and the previous sub-image, obtain the classification features of the current sub-image, without losing the image features of the current sub-image, but also can obtain the sequence correlation features before and after.By implementing attention mechanism to the obtained classification features, the soft attention to the defective area can be improved.The present application can not only detect the defects of industrial products in sub-image, but also can detect defects across sub-image, realize the defect detection in industrial product image without position labeling, can achieve the effect of fast detection speed and high detection precision, reduce the intensive requirement of model to data labeling.

Description

Technical Field

[0001] This disclosure relates to industrial product inspection, and more particularly to a visual inspection method and system for industrial products. Background Technology

[0002] In traditional factories, defect detection in industrial products relies on manual visual inspection. This not only requires workers to be familiar with all types of defects but also places a significant physical strain on them. Because manual inspection is slow, a large workforce is needed to complete quality checks. Furthermore, traditional manual quality inspection does not rely on computers, making it impossible to accurately count defects and other data, hindering the accurate identification of manufacturing process problems. With the continuous development of artificial intelligence, intelligent defect detection in industrial products has become a promising new direction for factories.

[0003] There are two main types of existing industrial product inspection technologies. One type uses target detection methods to extract and detect features from industrial product images, such as using the Faster-RCNN algorithm for feature extraction and defect identification. However, this approach requires large-scale defect annotation of the data and has a long detection time, which cannot meet the speed requirements of industrial inspection. The other type uses machine vision and pattern recognition methods to extract and detect features from industrial product images. First, the preprocessing module uses Otsu's thresholding method and morphological processing to extract the detection area from the background. Then, grayscale projection distribution anomaly detection and time-frequency domain template matching are used to analyze the spacing between product objects. However, this method requires template matching for products of the same type but different shapes separately, resulting in poor applicability. Summary of the Invention

[0004] In view of the above-mentioned prior art, the technical problems solved by the present invention include at least the following:

[0005] (1) It has high requirements for the dataset. It requires a high-precision labeled dataset to support model training in order to detect the specific location of defects in the image.

[0006] (2) The model has low efficiency in detecting defects.

[0007] (3) It cannot flexibly handle data of different sizes.

[0008] (4) Traditional image processing methods require the creation of a template library for each industrial product.

[0009] To solve the above-mentioned technical problems, the objective of this invention is:

[0010] (1) It is not necessary to label the location of defects in industrial product images, thus reducing the model's requirement for dense data labeling.

[0011] (2) On images without fine annotation, the location information of defects in the image can be inferred from the model.

[0012] (3) Reduce the aspect ratio requirements of the detection model for the submitted data, so that the impact of the detection device changing the size of the captured image on the accuracy of the detection result is smaller.

[0013] (4) It enables fast detection speed and high detection accuracy.

[0014] To achieve the above objectives, the technical solution of the present invention is as follows.

[0015] In a first aspect, the present invention provides a visual inspection method for industrial products, the method comprising the following steps:

[0016] The image of the industrial product to be inspected is divided into several sub-images, and adjacent sub-images have a set number of overlap lengths, so that the sub-images have sequential correlation.

[0017] Using a pre-defined first model, image features of multiple sub-images corresponding to the image to be detected are obtained;

[0018] A pre-defined second model is used to obtain classification features of sub-images based on their image features. The classification features include image features and sequence association features of the sub-images. An attention mechanism is used to obtain soft attention based on each classification feature.

[0019] The product is classified based on fusion features obtained from soft attention and classification features belonging to the same detection image.

[0020] In the above technical solution, several sub-images are obtained by overlapping segmentation of the image to be detected. This reduces the impact of image size on the accuracy of the detection results and gives the sub-images sequence association features. Each sub-image is input into a first model to obtain its image features. These features are then input into a second model, which extracts the classification features of the sub-images from these features. These classification features possess both the image features of the original sub-image and the sequence association features of the previous sub-image. An attention mechanism is used to enhance focus on defective regions. This invention not only detects defects in industrial products within sub-images but also detects defects across sub-images. It achieves high detection speed and accuracy without requiring location annotation for defect detection in product images, reducing the model's reliance on dense data annotation.

[0021] In the above technical solution, the sub-images are acquired through a sliding window method, ensuring that the size of the sub-images meets the model input requirements and that the product image is not distorted, thus preventing defects from affecting the accuracy of detection. Specifically, if the image to be detected is denoted as x, then the t-th sub-image x... tIt can be represented as:

[0022] x t =x[1+(db)×(t-1):d+(db)×(t-1)]

[0023] In the formula: d represents the window length, b represents the window overlap length, the symbol : indicates cropping from image x, and the pixel value ranges from (db)×(t-1) to d.

[0024] In the above technical solution, the first model is a residual network model, which enables smaller stacked modules to extract image features that can be used for classification.

[0025] In the above technical solution, the second model is an RNN, a Long Short-Term Memory Network (LSTM), or a Gated Recurrent Unit (GRU), which is beneficial for extracting sequence features of sub-image sequences.

[0026] In the above technical solution, the soft attention is calculated using the following formula:

[0027] Let the soft attention corresponding to the t-th sub-image be denoted as a. t :

[0028]

[0029] h t =σ(W e f t +W h h t-1 )

[0030] h0 = 0

[0031] In the formula: W a For soft attention learning matrix, h j h is the classification feature of the j-th sub-image. t h is the classification feature of the t-th sub-image. t-1 f is the classification feature of the (t-1)th sub-image. t Let W be the image features of the sub-images, t = 1, 2, ..., T, where T is the total number of sub-images, σ is the ReLU function, and W is the image features of the sub-images. e To extract the classification features of a sub-image from its image features, W is the extraction matrix. h Let be the sequence association matrix between the (t-1)th sub-image and the tth sub-image.

[0032] In the above technical solution, fused features are obtained based on soft attention and classification features belonging to the same detection image to classify industrial products, including:

[0033] Let the fused feature be denoted as C, then:

[0034]

[0035] Based on the fusion features, the probability of the industrial product belonging to a category is calculated:

[0036]

[0037] The category corresponding to the maximum probability is used as the category of industrial products in the image to be detected;

[0038] Where K represents the type of industrial product, and W... c For classification matrix, For the classification matrix W c The kth row, a t h is the soft attention corresponding to the t-th sub-image. t Let p(y) be the classification feature of the t-th sub-image, T be the total number of sub-images, and p(y) be the classification feature of the t-th sub-image. k ) represents the probability of belonging to the k-th category.

[0039] In the above technical solution, the first model and the second model update their parameters through backpropagation of the loss value, thereby reducing training time and improving the stability of both models. The total loss value is calculated using the following formula:

[0040] L=λL reg +L ce

[0041] In the formula: L is the total loss value, λ is the regularization coefficient, and L ce For standard cross-entropy, L reg This is a regularization term. The initial value of λ is 0.1.

[0042] In the above technical solution, one way to obtain the regularization term is: the regularization term is the square of the L2 norm of the first model parameter vector and the second model parameter vector.

[0043] In the above technical solution, the image input to the first model during the training phase undergoes enhancement processing; the enhancement processing includes random flipping, random occlusion, and random rotation. During implementation, one or more of these enhancement processes can be selected and combined to obtain general feature information through data augmentation training on a small amount of data.

[0044] Secondly, this invention proposes an industrial product visual inspection system. Applying any of the above methods to industrial product visual inspection can improve the efficiency of product defect detection. The system includes an external integration subsystem, a classification detector, and a database.

[0045] The external integration subsystem is configured to call the message center system and receive feedback from the message center system. The message center system is configured to control the startup of the machine system. The machine control system is configured to control the startup of the motor, camera, and conveyor belt containing the industrial products to obtain the images of the industrial products to be inspected.

[0046] The classification detector is configured to divide the image of an industrial product to be detected into several sub-images, and to ensure that adjacent sub-images have a set number of overlap lengths; to obtain image features of multiple sub-images corresponding to the image to be detected using a preset first model; to obtain classification features of the sub-images based on the image features of the sub-images using a preset second model, and to obtain soft attention based on each classification feature using an attention mechanism; and to obtain fusion features based on the soft attention and the classification features belonging to the same detection image, and to classify the product.

[0047] The database is configured to store the image to be detected and the classification result of the image to be detected.

[0048] The above system technical solution can flexibly process image data of different scales and obtain a good training model without high-precision annotation, thereby improving the efficiency of defect detection in industrial products.

[0049] Thirdly, the present invention provides a computer-readable storage medium storing a computer program that can be loaded by a processor and executed by any of the methods described above. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0051] Figure 1 , one A structural diagram of the detection system in one embodiment;

[0052] Figure 2 , one A schematic diagram of the detection process in one embodiment;

[0053] Figure 3 , one Internal structure diagram of the detection system in one embodiment;

[0054] Figure 4 , one A schematic diagram of the structure of the electronic device in one embodiment. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] The method of this invention can be implemented as a classification detector to classify whether industrial products have defects, and can be integrated into a quality inspection system, such as... Figure 1 As shown. During system operation, as... Figure 2 As shown, the worker inputs the product information to be inspected to start the machine, including initializing the front-end display. The product information is sent from the message center system to the quality inspection system. The quality inspection system obtains inspection information from external information systems, such as the material, type, size, and color of the industrial product. Simultaneously, the message center system controls the machine control system and industrial cameras to capture images of the industrial product. The machine control system includes a motor. After receiving the image capture signal, the motor drives the conveyor belt at a preset speed, causing the conveyor belt and product to sequentially pass through the industrial camera's capture area within the inspection machine for image acquisition. Each time the motor drives the conveyor belt for a preset length, the motor stops, and the message center system sends a signal to the industrial camera to take a picture. This process is repeated to capture image data from multiple products. The industrial product images are sent to the intelligent model scheduling engine of the quality inspection system. The intelligent model scheduling engine selects a suitable model from a multi-instance fusion learning framework for quality inspection, obtaining the inspection results. The quality inspection algorithm returns the inspection results to the message center system, which then controls the recording or stopping of the machine. The quality inspection system stores the image to be inspected and its classification result in an external storage system. The external storage system then stores the classification result in a database. For ease of retrieval and management, two databases are set up: an image database for storing the image to be inspected, and a result database for storing the classification result. The external storage system displays the inspection results on the front end. The operations described above can be performed out of order. Instead, they can be performed in reverse order or simultaneously. Furthermore, one or more additional operations can be added to the process, or one or more operations can be removed from the process.

[0057] The multi-instance fusion learning framework integrates algorithms for detecting various industrial products, including the classification detector of this invention, to meet different detection needs of the same type of industrial products or specific detection needs of different types of industrial products.

[0058] The classification detector of this invention is used for defect detection of industrial products, and its overall detection process is shown in the schematic diagram below. Figure 3 As shown, it includes the following steps:

[0059] Step 1: Perform data preprocessing on the captured industrial product images. Data preprocessing includes a data acquisition module, an image patch generator, and a data augmentation module.

[0060] Step 1-1: The data acquisition module obtains the image to be detected from the external storage system.

[0061] Steps 1-2: The data acquisition module transmits the image to be inspected to the image block generator for processing. The image block generator is mainly used to reduce the impact of different aspect ratios on product defect detection and classification. The image block generator performs a sliding window operation on the image to be inspected, segmenting and sampling the image according to a certain step size. Let the image to be inspected be denoted as x, d represent the window length, and b represent the window overlap length, then the t-th sub-image x t It can be represented as:

[0062] x t =x[1+(db)×(t-1):d+(db)×(t-1)]

[0063] In the formula: symbol: indicates cropping from image x, with pixel values ​​ranging from (db)×(t-1) to d.

[0064] For example, for an image with dimensions of 1600 pixels × 400 pixels, after processing it by sampling with a step size of 200 pixels and a segmentation length of 400 pixels, seven sub-images with dimensions of 400 pixels × 400 pixels can be obtained, and adjacent images have a 200-pixel overlap. Through sliding window sampling, the original 4:1 unbalanced aspect ratio of the image to be detected is adjusted to a balanced 1:1 aspect ratio. In other embodiments, the balanced aspect ratio may also be 2:1.

[0065] Steps 1-3: The data augmentation module is mainly used in the training phase of the defect detector. If the current phase is training, data augmentation operations such as random flipping, random occlusion, and random rotation are performed on the image blocks so that the defect detector can be trained on a small amount of data to obtain general feature information. If the current phase is detection, this module is skipped and the data is transmitted to the first model.

[0066] Step 2: The first model mainly extracts image features from the sub-images of the image to be detected. The second model extracts classification features from the image features of the sub-images and uses an attention mechanism to obtain soft attention based on each classification feature.

[0067] In implementation, the first model can be a residual network, and the second model can be an RNN, a Long Short-Term Memory (LSTM) network, or a Gated Recurrent Unit (GRU). To accelerate model training and improve the stability of the first and second models, the first model can be pre-trained using the ImageNet-1K dataset, and then the pre-trained parameters can be used to initialize the first model for training the second model.

[0068] The first and second models are trained using dynamic learning rates. These rates optimize the update speed of the model parameters and are further updated via backpropagation using the loss value. The total loss value is calculated using the following formula:

[0069] L=λL reg +L ce

[0070] In the formula: L is the total loss value, λ is the regularization coefficient, and the initial value of λ is 0.1. reg This is a regularization term. L ce The standard cross-entropy is calculated as follows:

[0071]

[0072] In the formula: N is the number of images x(i) to be detected; K is the type of industrial product; y(i) is the true label of the input sample x(i); p(y j ) represents the probability of belonging to the j-th category.

[0073] The Adam optimizer is further employed to optimize the training process. The learning rate is initialized to 0.001, and when the error rate of change meets a set condition, the learning rate is increased by a factor of 0.5. Another implementation of the dynamic learning rate is to increase it by a first set step size after each set number of training iterations; after reaching a set first baseline learning rate, it is decreased by a second set step size after each set number of training iterations, until a second baseline learning rate is reached. The dynamic learning rate is used to optimize and control the update speed of the parameters of the first and second models.

[0074] When acquiring classification features, in order to retain the original image features while also obtaining the sequential correlation features, the classification feature h of the t-th sub-image is determined. t It can be represented as:

[0075] h t =σ(W e f t +W h h t-1 )

[0076] Where: h t-1 f is the classification feature of the (t-1)th sub-image.t Let W be the image features of the sub-images, t = 1, 2, ..., T, where T is the total number of sub-images, σ is the ReLU function, and W is the image features of the sub-images. e To extract the classification features of a sub-image from its image features, W is the extraction matrix. h Let h0 be the sequence correlation matrix between the (t-1)th sub-image and the tth sub-image. If t = 1, then h0 = 0.

[0077] The classification features obtained in this way, except for the first image which lacks sequence association features, all other images can obtain sequence association features including the previous sub-image. The aforementioned classification features can also be sequence association features including the current image (the t-th sub-image) and the next sub-image (the (t+1)-th sub-image). That is, the classification features in this invention are extracted from the image features of the current image and the sequence association features between the current image and the previous or next image. If there is no previous or next image, the classification features are extracted only from the image features of the current image. The classification features obtained in this way focus more on rapid changes in the defect region.

[0078] Next, an attention mechanism is applied to the classification features of multiple sub-images obtained from a single image to be detected. This attention mechanism assigns more weight to defect targets to improve defect detection accuracy. The soft attention corresponding to the t-th sub-image is denoted as a. t :

[0079]

[0080] h t =σ(W e f t +W h h t-1 )

[0081] h0 = 0

[0082] In the formula: W a For soft attention learning matrix, h j h is the classification feature of the j-th sub-image. t h is the classification feature of the t-th sub-image. t-1 f is the classification feature of the (t-1)th sub-image. t Let W be the image features of the sub-images, t = 1, 2, ..., T, where T is the total number of sub-images, σ is the ReLU function, and W is the image features of the sub-images. e To extract the classification features of a sub-image from its image features, W is the extraction matrix. h Let be the sequence association matrix between the (t-1)th sub-image and the tth sub-image.

[0083] Step 3: Based on soft attention and classification features belonging to the same detection image, obtain fused features to classify the products, including:

[0084] Step 3-1: Let the fused feature be denoted as C, then:

[0085]

[0086] Among them, a t h is the soft attention corresponding to the t-th sub-image. t Let be the classification feature of the t-th sub-image, and T be the total number of sub-images.

[0087] Step 3-2: Calculate the probability of the product belonging to a category based on the fused features:

[0088]

[0089] Where: K represents the product type, W c For classification matrix, For the classification matrix W c The k-th row, p(y k ) represents the probability of belonging to the k-th category.

[0090] Step 3-3: Use the category corresponding to the maximum probability as the category of the product in the image to be detected. If the current stage is detection, output the detection result. If there is no defect, output the defect-free result directly; if there is a defect, output the location of the sub-image block where the defect is located.

[0091] In this implementation, only two categories are classified: "defective" and "no defect". If the detection result is "no defect", the quality inspection system increments the detection count on the front-end display and continues to detect defects in the next image. If the detection result is "defective", a "defective" signal is sent to the message center system, the number of defects displayed on the front-end display is incremented, and the defective sub-image area obtained from the detection result is marked on the original image, for example, by marking it with a red box. Then, the original image with the marked defective area is displayed on the front-end display, or this defective image is stored in an external storage system for subsequent manual analysis to determine if it is a model misjudgment. If the message center system receives the "defective" signal from the detection result, it controls the motor to stop rotating, controls the industrial camera to stop taking pictures, and pauses the inspection system. At the same time, the defective part is exposed outside the machine for workers to handle.

[0092] In other implementations, the categories can be further subdivided as needed.

[0093] The following embodiments use a specific industrial inductor as the detection object, but this does not mean that the detection method of the present invention is only applicable to inductors; it can also be applied to other industrial products. Two sets of inductor images were obtained, each set being 1600 pixels × 400 pixels. In one set of data, the adhesive surface of the inductor is blue, and in the other set, it is purple. Image data is shown in Table 1, and defect type data is shown in Table 2.

[0094] Table 1

[0095]

[0096] Table 2

[0097]

[0098] The first model uses a ResNet34 residual network neural network, and the second model uses an RNN neural network model. A sliding window operation is used to perform overlapping segmentation on the two sets of images. The window length d = W = 400, the window overlap length b = 200, and there are a total of 7 sub-images. The residual network consists of two convolutional layers with a kernel size of 3×3 and 64 kernels per layer. The extracted feature image dimension is 512, the soft attention dimension is 256, and the detection result is either "no defect" or "defective," so the product type K = 2. The initial value of the dynamic learning rate is 0.001, and the dynamic learning rate increases by 0.5 times when the error change rate meets the set condition.

[0099] Based on the datasets in Groups 1 and 2 above, the classification and detection method of this invention is compared with Faster C-RNN and YOLO-V5, as shown in Table 3. F1, AUC, and MCA are performance evaluation metrics for machine learning models, while Param (number of model parameters) and Flops (number of floating-point operations) are metrics used to analyze method complexity. As can be seen from Table 3, the method of this invention outperforms Faster R-CNN and YOLO-V5, exhibiting superior performance, requiring fewer parameters, and incurring less computational cost.

[0100] Table 3

[0101]

[0102] Next, we will make the following comparisons, which also include performance metrics for machine learning models and complexity metrics for analytical methods:

[0103] (1) Vanilla Instance Learning (VIL): The step of overlapping segmentation of the test image in the method of this invention is replaced by directly adjusting the image size, while the other steps remain unchanged. Taking the above image with a length and width of 1600 pixels × 400 pixels as an example, in VIL, it is directly adjusted to a sub-image with a length and width of 400 pixels × 400 pixels.

[0104] (2) Multi-instance Learning (MIL): The second model in the method of this invention is replaced with a multi-layer perceptron (MLP) with two layers.

[0105] (3) MIFL (woAttn) that does not employ attention mechanisms.

[0106] (4) MIFL (woRNN) without RNN.

[0107] (5) Reduce the resolution of the image dataset (LMIFL)

[0108] Table 4

[0109]

[0110] As can be seen from the above:

[0111] (1) Except for Flops (floating-point operations per second), MIL outperforms VIL in all other aspects. This is because MIL utilizes overlapping segmentation without distorting the image, extracting image features from an undistorted image. As for Param, VIL and MIL are almost identical. VIL uses global image features, but MIL uses a larger classification matrix because it needs to concatenate all sub-image features for judgment. However, VIL's Flops are much lower than MIL's.

[0112] (2)woAttn uses RNN to fuse sub-image features, unlike MIL which simply concatenates sub-image features together. Therefore, it is important to be able to fuse the features of each sub-image.

[0113] (3) By comparing MIL, woAttn and woRNN, it can be seen that the attention mechanism also plays an important role in aggregation, which makes fault detection more accurate.

[0114] (4) As can be seen from Table 4, feature fusion and soft attention promote each other, thus enabling MIFL with both aspects to perform well.

[0115] (5) As can be seen from LMIFL and MIFL, increasing the resolution of the image system as much as possible can improve the defect identification rate.

[0116] Finally, the generalization ability of the method of the present invention was further tested. The datasets of Group 1 and Group 2 were combined into a new group and compared with Group 1. The comparison results are shown in Table 5.

[0117] Table 5

[0118]

[0119] As can be seen from Table 5, the larger the amount of training data, the better the generalization of the method. By using color to make the input have different patterns, the generalization of the method can be improved without sacrificing performance, thereby improving the detection efficiency.

[0120] In summary, the method of this invention obtains several sub-images by performing overlapping segmentation on the image to be detected. This reduces the impact of image size on the accuracy of the detection results and endows the sub-images with sequence correlation features. By fusing the features of the current sub-image with the classification features of the previous sub-image, the classification features of the current sub-image are obtained, preserving the image features of the current sub-image while acquiring the sequence correlation features. By implementing an attention mechanism on the acquired classification features, the focus on defective regions can be improved. If this method is applied to defect detection, it can detect not only defects in industrial products within sub-images but also defects across sub-images. This achieves high detection speed and high accuracy without requiring location annotation for defect detection in product images, reducing the model's reliance on dense data annotation.

[0121] Another embodiment provides an industrial product visual inspection device that can perform the above-described method. Its implementation principle and technical effects are similar, and will not be repeated here.

[0122] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 4 The diagram shows a structural schematic of an electronic device 300 suitable for implementing embodiments of this application.

[0123] like Figure 4 As shown, the electronic device 300 includes a central processing unit (CPU) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage section 308 into a random access memory (RAM) 303. The RAM 303 also stores various programs and data required for the operation of the device 300. The CPU 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0124] The following components are connected to I / O interface 305: an input section 306 including a keyboard, mouse, etc.; an output section 307 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 308 including a hard disk, etc.; and a communication section 309 including a network interface card such as a LAN card, modem, etc. The communication section 309 performs communication processing via a network such as the Internet. Drive 310 is also connected to I / O interface 306 as needed. Removable media 311, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 310 as needed so that computer programs read from them can be installed into storage section 308 as needed.

[0125] In particular, according to embodiments of this disclosure, the above references Figure 1 The described process can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program tangibly embodied on a machine-readable medium, the computer program containing program code for performing the above-described method for constructing a cascade hydropower dispatch model. In such embodiments, the computer program can be downloaded and installed from a network via communication section 309, and / or installed from removable medium 311.

[0126] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0127] The units or modules described in the embodiments of this application can be implemented in software or hardware. The described units or modules can also be located in a processor. The names of these units or modules do not necessarily constitute a limitation on the unit or module itself.

[0128] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, a laptop computer, a mobile phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0129] In another aspect, this application also provides a storage medium, which may be the storage medium included in the aforementioned apparatus in the above embodiments; or it may be a standalone storage medium not assembled into the device. The storage medium stores one or more programs, which are used by one or more processors to execute the cascade hydropower scheduling model construction method described in this application.

[0130] Storage media, including both permanent and non-permanent, removable and non-removable media, can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0131] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0132] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

Claims

1. A visual inspection method for industrial products, characterized in that, The method includes the following steps: The image of the industrial product to be inspected is divided into several sub-images, and adjacent sub-images have a set number of overlap lengths, so that the sub-images have sequential correlation. Using a pre-defined first model, image features of multiple sub-images corresponding to the image to be detected are obtained; A pre-defined second model is used to obtain classification features of sub-images based on their image features. These classification features include both image features and sequence association features of the sub-images. A soft attention mechanism is then applied to each classification feature to obtain soft attention, which is calculated using the following formula: The first The soft attention corresponding to each subimage is denoted as : In the formula: , For soft attention learning matrix, For the first Classification features of individual images, For the first Classification features of individual images, For the first -1 classification features of sub-images, Image features of sub-images , The total number of sub-images. For ReLU function, This is an extraction matrix for extracting the classification features of a sub-image from its image features. For the first The sub-image and the first Sequence correlation matrix of sub-images; Based on soft attention and classification features belonging to the same detection image, fused features are obtained to classify products, including: The fusion feature is denoted as ,but: Based on the fusion features, the probability of the industrial product belonging to a category is calculated: The category corresponding to the maximum probability is used as the category of industrial products in the image to be detected; in, As a category of industrial products, For classification matrix, For classification matrix The OK, For the first Soft attention corresponding to each sub-image For the first Classification features of individual images, The total number of sub-images. Indicates belonging to the first The probability of each category.

2. The method according to claim 1, characterized in that, The sub-images are acquired via a sliding window method; The image to be detected is denoted as Then the first Sub-images It can be represented as: In the formula: Indicates the window length. Indicates the overlap length of windows, symbol Indicates from image Cropping, the range of pixel values ​​is from arrive .

3. The method according to claim 1, characterized in that, The first model is a residual network model; the second model is an RNN, a long short-term memory network (LSTM), or a gated recurrent unit (GRU).

4. The method according to claim 1, characterized in that, The first and second models update their parameters through backpropagation using the loss value; the loss value is calculated using the following formula: In the formula: The loss value. The regularization coefficient is... For standard cross-entropy, This is a regularization term.

5. The method according to claim 4, characterized in that, The regularization term is the square of the L2 norm of the first model parameter vector and the second model parameter vector.

6. The method according to claim 1, characterized in that, The image input to the first model during the training phase is enhanced; the enhancement process includes one or more of the following: random flipping, random occlusion, and random rotation.

7. An industrial product visual inspection system, characterized in that, The system includes an external integration subsystem, a classification detector, and a database; The external integration subsystem is configured to call the message center system and receive feedback from the message center system. The message center system is configured to control the startup of the machine system. The machine control system is configured to control the startup of the motor, camera, and conveyor belt containing the industrial products to obtain the images of the industrial products to be inspected. The classification detector is configured to divide the image of the industrial product to be detected into several sub-images, and to make adjacent sub-images have a set number of overlap lengths. Using a preset first model, image features of multiple sub-images corresponding to the image to be detected are obtained; using a preset second model, classification features of the sub-images are obtained based on the image features of the sub-images, and soft attention is obtained by an attention mechanism based on each classification feature. The soft attention is calculated using the following formula: [The formula is missing from the original text]. The soft attention corresponding to each subimage is denoted as , , , In the formula: , For soft attention learning matrix, For the first Classification features of individual images, For the first Classification features of individual images, For the first Classification features of individual images, Image features of sub-images , The total number of sub-images. For ReLU function, This is an extraction matrix for extracting the classification features of a sub-image from its image features. For the first The sub-image and the first Sequence correlation matrix of sub-images; Based on soft attention and classification features belonging to the same detection image, fused features are obtained to classify products, including: The fusion feature is denoted as ,but: Based on the fusion features, the probability of the industrial product belonging to a category is calculated: The category corresponding to the maximum probability is used as the category of industrial products in the image to be detected; in, As a category of industrial products, For classification matrix, For classification matrix The OK, For the first Soft attention corresponding to each sub-image For the first Classification features of individual images, The total number of sub-images. Indicates belonging to the first The probability of each category; The database is configured to store the image to be detected and the classification result of the image to be detected.

8. A computer-readable storage medium, characterized in that: The computer program is stored that can be loaded by a processor and executed according to any one of claims 1 to 6.

Images