An image detection method, device, program product, and storage medium

Abstract

An embodiment of the present invention discloses an image detection method, device, program product, and storage medium. The method includes: obtaining an image to be processed, and determining sliding window information according to the image to be processed; establishing each sliding window based on the sliding window information, and performing image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain window images of each sliding window; inputting the window images into a pre-determined target defect detection model, and performing image defect detection on the window images through a multi-scale detection module and a multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window images; obtaining a defect detection result of the image to be processed based on the defect detection data, and presenting the defect detection result to the user. The present invention can accurately detect large or small target defects in the image to be processed while saving the computing resources of the device, and improves the accuracy of image defect detection.

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of image processing technology, and in particular, to an image detection method, device, program product, and storage medium. Background Art

[0002] With the development of industrial automation and intelligence, the requirements for image defect detection are getting higher and higher. In many fields, such as semiconductor manufacturing, power inspection, and material inspection, high-resolution images are widely used in defect detection.

[0003] The current image detection method for small defects is to first increase the resolution of the image to be processed, so as to retain more detailed information of the image to be processed, which helps the defect detection model to more clearly capture the features of small defects. Then, the defect detection model is used to detect the defects in the image to be processed with increased resolution. In this way, the direct input of high-resolution images will significantly increase the computational burden and storage requirements of the model, resulting in the model being difficult to run on resource-constrained embedded devices or in real-time applications. And the model needs to process larger-sized input feature maps, and the training time and video memory requirements will also increase significantly. Summary of the Invention

[0004] Embodiments of the present invention provide an image detection method, device, program product, and storage medium, which, on the basis of ensuring the accuracy of image defect detection, do not directly input high-resolution images into the defect detection model, effectively reducing the occupancy of video memory and at the same time improving the inference efficiency of the image defect detection model.

[0005] In a first aspect, embodiments of the present invention provide an image detection method, including:

[0006] Obtain an image to be processed, and determine sliding window information according to the image to be processed;

[0007] Establish each sliding window based on the sliding window information, and perform image extraction on the image to be processed through each sliding window in a preset sliding direction to obtain window images of each sliding window;

[0008] Input the window images into a pre-determined target defect detection model, and perform image defect detection on the window images through a multi-scale detection module and a multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window images;

[0009] Obtain a defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to the user.

[0010] In a second aspect, embodiments of the present invention provide an image detection device, the device includes:

[0011] A data acquisition module, configured to acquire an image to be processed and determine sliding window information based on the image to be processed;

[0012] A first processing module, configured to establish each sliding window based on the sliding window information, and perform image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain window images of each sliding window;

[0013] A second processing module, configured to input the window image into a pre-determined target defect detection model, and perform image defect detection on the window image through a multi-scale detection module and a multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window image;

[0014] A result determination module, configured to obtain a defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to a user.

[0015] In a third aspect, an embodiment of the present invention further provides an electronic device, where the electronic device includes a memory, a processor, and a computer program stored on the memory and executable on the processor, and when the processor executes the program, the image detection method described in any one of the embodiments of the present invention is implemented.

[0016] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the image detection method described in any one of the embodiments of the present invention is implemented.

[0017] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, and when the computer program is executed by a processor, the image detection method described in any one of the embodiments of the present invention is implemented.

[0018] In the embodiments of the present invention, an image to be processed is obtained, and sliding window information is determined according to the image to be processed; based on the sliding window information, each sliding window is established, and the image to be processed is extracted through each sliding window in a preset sliding direction to obtain window images of each sliding window; the window images are input into a pre-determined target defect detection model, and the multi-scale detection module and multi-scale fusion module of the target defect detection model are used to detect image defects in the window images to obtain defect detection data corresponding to the window images; based on the defect detection data, a defect detection result of the image to be processed is obtained, and the defect detection result is displayed to the user. The method of the embodiments of the present invention does not directly input a high-resolution image into the target defect detection model, but accurately and comprehensively extracts the image information of the image to be processed through multiple sliding windows. By determining the sliding window information according to the image to be processed, the diversity and comprehensiveness of the information input into the target defect detection model are ensured. While saving the computing resources of the device, through the multi-scale detection module and multi-scale fusion module, large or small target defect detection can be performed on the image to be processed, improving the accuracy and efficiency of image defect detection, and at the same time improving the inference efficiency of the image defect detection model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is the first flowchart of an image detection method provided by the embodiments of the present invention;

[0021] Figure 2 It is a schematic diagram of extracting local image information by using a sliding window provided by the embodiments of the present invention;

[0022] Figure 3 It is the second flowchart of an image detection method provided by the embodiments of the present invention;

[0023] Figure 4 It is a schematic diagram of the structure of the target defect detection model provided by the embodiments of the present invention;

[0024] Figure 5 It is a schematic diagram of the structure of the multi-scale fusion module provided by the embodiments of the present invention;

[0025] Figure 6 It is the flowchart of the model training method provided by the embodiments of the present invention;

[0026] Figure 7Schematic structural diagram of an image detection device provided by an embodiment of the present invention;

[0027] Figure 8 Schematic structural diagram of an electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0028] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present invention, rather than limiting the present invention. Additionally, it should be noted that for the sake of description, only parts related to the present invention rather than all structures are shown in the drawings.

[0029] Figure 1 The first flowchart of an image detection method provided by an embodiment of the present invention. The method of the embodiment of the present invention can, on the basis of ensuring the accuracy of image defect detection, without directly inputting a high-resolution image into the defect detection model, effectively reducing the occupancy of video memory and improving the efficiency of image defect detection. This method can be executed by an image detection device provided by an embodiment of the present invention, and the device can be implemented in software and / or hardware. The following embodiments will take the integration of the device in an electronic device as an example for illustration. The electronic device can be a server or a computer device, etc. Refer to Figure 1 , and the method can specifically include the following steps:

[0030] Step 101: Obtain an image to be processed, and determine sliding window information according to the image to be processed.

[0031] Among them, the sliding window information includes the number of sliding windows and the size of each sliding window. The image to be processed is an image for which target defect detection needs to be performed. The sliding window is used to extract local information of the image to be processed. Different images to be processed can be set with different numbers and different sizes of sliding windows.

[0032] Specifically, when target defect detection needs to be performed on the image to be processed, the server can obtain the image to be processed uploaded by the user and requirement information, and determine the number of sliding windows and the size of each sliding window according to the size of the image to be processed and the requirement information. In order to capture information of different scales, multiple sliding windows of different sizes can be generated by means of external expansion and equal division. The sliding windows in this solution are all square. In an embodiment of this solution, optionally, determining the sliding window information according to the image to be processed includes: determining the side length of the smallest side of the image to be processed as the side length of the largest sliding window; determining the side lengths of the remaining sliding windows based on the pre-determined side length of the smallest sliding window, the pre-determined number of sliding windows, and the side length of the largest sliding window.

[0033] Among them, multiple sliding windows of different scales include a maximum sliding window with the largest size, a minimum sliding window with the smallest size, and several intermediate candidate windows (the remaining sliding windows). To ensure comprehensive coverage of defect information at different scales, the side length of the maximum sliding window can be set to the minimum side length of the image to be processed. The server can pre-determine the side length of the minimum sliding window and the number of sliding windows based on big data in the field, etc. Of course, it can also be dynamically determined according to the specific needs of the user and the size of the image to be processed.

[0034] In an alternative embodiment, after determining the size of the maximum sliding window, the size of the minimum sliding window, and the number of sliding windows, the sizes of the remaining sliding windows can be determined by means of equal-proportion scaling. Or, the sizes of the remaining sliding windows can also be determined by means of equal-interval division: , . represents the side length of the maximum sliding window, represents the side length of the minimum sliding window, i represents the i-th window, represents the side length of the i-th window, and N represents the number of sliding windows (including the maximum sliding window and the minimum sliding window). Through the above steps, sliding windows of different sizes can be accurately set according to the image to be processed, so as to accurately and comprehensively extract the information of the image to be processed through multiple sliding windows.

[0035] Step 102: Establish each sliding window based on the sliding window information, and perform image extraction on the image to be processed through each sliding window according to the pre-set sliding direction to obtain the window image of each sliding window.

[0036] Among them, the window image is a partial image of the image to be processed extracted by the sliding window. Specifically, after determining the number of sliding windows and the sizes of each sliding window, arrange each sliding window according to the size of the sliding window. Determine the sliding step according to the side length of the minimum sliding window, place the center point of the minimum sliding window at the sliding starting point, and slide each sliding window according to the pre-set sliding direction (such as from left to right and from top to bottom) and the sliding step to obtain the window image of each sliding window corresponding to each sliding step. In the embodiments of this solution, optionally, performing image extraction on the image to be processed through each sliding window according to the pre-set sliding direction to obtain the window image of each sliding window includes the following steps A1 - step A2:

[0037] Step A1: Determine the center point of the minimum sliding window as the center point of all sliding windows, and arrange each sliding window according to the center points of all sliding windows.

[0038] Specifically, after determining the sizes of the sliding windows, the center point of the smallest sliding window is determined as the center point of all sliding windows. The sliding windows are arranged in sequence in the way of expanding outwards from the smallest sliding window until the largest sliding window is arranged. Exemplarily, Figure 2 is a schematic diagram of extracting local image information using sliding windows provided by an embodiment of the present invention. As Figure 2 shown, Figure 2 includes the image to be processed and 4 sliding windows. Figure 2 The black dashed boxes in

[0039] represent the sliding windows. The sliding window with the smallest size is the smallest sliding window. Taking the center of the smallest sliding window as the center of all sliding windows, all sliding windows are arranged, where the side length of the largest sliding window is the same as the smallest side length of the image to be processed.

[0040] Step A2: Determine the side length of the smallest sliding window as the sliding step; determine the sliding starting point and the sliding ending point according to the image to be processed and the smallest sliding window, and slide each sliding window according to the sliding direction and the sliding step to obtain the window images extracted by each sliding window corresponding to each step. Figure 2 shown, Figure 2 includes 4 sliding windows, and the size of the smallest sliding window can be . Slide all sliding windows with a sliding step of 640, and 4 window images with different scales can be generated each time of sliding.

[0041] In this solution, the high-resolution image to be processed is processed from top to bottom and from left to right through sliding windows, ensuring comprehensive coverage of the defect information of the image to be processed at different scales, and ensuring the diversity and comprehensiveness of the information input into the target defect detection model.

[0042] Step 103: Input the window images into a pre-determined target defect detection model, and perform image defect detection on the window images through the multi-scale detection module and the multi-scale fusion module of the target defect detection model to obtain the defect detection data corresponding to the window images.

[0043] Among them, the target defect detection model is pre-determined and used to perform target defect detection on the image to be processed according to the image information extracted by the sliding window. The target defect model includes a multi-scale detection module and a multi-scale fusion module. The multi-scale detection module is used to extract features from the window image, and the multi-scale fusion module is used to perform feature fusion and defect detection on the features output by the multi-scale detection module to obtain defect detection data.

[0044] Specifically, after obtaining the N (number of windows) window images of different scales generated M (number of sliding times), the N window images of different scales each time are processed spatially into image features of the same size (for example ). Further, the window images can be grouped according to the different sizes of the window images (the original sizes of the window images before being processed), and the window images are input into the multi-scale detection module according to the groups. The multi-scale detection module performs independent convolution on the window images of each group, and the feature information of different groups does not interfere with each other. In this way, the situation of sequentially traversing and training the inputs of different scales, which increases the training time and inference time of the target defect detection model, is avoided, and the real-time performance of the target defect detection model is improved. Through the multi-scale detection module, the convolved features are fused at different scales (in this solution, the scales are divided according to the sizes of the image features) to obtain the detection features output by the multi-scale detection module. After obtaining the detection features, the detection features are input into the multi-scale fusion module, and the multi-scale fusion module performs feature fusion and defect detection on the detection features to obtain the defect detection data corresponding to the window images. Through the multi-scale sliding window in this solution, the target defect detection model only needs to process the input of 4 pieces instead of directly inputting the high-resolution image into the model, effectively reducing the occupancy of the video memory, so that the small defect detection problem of high-resolution images can also be processed on a computer with a common 6G video memory. Taking the high-resolution image with an input of as an example, when directly training the model with 1 training step, it requires 22930M (megabytes) of video memory, while the method of this solution only needs to use 2550M of video memory, reducing the video memory computing resources by 88%.

[0045] In an alternative embodiment, for the window images extracted for each sliding window, all the current window images extracted from the current sliding window are determined as the current feature group; the current feature group is convolved by a multi-scale detection module to obtain a feature tensor of the current feature group; the feature tensor of the current feature group is subjected to feature fusion according to the feature tensors of other feature groups to obtain a fused feature of the current feature group, and the fused features of all feature groups are determined as the detection features output by the multi-scale detection module. The detection features are input into a multi-scale fusion module, and spatial transformation is performed on the detection features based on a pre-determined scaling scale, offset, and spatial transformation unit to obtain transformed features of each channel group corresponding to the detection features; the transformed features of each channel group are subjected to feature splicing and convolution to obtain defect detection data corresponding to the window images.

[0046] Step 104: Obtain a defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to the user.

[0047] Among them, the defect detection result includes the image to be processed with defects marked and defect analysis results (such as defect severity and defect suggestions), etc. Specifically, after obtaining the defect detection data output by the target defect detection model, a rectangular box, a circle, or other shapes are drawn at the defect locations on the image to be processed according to the defect detection data to obtain the image to be processed with defects marked. The defect detection data is analyzed, and the defect type and confidence level of the image to be processed are determined according to each pre-defined defect type, and they are sorted out with the image to be processed with defects marked to obtain the defect detection result. The defect detection result is displayed to the user so that the user can intuitively see the defects in the image to be processed and perform further processing and analysis on the defect detection result as needed.

[0048] The technical solution of this embodiment is to obtain an image to be processed and determine sliding window information based on the image to be processed; establish each sliding window based on the sliding window information, and perform image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain the window image of each sliding window; input the window image into a preset target defect detection model, and perform image defect detection on the window image through the multi-scale detection module and multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window image; obtain the defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to the user. The technical solution of this embodiment does not directly input a high-resolution image into the target defect detection model, but accurately and comprehensively extracts the image information of the image to be processed through multiple sliding windows. By determining the sliding window information according to the image to be processed, the diversity and comprehensiveness of the information input into the target defect detection model are ensured. While saving the computing resources of the device, the multi-scale detection module and multi-scale fusion module accurately perform defect detection on large or small targets of the image to be processed, improving the accuracy and efficiency of image defect detection.

[0049] Figure 3 It is the second flowchart of an image detection method provided by an embodiment of the present invention. This embodiment is a refinement based on the above embodiment. The specific method can be as Figure 3 shown, and the method may include the following steps:

[0050] Step 301, obtain an image to be processed, and determine sliding window information based on the image to be processed.

[0051] The sliding window information includes the number of sliding windows and the size of each sliding window; each sliding window is a square window.

[0052] Step 302, establish each sliding window based on the sliding window information, and perform image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain the window image of each sliding window.

[0053] Step 303, input the window image into the multi-scale detection module, and perform grouping processing on the window image through the multi-scale detection module to obtain the detection features output by the multi-scale detection module.

[0054] Among them, the target defect detection model is pre-determined by the server and is used to perform target defect detection on the image to be processed according to the image information extracted by the sliding window. The target defect model includes a multi-scale detection module and a multi-scale fusion module, and the multi-scale detection module is used to extract features from the window image. In this solution, optionally, performing grouping processing on the window image through the multi-scale detection module to obtain the detection features output by the multi-scale detection module includes the following steps B1 - step B2:

[0055] Step B1: For the window images extracted from each sliding window, determine all the current window images extracted from the current sliding window as the current feature group; perform convolution on the current feature group through the multi-scale detection module to obtain the feature tensor of the current feature group.

[0056] Specifically, after obtaining the window images generated by all sliding windows, group all the window images according to different window sizes to obtain each group of feature groups. Exemplarily, as Figure 2 shown, there are 4 sliding windows with different sizes, then the number of groups of feature groups is 4. Input the 4 groups of feature groups into the multi-scale detection module, and use the multi-scale detection module to perform convolution on the 4 groups of features to obtain the feature tensors corresponding to each group of feature groups.

[0057] Step B2: Perform feature fusion on the feature tensor of the current feature group according to the feature tensors of other feature groups to obtain the fused feature of the current feature group, and determine the fused features of all feature groups as the detection features output by the multi-scale detection module.

[0058] After obtaining the feature tensors of each feature group, perform feature fusion on each group of feature tensors and the feature tensors of other groups respectively to obtain the fused feature corresponding to each feature group, and determine the fused features of all feature groups as the detection features output by the multi-scale detection module.

[0059] Exemplarily, Figure 4 is a schematic structural diagram of the target defect detection model provided by an embodiment of the present invention. As Figure 4 shown, the size of the current window image input (Input) to the multi-scale detection module of the target detection model is BatchSize (the number of samples input to the network simultaneously during one training process) = 1, Channel (number of channels) = 3N (N = 4), Height (height) = 640, Width (width) = 640. Perform convolution on it through the multi-scale detection module to obtain the feature tensors F3, F4, and F5 of the current window image. Among them, the size of F3 is 1, 32N, 80, 80; the size of F4 is 1, 64N, 40, 40; the size of F5 is 1, 128N, 20, 20. After obtaining F3, F4, and F5, perform feature fusion on them to obtain the fused features O3, O4, and O5. The size of O3 is 1, 32N, 80, 80; the size of O4 is 1, 64N, 40, 40; the size of O5 is 1, 128N, 20, 20. Among them Figure 4CT in it represents the connection layer, whose function is to splice two or more feature maps along a specific dimension. C3 represents the cross-stage partial network, which is used to reduce the computational amount and improve the learning ability of the model. Input O3, O4, and O5 into the MSFF (Multi-Scale Fusion Module) to obtain the fused features O3', O4', and O5'. Finally, target defect detection is performed on the image to be processed according to O3', O4', and O5' (that is Figure 4 Detect in it). Figure 4 CONV in it represents convolution, and UP represents upsampling. Through the multi-scale detection module, independent convolution processing is performed on inputs of different scales, reducing the interference of feature information and improving the training and inference efficiency.

[0060] Step 304: Input the detection features into the multi-scale fusion module, and perform multi-scale fusion on the detection features through the multi-scale fusion module to obtain the defect detection data corresponding to the window image.

[0061] The multi-scale fusion module is used to perform feature fusion on the features output by the multi-scale detection module to obtain defect detection data. In this solution, the multi-scale fusion module performs multi-scale fusion on the detection features to obtain the defect detection data corresponding to the window image, including: inputting the detection features into the multi-scale fusion module, calculating the scaling scale and offset corresponding to the detection features, and performing spatial transformation on the detection features based on the scaling scale, offset, and spatial transformation unit to obtain the transformed features of each channel group corresponding to the detection features; performing feature splicing and convolution on the transformed features of each channel group to obtain the defect detection data corresponding to the window image.

[0062] The scaling scale is a parameter set in advance for magnifying or reducing the detection features; the offset is a parameter set in advance for translating the detection features, which is used to adjust the position of the detection features. Specifically, in the standard convolution operation, the convolution kernel calculates on all input channels and then accumulates the results into the output channel. In this solution, the input channels are divided into multiple groups, and each group uses an independent convolution kernel for calculation. Figure 5 It is the structural schematic diagram of the multi-scale fusion module provided by the embodiment of the present invention. As Figure 5 shown, after the input end (Input) of the multi-scale fusion module receives the detection features, it calculates the scaling scale and offset of the detection features of different scales (sizes) relative to the smallest sliding window. Taking N = 4 as an example, according to the scaling scale and offset, the detection features are input into the spatial transformation unit (spatialtransformer networks, STN) through channels, the features output by the STN unit are spliced together, and then convolution is performed on the spliced features, and finally the defect detection data that fully fuses different scales is obtained. Figure 5Concat in it represents concatenation, and CONV represents convolution. By performing spatial transformation on channel features of different scales and concatenating them, it is ensured that the final feature map can fully integrate context information and provide comprehensive feature support for image defect detection.

[0063] Step 305: Obtain the defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to the user.

[0064] In the technical solution of this embodiment, the image to be processed is obtained, and the sliding window information is determined according to the image to be processed. The sliding window information includes the number of sliding windows and the size of each sliding window; each sliding window is a square window. Each sliding window is established based on the sliding window information, and the image to be processed is extracted through each sliding window according to a preset sliding direction to obtain the window image of each sliding window. The window image is input into the multi-scale detection module, and the window image is grouped by the multi-scale detection module to obtain the detection features output by the multi-scale detection module. The detection features are input into the multi-scale fusion module, and the detection features are multi-scale fused by the multi-scale fusion module to obtain the defect detection data corresponding to the window image. The defect detection result of the image to be processed is obtained based on the defect detection data, and the defect detection result is displayed to the user. The technical solution of this embodiment can better learn context information through the multi-scale detection module and the multi-scale fusion module, effectively improving the robustness and accuracy of defect detection in high-resolution images. In addition, for inputs of different scales, there is no need to traverse and loop train, only one training is required, improving the training and inference efficiency of the model.

[0065] Figure 6 It is a flowchart of the model training method provided by an embodiment of the present invention. The specific method can be as Figure 6 shown, and the method may include the following steps:

[0066] Step 601: When the pre-established initial defect detection model does not meet the preset conditions, obtain the training image set.

[0067] Among them, the initial defect detection model is pre-established and has not been trained yet, and it cannot accurately perform target defect detection on high-resolution images. The preset conditions are pre-determined according to domain big data, etc., and are used to determine whether the training of the initial defect detection model is completed, such as whether the number of iterations of model training reaches the preset number, whether the loss function converges, etc. The training image set is used to train the initial defect target model, and the training image set includes an image data set with labels, and the image set contains normal images and defect images. When the initial defect model needs to be trained, the server can obtain the training image set sent by the user.

[0068] Step 602: Input the training image set into the initial defect detection model to obtain the output data of the initial defect detection model. Optimize the parameters of the initial defect detection model based on a preset loss function and the output data to obtain a target defect detection model that meets the preset conditions.

[0069] Specifically, after obtaining the training image set, input the images in the training image set into the initial defect model to obtain the output data of the initial defect model. Calculate the difference between the output data and the actual labels using the loss function with the following formula:

[0070] ;

[0071] where represent the corresponding loss weights respectively, and in this solution . is the target box localization loss, is the target confidence loss, is the classification loss. Adjust the model parameters according to the value of the loss function to obtain a target defect detection model that meets the preset conditions. In this solution, the initial learning rate is set to 0.001.

[0072] In the technical solution of this embodiment, when the pre-established initial defect detection model does not meet the preset conditions, obtain the training image set; input the training image set into the initial defect detection model to obtain the output data of the initial defect detection model; optimize the parameters of the initial defect detection model based on a preset loss function and the output data to obtain a target defect detection model that meets the preset conditions. In the technical solution of this embodiment, the loss function is used to guide the model to optimize the parameters, significantly improving the performance of the defect detection model, enabling it to more accurately detect target defects in high-resolution images. It not only improves the accuracy and generalization ability of the model but also provides the ability of flexibility and targeted training.

[0073] Figure 7 FIG. is a schematic structural diagram of an image detection device provided by an embodiment of the present invention. This device is applicable to execute the image detection method provided by an embodiment of the present invention. As Figure 7 shown, this device may specifically include:

[0074] A data acquisition module 701, configured to acquire an image to be processed and determine sliding window information according to the image to be processed;

[0075] A first processing module 702, configured to establish each sliding window based on the sliding window information, and extract images from the image to be processed in a preset sliding direction through each sliding window to obtain window images of each sliding window;

[0076] ​​The second processing module 703 is configured to input the window image into a pre-determined target defect detection model, and perform image defect detection on the window image through the multi-scale detection module and the multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window image;

[0077] The result determination module 704 is configured to obtain a defect detection result of the image to be processed based on the defect detection data, and display the defect detection result to the user.

[0078] Optionally, the sliding window information includes the number of sliding windows and the sizes of the respective sliding windows; the sliding windows are square, and the first processing module 702 is specifically configured to: determine the minimum side length of the image to be processed as the side length of the largest sliding window;

[0079] Based on the pre-determined side length of the minimum sliding window, the pre-determined number of sliding windows, and the side length of the largest sliding window, determine the side lengths of the remaining sliding windows.

[0080] Optionally, the first processing module 702 is further configured to: determine the center point of the minimum sliding window as the center point of all the sliding windows, and arrange the sliding windows according to the center points of all the sliding windows;

[0081] Determine the side length of the minimum sliding window as the sliding step;

[0082] According to the image to be processed and the minimum sliding window, determine the sliding starting point and the sliding ending point, and slide the respective sliding windows according to the sliding direction and the sliding step to obtain window images extracted from the respective sliding windows corresponding to each step.

[0083] Optionally, the second processing module 703 is specifically configured to: input the window image into the multi-scale detection module, and perform grouping processing on the window image through the multi-scale detection module to obtain detection features output by the multi-scale detection module;

[0084] Input the detection features into the multi-scale fusion module, and perform multi-scale fusion on the detection features through the multi-scale fusion module to obtain defect detection data corresponding to the window image.

[0085] Optionally, the second processing module 703 is further configured to: for the window images extracted from the respective sliding windows, determine all the current window images extracted from the current sliding window as the current feature group;

[0086] Perform convolution on the current feature group through the multi-scale detection module to obtain a feature tensor of the current feature group;

[0087] Feature fusion is performed on the feature tensor of the current feature group according to the feature tensors of other feature groups to obtain the fused feature of the current feature group, and the fused features of all feature groups are determined as the detection features output by the multi-scale detection module.

[0088] Optionally, the second processing module 703 is further configured to: input the detection features into the multi-scale fusion module, calculate the scaling scale and offset corresponding to the detection features, and perform spatial transformation on the detection features based on the scaling scale, the offset, and the spatial transformation unit to obtain the transformed features of each channel group corresponding to the detection features;

[0089] Perform feature splicing and convolution on the transformed features of each channel group to obtain the defect detection data corresponding to the window image.

[0090] Optionally, before obtaining the image to be processed, the second processing module 703 is further configured to: when the pre-established initial defect detection model does not meet the preset conditions, obtain a training image set;

[0091] Input the training image set into the initial defect detection model to obtain the output data of the initial defect detection model;

[0092] Optimize the parameters of the initial defect detection model based on a preset loss function and the output data to obtain a target defect detection model that meets the preset conditions.

[0093] The image detection device provided by the embodiments of the present invention can execute the image detection method provided by any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. The content not described in detail in this embodiment can be referred to the description in any method embodiment of the present invention.

[0094] The embodiments of the present invention also provide a computer program product.

[0095] The various embodiments of the systems and techniques described above in this specification can be implemented in digital electronic circuitry, integrated circuit systems, field programmable gate arrays (FPGA), application specific integrated circuits (ASIC), application specific standard products (ASSP), systems on a chip (SOC), complex programmable logic devices (CPLD), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer program products, which can include one or more computer programs, the one or more computer programs being executable and / or interpretable on a programmable system including at least one programmable processor, the programmable processor being a special-purpose or general-purpose programmable processor, receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting the data and instructions to the storage system, the at least one input device, and the at least one output device.

[0096] Figure 8 The structural schematic diagram of an electronic device provided for an embodiment of the present invention is shown in reference Figure 8 , Figure 8 The electronic device 12 shown is merely an example and should not impose any limitation on the functions and usage scope of the embodiments of this application. As Figure 8 shown, the electronic device 12 is presented in the form of a general-purpose computing device. The components of the electronic device 12 can include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0097] The bus 18 represents one or more of several types of bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus using any of the various bus architectures. By way of example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0098] The electronic device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the electronic device 12, including volatile and non-volatile media, removable and non-removable media.

[0099] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The electronic device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media ( Figure 8 not shown, typically referred to as a "hard disk drive"). Although Figure 8 not shown in the figure, a disk drive for reading and writing on removable non-volatile disks (such as a "floppy disk"), and an optical disk drive for reading and writing on removable non-volatile optical disks (such as CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to the bus 18 through one or more data media interfaces. The memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the embodiments of the present application.

[0100] A program / utility 40 having a set (at least one) of program modules 46 can be stored, for example, in the memory 28. Such program modules 46 include, but are not limited to, an operating system, one or more application programs, other program modules, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules 46 generally execute the functions and / or methods in the embodiments described in the present application.

[0101] The electronic device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, etc.), and can also communicate with one or more devices that enable a user to interact with the electronic device 12, and / or communicate with any device that enables the electronic device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through the input / output (I / O) interface 22. Moreover, the electronic device 12 can also communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) through the network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the electronic device 12 through the bus 18. It should be understood that although Figure 8 not shown in the figure, other hardware and / or software modules can be used in combination with the electronic device 12, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0102] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28. For example, it implements an image detection method provided by an embodiment of the present invention: obtaining an image to be processed, and determining sliding window information based on the image to be processed; establishing each sliding window based on the sliding window information, and performing image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain the window images of each sliding window; inputting the window images into a pre-determined target defect detection model, and performing image defect detection on the window images through the multi-scale detection module and multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window images; obtaining a defect detection result of the image to be processed based on the defect detection data, and presenting the defect detection result to the user.

[0103] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the program is executed by a processor, it implements an image detection method provided by all embodiments of the present invention: obtaining an image to be processed, and determining sliding window information based on the image to be processed; establishing each sliding window based on the sliding window information, and performing image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain the window images of each sliding window; inputting the window images into a pre-determined target defect detection model, and performing image defect detection on the window images through the multi-scale detection module and multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window images; obtaining a defect detection result of the image to be processed based on the defect detection data, and presenting the defect detection result to the user. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electronic device, apparatus, or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and the program can be used by or combined with an instruction-executing electronic device, apparatus, or device

[0104] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction-executing electronic device, apparatus, or device.

[0105] The program code contained on a computer-readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wire, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0106] The computer program code for performing the operations of the present invention may be written in one or more programming languages or combinations thereof. The programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., connected through the Internet using an Internet service provider).

[0107] Note that the above is only the preferred embodiment of the present invention and the technical principles applied. Those skilled in the art will understand that the present invention is not limited to the specific embodiments herein, and various obvious changes, re-adjustments, and substitutions can be made by those skilled in the art without departing from the protection scope of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments. Without departing from the concept of the present invention, more other equivalent embodiments may be included, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. An image detection method, characterized in that, The method includes: obtaining an image to be processed and determining sliding window information according to the image to be processed; establishing each sliding window based on the sliding window information, and performing image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain window images of each sliding window; wherein, each sliding window is a sliding window of multiple different scales, and the sizes of each sliding window are determined by an equal-proportion scaling method; the side length of the largest sliding window is the minimum side length of the image to be processed; inputting the window images into a preset target defect detection model, and performing image defect detection on the window images through a multi-scale detection module and a multi-scale fusion module of the target defect detection model to obtain defect detection data corresponding to the window images, including: inputting the window images into the multi-scale detection module, performing grouping processing on the window images through the multi-scale detection module to obtain detection features output by the multi-scale detection module; inputting the detection features into the multi-scale fusion module, and performing multi-scale fusion on the detection features through the multi-scale fusion module to obtain defect detection data corresponding to the window images; obtaining a defect detection result of the image to be processed based on the defect detection data, and presenting the defect detection result to a user.

2. The method according to claim 1, wherein The sliding window information includes the number of sliding windows and the sizes of each sliding window; The sliding window is square, and determining the sliding window information according to the image to be processed includes: determining the side lengths of the remaining sliding windows based on the side length of a preset minimum sliding window, the number of preset sliding windows, and the side length of the largest sliding window.

3. The method according to claim 2, wherein Performing image extraction on the image to be processed in a preset sliding direction through each sliding window to obtain window images of each sliding window, including: determining the center point of the minimum sliding window as the center point of all sliding windows, and arranging each sliding window according to the center points of all sliding windows; determining the side length of the minimum sliding window as a sliding step; determining a sliding starting point and a sliding ending point according to the image to be processed and the minimum sliding window, and sliding each sliding window according to the sliding direction and the sliding step to obtain window images extracted by each sliding window corresponding to each step.

4. The method according to claim 1, wherein Inputting the window images into the multi-scale detection module, and performing grouping processing on the window images through the multi-scale detection module to obtain detection features output by the multi-scale detection module, including: for the window images extracted by each sliding window, determining all current window images extracted by the current sliding window as a current feature group; performing convolution on the current feature group through the multi-scale detection module to obtain a feature tensor of the current feature group; performing feature fusion on the feature tensor of the current feature group according to the feature tensors of other feature groups to obtain a fusion feature of the current feature group, and determining the fusion features of all feature groups as the detection features output by the multi-scale detection module.

5. The method according to claim 1, wherein The multi-scale fusion module includes a spatial transformation unit. The detection features are input into the multi-scale fusion module, and the multi-scale fusion of the detection features is performed through the multi-scale fusion module to obtain the defect detection data corresponding to the window image, including: The detection features are input into the multi-scale fusion module, the scaling scale and offset corresponding to the detection features are calculated, and spatial transformation is performed on the detection features based on the scaling scale, the offset, and the spatial transformation unit to obtain the transformed features of each channel group corresponding to the detection features; Feature splicing and convolution are performed on the transformed features of each channel group to obtain the defect detection data corresponding to the window image.

6. The method according to claim 1, characterized in that, Before obtaining the image to be processed, the method further includes: When the pre-established initial defect detection model does not meet the preset conditions, a training image set is obtained; The training image set is input into the initial defect detection model to obtain the output data of the initial defect detection model; Based on the preset loss function and the output data, parameter optimization is performed on the initial defect detection model to obtain a target defect detection model that meets the preset conditions.

7. A computer program product, comprising a computer program, characterized in that, The computer program, when executed by a processor, implements an image detection method according to any one of claims 1-6.

8. An electronic device, the electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image detection method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the image detection method according to any one of claims 1-6.

Images