Image cropping method and image defect detection method

Abstract

The application provides an image cutting method and an image defect detection method; the image cutting method comprises the following steps: calculating an estimated display memory occupied by a to-be-cut image in a target process; comparing the estimated display memory with maximum display memory of an electronic device; when the estimated display memory is greater than or equal to the maximum display memory, obtaining a cutting parameter according to a current size of the to-be-cut image; and performing cutting processing on the to-be-cut image according to the cutting parameter, so that an estimated display memory occupied by a cut image in the target process is less than the maximum display memory; the application can find a suitable cutting parameter through intelligent algorithm design for electronic devices with different display memories and to-be-cut images with different resolution sizes, so as to perform cutting processing on the to-be-cut image according to the cutting parameter.

Description

Technical Field

[0001] This invention relates to the field of industrial automation, to image processing, and particularly to image segmentation, specifically an image cropping method and an image defect detection method. Background Technology

[0002] In the defect detection process of industrial automation, image segmentation algorithms based on deep learning principles, such as U-Net and the DeepLab series, are now widely used. These algorithms have achieved good results in many practical inspection projects, thus enabling the implementation of many defect detection projects.

[0003] However, not all projects can achieve ideal results. In some projects with high detection difficulty, even the best deep learning algorithms cannot guarantee high detection accuracy, such as the detection of small defects at high resolutions. The conventional approach to deep learning-based image segmentation involves compressing the image to a relatively small resolution (e.g., 512×512) before forward propagation through a neural network. However, compressing high-resolution images to a smaller value results in significant pixel loss. If the defect to be detected is very small, the compressed image will contain almost no information about the defect. Even the best deep learning algorithm cannot guarantee high accuracy for defects that are almost "invisible." On the other hand, maintaining the original resolution or a higher resolution without compression can lead to insufficient GPU memory for many industrial automation devices. For example, if the original image resolution is 4096×4096 or higher, training the ResUnet34 model on a GPU with only 12GB of memory will result in memory overflow. Therefore, regardless of the approach, the ultimate requirement of a high detection rate for small defects cannot be achieved.

[0004] In summary, given the limited equipment available in the field of industrial automation, providing an image segmentation algorithm that enables high-detection-rate small defect detection has become a major challenge. Summary of the Invention

[0005] The purpose of this invention is to provide an image cropping method and an image defect detection method to solve the problems mentioned in the background art.

[0006] In a first aspect, the present invention provides an image cropping method applied to an electronic device. The image cropping method includes: calculating the estimated video memory occupied by the image to be cropped during the target process; comparing the estimated video memory with the maximum video memory of the electronic device; when the estimated video memory is greater than or equal to the maximum video memory, obtaining cropping parameters based on the current size of the image to be cropped; and cropping the image to be cropped according to the cropping parameters so that the estimated video memory occupied by the cropped image during the target process is less than the maximum video memory.

[0007] This invention provides an image cropping method that can find suitable cropping parameters through intelligent algorithm design for electronic devices with different video memory and images of different resolutions, and then perform cropping processing on the image to be cropped according to the cropping parameters. By applying this image cropping method to image defect detection (i.e. when the target process is defect detection), it can automatically realize the segmentation processing of the image to be cropped, thereby achieving a high detection rate of defects in the image to be cropped. This image segmentation method does not require manual design of cropping parameters, lowers the threshold for use, and improves the efficiency of image segmentation.

[0008] In one implementation of the first aspect, the formula for calculating the estimated video memory occupied by the image to be cropped during the target process is as follows:

[0009] M = k × S + b;

[0010] Wherein, S represents the area of ​​the image to be cropped; M represents the estimated video memory; k and b are both training parameters; the method for obtaining the training parameters includes: inputting input images of different sizes to a target model on the electronic device, so that the target model sequentially executes the target process on the input images; obtaining the video memory value occupied by the input image when the target model executes the target process on the input image; and fitting and obtaining the training parameters based on the video memory value and the area of ​​the input image.

[0011] This implementation provides a reliable, end-to-end method for accurate estimation of video memory. For the target model and electronic device, the algorithm obtains the functional relationship between the estimated video memory and the area of ​​the image by using the input image and fitting methods. This allows the algorithm to accurately calculate the estimated video memory occupied by the image to be cropped in the subsequent target process based on the size of the image.

[0012] In one implementation of the first aspect, the step of sequentially performing the target process on the input image through the target model includes: sequentially performing the target process on the input image in ascending order of size through the target model.

[0013] In this implementation, the target model performs the target process on the input image in ascending order of size to avoid the target model processing a large input image at the beginning, which would cause the electronic device's video memory to overflow.

[0014] In one implementation of the first aspect, obtaining the cropping parameters based on the current size of the image to be cropped includes: Step 1, determining the number of cropping segments to be cropped in one direction to be two based on the current size; Step 2, calculating the cropping length to be cropped in one direction based on the current size and the number of cropping segments; Step 3, updating the cropping length to align the cropping length upwards to an integer multiple of 32, and obtaining the updated length; Step 4, calculating the overlap length between the two cropped images based on the updated length and the number of cropping segments; Step 5, obtaining the ratio of the overlap length to the updated length; Step 6, comparing the ratio with a set threshold; if the ratio is less than the set threshold, then proceeding to Step 7, increasing the updated length by 32, and repeating Steps 4 to 6 based on the current updated length until the ratio is reached. The target length is determined by the ratio being greater than or equal to the set threshold. The target length is the update length corresponding to the ratio being greater than or equal to the set threshold. Step 8: Determine whether the estimated video memory occupied by the cropped image in the target process is less than the maximum video memory at the target size. The target size includes the target length and the length of the image to be cropped in another direction. Step 9: If the determination result is negative, increment the number of cropped images in one direction and use the current cropped image as the image to be cropped. Repeat steps 1 to 8 until the determination result is positive to obtain the cropping parameters. The cropping parameters include at least the number of cropped images in both directions, the length of the cropped image in both directions, and the overlap length of the cropped images in both directions between adjacent cropped images.

[0015] This implementation provides a method for intelligently obtaining cropping parameters. The cropping parameters obtained by this method enable automated segmentation of the image to be cropped, eliminating the hassle of manually designing cropping parameters, accelerating image segmentation efficiency, and ensuring the quality of the cropped image. This also reduces the adverse effects on subsequent target processes when the cropped image is applied to them.

[0016] In one implementation of the first aspect, the current size includes: a first length of the image to be cropped in one direction and a second length in another direction; determining that the number of cropped segments of the image to be cropped in one direction is two based on the current size includes: determining that the first length is greater than or equal to the second length based on the current size, so as to determine that the number of cropped segments of the image to be cropped in one direction is two.

[0017] In one implementation of the first aspect, in step four, the formula for calculating the overlap length is:

[0018] ;

[0019] in, Indicates the overlap length; This indicates the current update length; The number of cut pieces is two; The length of the current image to be cropped in one direction is indicated by the current size.

[0020] Secondly, the present invention provides an image defect detection method based on the above-mentioned image cropping method, applied to an electronic device. The image defect detection method is used to detect defects in an image to be cropped. The image defect detection method includes: performing a target process on the cropped image using a target model on the electronic device to obtain a detection result image corresponding to the cropped image; wherein, the target model is a trained image defect detection model; the target process is defect detection; summarizing the detection result images according to the position of the cropped image in the image to be cropped to obtain a final result image of defect detection of the image to be cropped; during the summarization, averaging is performed on the overlapping portion between two adjacent detection result images.

[0021] In this invention, by applying the above-mentioned image cropping method to image defect detection, a high detection rate of defect detection can be achieved, reducing the threshold for using image defect detection and improving the accuracy and efficiency of image defect detection. In addition, by averaging the overlapping part between two detection result images, when a defect falls at the overlapping area, as long as one detection result image detects the defect through the image defect detection model, even if the defect is in other cropped images and is not recognized by the model.

[0022] In one implementation of the second aspect, before the step of performing the target process on the cropped image using the target model on the electronic device, the image defect detection method further includes: acquiring a training image; the training image is acquired after performing the image cropping method on the original image; wherein, when performing the image cropping method on the original image, the original image is used as the image to be cropped; the training image corresponds to the cropped image; and the original image defect detection model is trained using the training image to obtain the target model.

[0023] In one implementation of the second aspect, acquiring the training image includes: determining whether the new cropping starting point exceeds the boundary, and if the new cropping starting point does not exceed the boundary, performing the image cropping method on the original image using the new cropping starting point to acquire the training image; the new cropping starting point is the original starting point plus a random offset; and / or after the step of acquiring the training image and before the step of training the original image defect detection model using the training image, the image defect detection method further includes: randomly scaling the training image, acquiring the scaled image, restoring the scaled image to the size corresponding to the training image, acquiring the data-augmented training image, and using the data-augmented training image to train the original image defect detection model; wherein the center point of the scaled image is consistent with the center point of the training image.

[0024] In this implementation, by performing data augmentation processing on the training images during the training phase, the learning effect of the image defect detection model is further enhanced, and the model's adaptability to defects in images is improved.

[0025] In one implementation of the second aspect, training the original image defect detection model using the training images includes: determining whether the number of training images used as background images in the training images reaches a preset condition; if the determination result is yes, then randomly deleting a portion of the training images used as background images, and using the training images after the random deletion to train the original image defect detection model.

[0026] This implementation provides a mechanism for filtering background images, which further enhances the defect detection capability of the image defect detection model.

[0027] As described above, the image cropping method and image defect detection method of the present invention have the following beneficial effects:

[0028] (1) Compared with the prior art, the present invention provides an intelligent image cropping algorithm that can truly realize automated image cropping, eliminating the need for manual design of cropping parameters and lowering the threshold for users.

[0029] (2) For small defect detection projects, by applying the image cropping method provided by the present invention to image defect detection, the detection efficiency and the accuracy of defect detection are improved.

[0030] (3) The present invention designs data augmentation, which further improves the learning effect of the image defect detection model, enabling the model to adapt to defects of various sizes and different fields of view, thereby ensuring a more stable defect detection rate.

[0031] (4) The present invention designs a mechanism for filtering out background images, which further improves the problem of imbalance between positive and negative samples and enhances the model's ability to learn small defect features under the clipping mode. Attached Figure Description

[0032] Figure 1 The flowchart shown is an example of the image cropping method described in an embodiment of the present invention.

[0033] Figure 2 The flowchart shown is a process for image defect detection according to an embodiment of the present invention.

[0034] Figure 3 The image shown is a schematic diagram of the image to be cropped according to an embodiment of the present invention.

[0035] Figure 4 Displayed as Figure 3 A schematic diagram of the corresponding cropped image.

[0036] Figure 5 Displayed as Figure 4 The corresponding test result image.

[0037] Figure 6 Displayed as Figure 3 The corresponding final result image. Detailed Implementation

[0038] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0039] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. The illustrations only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0040] When dealing with small defects at high resolution, the most effective and direct approach is to first crop the original image, for example, by cropping it into nine parts, with three parts cropped horizontally and three parts cropped vertically. Then, the cropped images are not compressed or are only slightly compressed before training the image defect detection model. This ensures that the information of small defects is almost uncompressed, thereby improving the detection rate of small defects.

[0041] The above process requires customizing many parameters based on the device and the resolution of the original image, including the size and number of cropped images, and the overlap between the two cropped images. A high cropped image resolution may still cause memory overflow, while a low resolution may disrupt the position and continuity of some features, affecting detection performance. If the overlap between the two cropped images is too small, and a defect to be detected happens to fall at the cropping point, neither image may be able to record the defect completely. If the overlap is too large, it will lead to data redundancy and affect detection efficiency. Therefore, for each device and each project, using this cropping mode for preprocessing requires specific parameter settings and optimizations. This process is time-consuming and labor-intensive, especially for users without image processing algorithm backgrounds. Directly using image segmentation training software to perform a high-resolution small defect detection project presents a significant barrier to entry.

[0042] Therefore, to address this problem, this invention proposes an intelligent automatic cropping mode that can find suitable cropping parameters (including cropping image resolution, number of copies, and overlap size) through intelligent algorithm design for images of any device with different video memory and different resolution sizes. Then, it performs training and prediction, and finally achieves high detection rate of small defects. In this way, the preprocessing of cropped images (i.e., cropping processing) can be automated without the need for manual parameter design, which lowers the threshold for use and improves the efficiency of preprocessing.

[0043] See Figures 1 to 6The following embodiments of the present invention provide an image cropping method and an image defect detection method. Compared with the prior art, the intelligent image cropping algorithm provided by the present invention can truly realize automated image cropping, eliminating the need for manual design of cropping parameters and lowering the threshold for users. For small defect detection projects, by applying the image cropping method provided by the present invention to image defect detection, the detection efficiency and accuracy of defect detection are improved. The present invention designs data augmentation to further improve the learning effect of the image defect detection model, enabling the model to adapt to defects of various sizes and under different field of view, thereby ensuring a more stable defect detection rate. The present invention designs a mechanism for filtering background images, further improving the problem of imbalance between positive and negative samples and enhancing the model's ability to learn small defect features under the cropping mode.

[0044] The technical solutions of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] First, based on the problems mentioned above, this invention proposes two basic principles for the parameters of the cropped image.

[0046] 1. The first principle to be maintained in the cropping process of the image to be cropped (the principle of the number of cropping parts) is to avoid cropping if possible, and to crop as little as possible.

[0047] Because the definition of some defects is not only determined by the pixels of the defect itself, but also by the pixel distribution of the surrounding area, and the pattern of some defects can only be shown by a larger area, cropping a large image into multiple smaller images in this case will disrupt the continuity of the pixel distribution around the defect and affect the detection of small defects, or the algorithm may identify suspected small defects as defects, leading to over-detection. Therefore, the principle of this intelligent cropping process is to crop as little as possible.

[0048] However, if the original image with a high resolution is not cropped, it will often cause the graphics card to overflow on devices with small video memory, making training impossible. Therefore, the parameter for selecting the resolution of the cropped image must first ensure that the graphics card's video memory does not overflow. Thus, the first premise of image cropping algorithm design is to select the largest possible cropping image based on the maximum value of the device's video memory, while ensuring that the graphics card's video memory does not overflow.

[0049] 2. The second principle concerns the overlap between the two cropped images.

[0050] If the overlap size is too large, it will lead to repeated calculation of many pixel data, increasing the detection time and affecting the detection efficiency; if the overlap size is too small, if the detected defect happens to fall at the clipping point, neither of the two images will be able to record the defect completely, which will affect the subsequent detection of small defects.

[0051] Therefore, the parameter setting for the overlap size of the two cropped images should be neither too large nor too small.

[0052] Based on the two cutting principles mentioned above, this invention proposes a corresponding algorithm design that can ultimately achieve preprocessing of the cut image in fully automatic mode, thereby improving the accuracy of small defect detection.

[0053] like Figure 1 As shown, in one embodiment, the present invention provides an image cropping method applied to an electronic device, the image cropping method comprising:

[0054] Step S11: Calculate the estimated video memory occupied by the image to be cropped during the target process.

[0055] It should be noted that, in the above embodiment of image defect detection, the "target process" is defect detection.

[0056] In one embodiment, the formula for calculating the estimated video memory occupied by the image to be cropped during the target process is as follows:

[0057] M = k × S + b;

[0058] Where S represents the area of ​​the image to be cropped; M represents the estimated video memory; and k and b are both training parameters.

[0059] In this embodiment, the method for obtaining the training parameters includes:

[0060] Step S111: Input images of different sizes are collected and input to the target model on the electronic device, so that the target process is executed on the input images sequentially through the target model.

[0061] In one embodiment, the step of sequentially performing the target process on the input image through the target model includes: sequentially performing the target process on the input image in ascending order of size through the target model.

[0062] It should be noted that, in the above embodiment of image defect detection, the "target model" is an image defect detection model.

[0063] Step S112: Obtain the video memory value occupied by the input image when the target model performs the target process on the input image.

[0064] Specifically, the video memory value in step S112 is obtained in real time through software.

[0065] It should be noted that obtaining the video memory value in step S112 in real time through software is a conventional technique in the field, so its working principle will not be described in detail here.

[0066] In one embodiment, the software obtains in real time the video memory required by the size of the input image when the target model performs the target process on the input image, and then performs data cleaning and filtering on the video memory required to obtain the video memory value.

[0067] In one embodiment, for an input image, the target model continues to execute the target process on the next input image only after step S112 is completed, that is, after obtaining the corresponding occupied video memory value. This method can avoid the situation where the electronic device overflows the video memory when the target model executes the target process on the next input image because the size of the next input image is too large, which would cause step S112 to fail.

[0068] Step S113: Based on the memory value and the area of ​​the input image, fit and obtain the training parameters.

[0069] Specifically, by using the memory occupied by input images of different sizes and the area of ​​the input image, the training parameters k and b are obtained by fitting.

[0070] In one embodiment, the training parameters are obtained by fitting using the least squares method.

[0071] Step S12: Compare the estimated video memory with the maximum video memory of the electronic device.

[0072] Step S13: When the estimated video memory is greater than or equal to the maximum video memory, obtain the cropping parameters based on the current size of the image to be cropped.

[0073] In one embodiment, obtaining the cropping parameters based on the current size of the image to be cropped includes:

[0074] Step S131: Determine the number of cropping segments to be cut in one direction for the image to be cropped to be two based on the current size.

[0075] In one embodiment, the current size includes: a first length of the current image to be cropped in one direction and a second length in another direction.

[0076] In this embodiment, determining that the number of cropping segments to be cut in one direction of the image to be cropped is two based on the current size includes: determining that the first length is greater than or equal to the second length based on the current size, so as to determine that the number of cropping segments to be cut in one direction of the image to be cropped is two.

[0077] Specifically, when the first length is greater than or equal to the second length, the image to be cropped is cropped once in one direction.

[0078] It should be noted that the original image to be cropped had only one copy in one direction. After one cropping, the image to be cropped was divided into two copies in that direction, which corresponds to the number of cropped copies mentioned above.

[0079] It should be noted that the "direction" may be the horizontal direction of the image to be cropped or its vertical direction. Regardless of whether it is horizontal or vertical, the working principle is the same: if the horizontal length (corresponding) is greater than or equal to the vertical length, the horizontal length is taken as the first length and the vertical length is taken as the second length (indicating that the subsequent cropping is performed along the horizontal direction); conversely, if the vertical length is greater than the horizontal length, the vertical length is taken as the first length and the horizontal length is taken as the second length (indicating that the subsequent cropping is performed along the vertical direction).

[0080] Step S132: Calculate the cutting length of the image to be cut in one direction based on the current size and the number of cutting parts.

[0081] In one embodiment, the cutting length = first length / number of cuts.

[0082] Step S133: Update the cutting length to align the cutting length upwards to an integer multiple of 32, and obtain the updated length.

[0083] Step S134: Calculate the overlap length between the two cropped images based on the update length and the number of cropped parts.

[0084] In one embodiment, the formula for calculating the overlap length in step S134 is:

[0085] ;

[0086] in, Indicates the overlap length; This indicates the current update length; The number of cut pieces is two; This represents the length of the current image to be cropped in one direction, obtained based on the current size, which corresponds to the first length mentioned above.

[0087] Step S135: Obtain the ratio of the overlap length to the update length.

[0088] Step S136: Compare the set threshold with the ratio.

[0089] It should be noted that the specific value of the threshold is not intended to limit the invention. In practical applications, it can be set according to the specific application scenario. For example, in one embodiment, the threshold is set to 0.125.

[0090] If the ratio is less than the set threshold, it indicates that the overlap length is too small, and then step S137 is executed.

[0091] Step S137: After increasing the update length by 32, repeat steps S134 to S136 based on the current update length until the ratio is greater than or equal to the set threshold to determine the target length.

[0092] In this embodiment, the target length is the update length corresponding to the ratio being greater than or equal to the set threshold (indicating that the overlap length is large enough to ensure that the feature distribution around the small defect is completely preserved in at least one cropped image).

[0093] Step S138: Determine whether the estimated video memory occupied by the cropped image at the target size during the target process is less than the maximum video memory.

[0094] In this embodiment, the target size includes: the target length and the length of the image to be cropped in another direction (i.e., the second length).

[0095] Specifically, after obtaining the target size, the estimated video memory occupied by the cropped image at that target size is obtained according to the above-mentioned formula for calculating the estimated video memory. Then, it is compared with the maximum video memory to determine whether it is less than the maximum video memory.

[0096] If the result is negative, proceed to step S139.

[0097] Step S139: Increment the number of cropped images in one direction, and use the current cropped image as the cropped image. Repeat steps S131 to S138 until the judgment result is yes, so as to obtain the cropping parameters.

[0098] It should be noted that the original number of images to be cropped in one direction is two (corresponding to the number of cropped parts in step S131 above); each time step S139 is executed, the number of images to be cropped in one direction is updated, that is, it is incremented by one, to obtain a new number of images to be cropped in one direction; specifically, taking the horizontal direction as an example, when step S139 is executed for the first time, the number of images to be cropped in the horizontal direction becomes 3 (2+1), while the number of images to be cropped in the vertical direction remains 1 (no cropping in the vertical direction); assuming that when step S139 is executed for the second time, the direction becomes vertical again, then after executing step S139, the number of images to be cropped in the vertical direction becomes 3 (2+1), while the number of images to be cropped in the horizontal direction remains the same as the 3 obtained when step S139 was executed for the first time.

[0099] Specifically, if the judgment result is negative, it means that the current cropped image will cause memory overflow during the target process at the target size. Therefore, the current cropped image needs to be cropped again. That is, the current cropped image is used as the image to be cropped, and the above steps S131 to S138 are executed again until the judgment result in step S138 is positive, which means that the current cropped image will not cause memory overflow of the electronic device during the target process.

[0100] It should be noted that the "direction" in step S131 above may be the horizontal direction of the image in one cycle, and may become the vertical direction of the image in the next cycle.

[0101] In this embodiment, the cropping parameters include at least: the number of images to be cropped in both directions, the length of the cropped images in both directions, and the overlap length of the cropped images in both directions between adjacent images.

[0102] The number of images to be cropped in both directions is obtained based on step S139; the length of the cropped image in both directions is the target size of the current cropped image when the judgment result in step S138 is yes; the overlap length of two adjacent cropped images in both directions is the overlap length in step S134 when the judgment result in step S138 is yes.

[0103] Step S14: Perform cropping processing on the image to be cropped according to the cropping parameters, so that the estimated video memory occupied by the cropped image in the target process is less than the maximum video memory.

[0104] It should be noted that after cropping the image to be cropped based on the cropping parameters obtained in steps S11 to S13, the resulting cropped image ensures that the subsequent target process can proceed normally (the electronic device will not experience memory overflow).

[0105] The scope of protection of the image cropping method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principle of this invention is included within the scope of protection of this invention.

[0106] like Figure 2 As shown, in one embodiment, the present invention also provides an image defect detection method based on the above-described image cropping method, applied to an electronic device. The image defect detection method is used to detect defects in an image to be cropped, and includes:

[0107] Step S23: Perform a target process on the cropped image using the target model on the electronic device to obtain the detection result image corresponding to the cropped image.

[0108] In this embodiment, the target model is a trained image defect detection model; the target process is defect detection.

[0109] It should be noted that this image defect detection model uses existing technology in the field, so it will not be described in detail here.

[0110] Step S24: Summarize the detection result images according to the positions of the cropped images in the image to be cropped, so as to obtain the final result image of defect detection of the image to be cropped.

[0111] It should be noted that after obtaining the detection result image corresponding to each cropped image, each detection result image needs to be summarized and placed in the correct position on the corresponding original image to be cropped. For example, in one embodiment, the number of cropped images in the horizontal direction is 3 and the number of cropped images in the vertical direction is also 3. Then, the detection result image of the cropped image in the upper left corner of the image to be cropped, obtained by the target model, needs to be used as the upper left corner of the final result image. The detection result images of the remaining eight regions are also the same. However, the overlapping part between the two cropped images also needs to be considered.

[0112] In this embodiment, when summarizing, the overlapping portion between two adjacent detection result images is averaged.

[0113] Specifically, the averaged result is used as the final result. The advantage of doing this is that if a small defect falls at the overlap, as long as one cropped image detects the defect through the target model, even if the defect is not recognized by the target model in other cropped images, the defect can be identified by the target model. Finally, the final grayscale result image (i.e. the final result image) can be identified by applying a low threshold.

[0114] like Figures 3 to 6 As shown, Figure 3 It is the image to be cropped. Figure 4 These are 9 cropped images. Figure 5 These are the detection results images corresponding to 9 cropped images. Figure 6 It is Figure 5 The final result image is a summary of the nine detection result images according to their corresponding positions. It can be seen that after the preprocessing of the cropping mode, the model can ensure that all small defects in the image to be cropped can be detected, thus achieving the requirement of detecting small defects in high-resolution images.

[0115] like Figure 2 As shown, in one embodiment, before the step of performing a target process on the cropped image using a target model on the electronic device, the image defect detection method further includes:

[0116] Step S21: Obtain training images.

[0117] In this embodiment, the training image is obtained by performing the image cropping method on the original image; wherein, when performing the image cropping method on the original image, the original image is used as the image to be cropped; the training image corresponds to the cropped image.

[0118] In one embodiment, obtaining the training image includes: determining whether the new cropping starting point exceeds the boundary, and if the new cropping starting point does not exceed the boundary, performing the image cropping method on the original image using the new cropping starting point to obtain the training image.

[0119] In this embodiment, the new starting point for cutting is the original starting point plus a random offset.

[0120] In one embodiment, the original starting point is the top left corner of the original image, i.e., coordinates (0,0).

[0121] In one embodiment, the random offset is a pair of random numbers with upper and lower limits.

[0122] Specifically, let X and Y be the pair of random numbers with upper and lower limits, and let (X,Y) be the random offset. Then the new starting point for cutting is (0+X,0+Y).

[0123] It should be noted that the specific values ​​of the random number and the upper and lower limits are not intended to limit the present invention. In practical applications, they can be set according to the specific application scenario.

[0124] It should be noted that, regarding random cropping, under normal circumstances, the cropping image described above is cropped from the top left corner of the image to be cropped, i.e., coordinate (0,0), to obtain the cropped image (whose size is determined by the cropping parameters). If data augmentation for random cropping is applied, the starting point of the cropped image can be modified. Specifically, a pair of random numbers with upper and lower limits is generated as offsets (X,Y). Under normal circumstances, the starting point of the cropping plus this offset (X,Y) is the new starting point. If it is found that the starting point exceeds the boundary of the image to be cropped, then no data augmentation will be performed in this training process.

[0125] It should be noted that after determining the starting point for cropping, the cropping process is then carried out. In this way, the same image to be cropped may show complete small defects in some training processes, while in other cases it may show cut-off small defects. This makes the trained model more adaptable to the features of defects in various situations, and the prediction effect will naturally be better.

[0126] Step S22: Train the original image defect detection model using the training images to obtain the target model.

[0127] In one embodiment, after the step of acquiring the training image and before the step of training the original image defect detection model using the training image, the image defect detection method further includes: randomly scaling the training image, acquiring the scaled image, restoring the scaled image to the size corresponding to the training image, acquiring the data-enhanced training image, and using the data-enhanced training image to train the original image defect detection model.

[0128] In this embodiment, the center point of the scaled image is the same as the center point of the training image.

[0129] Specifically, random scaling data augmentation is applied by generating a random number (e.g., between 0.9 and 1.1) and multiplying it by the size of the cropped image under normal conditions. This ensures that the center point of the cropped image after data augmentation is the same as the center point of the original cropped image. Then, it is compressed to the size of the cropped image under normal conditions. Alternatively, the cropped image after data augmentation may be enlarged to the size of the cropped image under normal conditions. In this way, the model can adapt to defects of various sizes and different field of view. The model can better ensure a stable detection rate when facing defects that are different from those in the training set during the prediction stage.

[0130] It should be noted that the data augmentation described above is different from ordinary data augmentation. Ordinary random cropping and random scaling, when performed on the original image (corresponding to the image to be cropped), will inevitably result in the processed area exceeding the boundaries of the original image. However, since the random processing is performed at the location of the cropped image, the area exceeding the cropped image may still be the area of ​​other cropped images. Therefore, the image data is real data and no padding operation is required.

[0131] In one embodiment, training the original image defect detection model using the training images includes:

[0132] Determine whether the number of training images used as background images in the training images meets a preset condition;

[0133] If the judgment result is yes, then a portion of the training image used as the background image is randomly deleted, and the original image defect detection model is trained using the training image after the random deletion.

[0134] In one embodiment, the preset condition is that the ratio of the number of training images used as background images to the total number of training images is greater than or equal to a fixed value.

[0135] It should be noted that the fixed value is preset, and its specific setting is not a limitation of the present invention. In practical applications, it can be set according to the specific application scenario; for example, in one embodiment, the fixed value is set to 1 / 3.

[0136] It should be noted that this embodiment provides an algorithm optimization method for cases where there are few small defects in the image to be cropped. For example, an image to be cropped is cropped into 3×3 smaller images according to the image cropping method described above, resulting in a total of 9 cropped images. However, most, or even all, of the images to be cropped may only have one small defect. In this case, only one of the 9 cropped images has labeled content, while the other 8 are all unlabeled background images. If all the cropped images are directly used for training, the model will learn mostly background images without defects, resulting in a very small proportion of truly effective information learned, which is an imbalance of positive and negative samples. Experiments have shown that in this situation, the model often struggles to learn the unique defect feature in the 9 cropped images. Even if the model doesn't learn it or simply identifies any image as background, its evaluation metrics will still be very high.

[0137] Therefore, it is necessary to perform some screening in this situation. That is, when the proportion of the background image in the cropped image is too high, it is necessary to screen it. There is no need to put all the images into training.

[0138] Specifically, after cropping an original image, if the number of cropped images used as background images (i.e., the corresponding training images) reaches a fixed value or higher than the total number of cropped images, then a portion of these cropped images used as background images will be randomly deleted and will not be used as samples for the current training.

[0139] It should be noted that, through the above-mentioned mechanism for selecting background images, experiments have shown that the ability to detect small defects in the cropping mode has been further improved, verifying the effectiveness of the algorithm design of this invention.

[0140] The scope of protection of the image defect detection method described in this embodiment is not limited to the execution order of the steps listed in this embodiment. Any solution implemented by adding, subtracting, or replacing steps in the prior art based on the principle of this invention is included within the scope of protection of this invention.

[0141] Those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing a processor. The program can be stored in a computer-readable storage medium, which is a non-transitory medium, such as random access memory, read-only memory, flash memory, hard disk, solid-state drive, magnetic tape, floppy disk, optical disk, and any combination thereof. The storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. This available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state drive (SSD)).

[0142] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, or methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules / units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or units may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection of apparatuses or modules or units may be electrical, mechanical, or other forms.

[0143] The modules / units described as separate components may or may not be physically separate. The components shown as modules / units may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules / units can be selected to achieve the objectives of the embodiments of the present invention, depending on actual needs. For example, the functional modules / units in the various embodiments of the present invention may be integrated into one processing module, or each module / unit may exist physically separately, or two or more modules / units may be integrated into one module / unit.

[0144] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0145] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0146] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. An image cropping method, applied to electronic devices, characterized in that, The image cropping method includes: Calculate the estimated video memory occupied by the image to be cropped during the target process; Compare the estimated video memory with the maximum video memory of the electronic device; When the estimated video memory is greater than or equal to the maximum video memory, the cropping parameters are obtained based on the current size of the image to be cropped. The image to be cropped is cropped according to the cropping parameters so that the estimated video memory occupied by the cropped image in the target process is less than the maximum video memory.

2. The image cropping method according to claim 1, characterized in that, The formula for calculating the estimated video memory occupied by the image to be cropped during the target process is as follows: M = k × S + b; Where S represents the area of ​​the image to be cropped; M represents the estimated video memory; k and b are both training parameters; the method for obtaining the training parameters includes: Input images of different sizes are collected and input to the target model on the electronic device, so that the target process is executed on the input images sequentially through the target model; Obtain the video memory value occupied by the input image when the target model performs the target process on the input image; The training parameters are obtained by fitting the memory value and the area of ​​the input image.

3. The image cropping method according to claim 2, characterized in that, The step of sequentially executing the target process on the input images through the target model includes: sequentially executing the target process on the input images in ascending order of size through the target model.

4. The image cropping method according to claim 1, characterized in that, The step of obtaining the cropping parameters based on the current size of the image to be cropped includes: Step 1: Based on the current size, determine the number of cropping segments to be cut in one direction for the image to be cropped to be two; Step 2: Calculate the cutting length of the image to be cut in one direction based on the current size and the number of cutting segments; Step 3: Update the cutting length to align it upwards to an integer multiple of 32, and obtain the updated length; Step 4: Calculate the overlap length between the two cropped images based on the update length and the number of cropped parts; Step 5: Obtain the ratio of the overlap length to the update length; Step 6: Compare the set threshold with the ratio; If the ratio is less than the set threshold, then proceed to step seven, increase the update length by 32, and repeat steps four to six based on the current update length until the ratio is greater than or equal to the set threshold to determine the target length; the target length is the update length corresponding to when the ratio is greater than or equal to the set threshold. Step 8: Determine whether the estimated video memory occupied by the cropped image at the target size during the target process is less than the maximum video memory; the target size includes: the target length and the length of the image to be cropped in another direction; Step 9: If the judgment result is negative, increment the number of cropped images in one direction, and use the current cropped image as the image to be cropped. Repeat steps 1 to 8 until the judgment result is positive to obtain the cropping parameters. The cropping parameters include at least the number of cropped images in two directions, the length of cropped images in two directions, and the overlap length of cropped images in two adjacent directions.

5. The image cropping method according to claim 4, characterized in that, The current size includes: a first length of the current image to be cropped in one direction and a second length in another direction; The step of determining that the number of cropping segments to be cut in one direction of the image to be cropped is two based on the current size includes: determining that the first length is greater than or equal to the second length based on the current size, so as to determine that the number of cropping segments to be cut in one direction of the image to be cropped is two.

6. The image cropping method according to claim 4, characterized in that, In step four, the formula for calculating the overlap length is: ； in, Indicates the overlap length; This indicates the current update length; The number of cut pieces is two; The length of the current image to be cropped in one direction is indicated by the current size.

7. An image defect detection method based on the image cropping method according to any one of claims 1 to 6, applied to electronic devices, characterized in that, The image defect detection method is used to detect defects in an image to be cropped. The image defect detection method includes: The target process is performed on the cropped image using the target model on the electronic device to obtain the detection result image corresponding to the cropped image; wherein, the target model is a trained image defect detection model; and the target process is defect detection. The detection result images are summarized according to the position of the cropped image in the image to be cropped, so as to obtain the final result image of defect detection of the image to be cropped; when summarizing, the overlapping part between two adjacent detection result images is averaged.

8. The image defect detection method according to claim 7, characterized in that, Prior to the step of performing the target process on the cropped image using the target model on the electronic device, the image defect detection method further includes: A training image is obtained; the training image is obtained after performing the image cropping method on the original image; wherein, when performing the image cropping method on the original image, the original image is used as the image to be cropped; the training image corresponds to the cropped image; The original image defect detection model is trained using the training images to obtain the target model.

9. The image defect detection method according to claim 8, characterized in that, The acquisition of the training image includes: determining whether the new cropping starting point exceeds the boundary, and if the new cropping starting point does not exceed the boundary, performing the image cropping method on the original image using the new cropping starting point to acquire the training image; the new cropping starting point is the original starting point plus a random offset; and / or After the step of acquiring the training image and before the step of training the original image defect detection model using the training image, the image defect detection method further includes: randomly scaling the training image, acquiring the scaled image, restoring the scaled image to the size corresponding to the training image, acquiring the data-augmented training image, and using the data-augmented training image to train the original image defect detection model; wherein the center point of the scaled image is consistent with the center point of the training image.

10. The image defect detection method according to claim 8, characterized in that, The step of training the original image defect detection model using the training images includes: Determine whether the number of training images used as background images in the training images meets a preset condition; If the judgment result is yes, then a portion of the training image used as the background image is randomly deleted, and the original image defect detection model is trained using the training image after the random deletion.

Images