Deep learning model generation device and deep learning model generation method

Abstract

To generate a deep learning model robust to a difference in image size.SOLUTION: A deep learning model generation device that generates a deep learning model, the deep learning model including at least an encoder that executes an operation based on a token in which position information of an input image is embedded in each of a plurality of patches generated from the input image, the deep learning model generation device includes a crop image generation unit configured to generate at least one type of crop image from an original image, and an encoder training unit configured to train the encoder using the crop image as the input image, in which the crop image generation unit is configured to change at least one of a size of each crop image and the number of crop images for each training set in the training based on a predetermined probability within a predetermined condition.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a deep learning model generation device and a deep learning model generation method. [Background technology]

[0002] For example, a surface texture inspection device that inspects the surface texture of an object such as a steel plate is an inspection device that uses a deep learning (DL) model to estimate the type and shape of defects from an image of the surface of the object, and outputs these values.

[0003] Known deep learning models that can be applied to the above-mentioned inspection devices include models that are trained to reduce the error between the estimated result and the correct marking image when a defect image (marking image) with labels assigned to each pixel in advance or a defect image (labeling data) with labels assigned to each image is input.

[0004] Models that use images as input and perform classification and segmentation are mainly divided into Conv-type (CNN; Convolutional Neural Network) and ViT-type (Vision Transformer).

[0005] Patent Document 1 discloses that feature extraction of an input image is performed using a vision transformer and a CNN in parallel. It discloses that in model training, multitask training that connects the vision transformer and the CNN in parallel can create features that combine global features and local features.

[0006] Patent Document 2 discloses a model called a multi-axis vision transformer (MaxViT) that includes one or more multi-axis blocks that include a local attention axis and a global attention axis in series. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Publication No. 2025-077824 [Patent Document 2] Special Publication No. 2025-513748 Summary of the Invention [Problem to be solved by the invention]

[0008] Conventional deep learning models are static shape models, and the input image size of the network is fixed in advance. The ViT system is a technology in which the encoder divides the input image into multiple small images (patch images) for processing. Since it is possible to specify the number of divisions, the output itself can be changed according to the input size. However, the more the input image deviates from the size used during training or the aspect ratio (the ratio of the number of vertical pixels to the number of horizontal pixels) of the image, the more the inference behavior (performance) can deteriorate.

[0009] When inputting images of a different size from the size used during training in a deep learning model, image processing (resizing and padding) or the creation of a separate model is required. When performing image processing when inputting images of different sizes, there is a concern that information may be lost due to the image processing. Furthermore, when training a model for each size, there are problems such as the cost of training the model (calculation for training and creation of training data) and the cost of inference (preparing an inference environment with multiple models in parallel).

[0010] Therefore, an object of the present invention is to generate a deep learning model that is robust to differences in image size. [Means for solving the problem]

[0011] The gist of the present invention is as follows. (1) A deep learning model generation device that generates a deep learning model, comprising: The deep learning model includes at least an encoder that performs an operation based on a token in which position information of the input image is embedded in each of a plurality of patches generated from the input image; The deep learning model generation device a cropped image generating unit that generates at least one type of cropped image from an original image; an encoder learning unit that uses the cropped image as the input image and executes learning of the encoder, The cropped image generation unit The image processing system is configured to change at least one of the size of each cropped image and the number of cropped images for each learning set in the learning based on a predetermined probability within a predetermined condition. Deep learning model generator. (2) The encoder learning unit includes a memory that temporarily stores at least the original image and the cropped image in one learning set; The predetermined condition is: The deep learning model generation device described in (1) above, including a condition for determining the total size of the original image and the cropped image so that it is within the upper limit capacity that can be stored in the memory. (3) The deep learning model generation device according to (1) or (2) above, wherein the predetermined probability regarding the size of the cropped image includes a probability that is set in a skewed manner so that the closer the aspect ratio of the size is to 1, the higher the probability. (4) The deep learning model generation device according to (1) or (2) above, wherein the predetermined probability regarding the size of the cropped image includes a probability set in a gradient that increases as the size becomes smaller. (5) The encoder learning unit The deep learning model generation device according to (1) or (2) above, configured to embed position information in the input image by adding noise to each of a plurality of patches generated using the cropped image as the input image. (6) The cropped image generating unit The deep learning model generation device described in (5) above, which is configured to change the noise parameters of the position information for each of the cropped images for each of the learning sets in the learning based on a predetermined probability within predetermined conditions. (7) The cropped image generating unit configured to enlarge the original image so as to be able to generate the cropped image if the original image is smaller than the size of the cropped image; Extending the original image includes: (a) enlarging and transforming the original image so that the original image is equal to or larger than the size of the cropped image; (b) copying a portion of the original image directly or by flipping it vertically or horizontally to an undefined area of ​​the original image that is insufficient for generating the cropped image; and (c) filling in the undefined areas of the original image that are missing for generating the cropped image with zeros; The deep learning model generation device according to (1) or (2) above, including at least one of the methods. (8) The encoder learning unit The deep learning model generation device according to (1) or (2) above, configured to perform the learning of the encoder using self-supervised learning. (9) The self-supervised learning The deep learning model generation device described in (8) above includes executing one or both of a consistency task of matching feature amounts output from each of the different types of cropped images, and a completion task of masking a part of the cropped image and restoring the masked part. (10) The deep learning model a processing unit that performs semantic association processing on the meaning of the feature in the input image based on the result of the operation by the encoder, The deep learning model generation device The deep learning model generation device according to (1) or (2) above, comprising a processing unit learning unit that executes learning of the processing unit so as to estimate the meaning for the feature of the input image. (11) The deep learning model generation device according to (1) or (2) above, wherein the original image includes an image obtained of a steel product during or after manufacturing. (12) A deep learning model generation method for generating a deep learning model, comprising: The deep learning model includes at least an encoder that performs an operation based on a token in which position information of the input image is embedded in each of a plurality of patches generated from the input image; The deep learning model generation method includes: generating at least one cropped image from the original image; and performing training of the encoder using the cropped image as the input image; generating the cropped image changing at least one of the size of each of the cropped images and the number of the cropped images for each learning set in the learning based on a predetermined probability within a predetermined condition; Deep learning model generation method. [Effects of the Invention]

[0012] According to the present invention, it is possible to generate a deep learning model that is robust to differences in image size. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is a diagram schematically illustrating an example of the hardware configuration of a model generation device 1. FIG. [Figure 2] FIG. 2 is a diagram illustrating an example of the functions of the model generating device 1. [Figure 3A] FIG. 10 is a diagram illustrating an example of the adoption probability of the size of a cropped image. [Figure 3B] FIG. 10 is a diagram illustrating another example of the adoption probability of the size of a cropped image. [Figure 4] FIG. 10 is a diagram illustrating an example of a method for extending an original image. [Figure 5]FIG. 10 is a diagram illustrating an example of the functions of a model generating device when performing learning including a consistency task. [Figure 6] FIG. 10 is a diagram showing an example of the adoption probability of global crop sizes in training including a consistency task. [Figure 7] FIG. 10 is a diagram illustrating an example of the adoption probability of local crop sizes in training including a consistency task. DETAILED DESCRIPTION OF THE INVENTION

[0014] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In this specification and the drawings, elements having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0015] [1. Example of the configuration of a deep learning model generation device] An outline of a hardware configuration example of a model generation device 1 as a deep learning model generation device according to the present disclosure will be described below. Fig. 1 is a block diagram schematically illustrating an outline of a configuration example of a model generation device 1 according to this embodiment.

[0016] 1, the model generation device 1 mainly includes a processor 11, a RAM 12, and a ROM 13. The model generation device 1 further includes a bus 21, an input device 22, an output device 23, a storage device 24, a drive 25, a connection port 26, and a communication device 27.

[0017] The processor 11 functions as a processing device and a control device, and controls all or part of the operations within the model generation device 1 in accordance with various programs recorded in the RAM 12, the ROM 13, the storage device 24, or the removable recording medium 31. The RAM 12 temporarily stores the programs used by the processor 11 and parameters that change as appropriate during program execution. The ROM 13 stores the programs and calculation parameters used by the processor 11. These are interconnected by a bus 21 that is constituted by the internal bus of the processor 11.

[0018] The processor 11 according to this embodiment includes a GPU, and the RAM 12 includes a VRAM. The GPU as the processor 11 performs processing in a deep learning model generation method described below, including generating a cropped image from an original image, generating multiple patches using the cropped image as an input image, embedding position information of the input image in each patch, and various calculations for encoder training. The GPU operates multiple calculation cores in parallel to quickly execute matrix calculations and the like required for encoder training. From a certain perspective, any desired calculation device other than a GPU that can quickly execute matrix calculations and the like required for encoder training can be suitably used as the processor 11.

[0019] The VRAM as memory is configured to have an upper limit capacity capable of holding the total size of the original images and cropped images used in one training set in the deep learning model generation method described below. From another perspective, the deep learning model generation method includes a condition for determining the total size of the original images and cropped images used in one training set so that it is within the upper limit capacity of the VRAM.

[0020] In one embodiment, the processor 11 includes a CPU, and the RAM 12 includes RAM accessible by the CPU. The CPU assists various processes in the GPU in a deep learning model generation method described below.

[0021] The bus 21 is connected to an external bus such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge.

[0022] The input device 22 is an operation means operated by a user, such as a mouse, keyboard, touch panel, button, switch, or lever. The input device 22 may also be, for example, a remote control means (so-called remote control) that uses infrared or other radio waves, or an externally connected device 33 such as a PDA that supports operation of the model generation device 1. The input device 22 is further composed of, for example, an input control circuit that generates an input signal based on information input by the user using the operation means and outputs the signal to the processor 11. The user of the model generation device 1 can input various data to the model generation device 1 and instruct processing operations by operating the input device 22.

[0023] The output device 23 is configured with a device capable of visually or audibly notifying the user of acquired information. Examples of such devices include display devices such as CRT display devices, liquid crystal display devices, plasma display devices, EL display devices, and lamps, audio output devices such as speakers and headphones, printers, mobile phones, and facsimiles. The output device 23 outputs, for example, results obtained from various processes performed by the model generation device 1. Specifically, the display device displays the results obtained from various processes performed by the model generation device 1 as text or images. On the other hand, the audio output device converts audio signals consisting of reproduced voice data, acoustic data, etc. into analog signals and outputs them.

[0024] The storage device 24 is a data storage device configured as an example of a storage unit of the model generation device 1. The storage device 24 is configured, for example, by a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 24 stores programs and various data executed by the processor 11, as well as various data acquired from the outside.

[0025] The drive 25 is a reader / writer for a recording medium, and is built into or externally attached to the model generating device 1. The drive 25 reads information recorded on a removable recording medium 31, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 12. The drive 25 can also write information to a removable recording medium 31, such as an attached magnetic disk, optical disk, magneto-optical disk, or semiconductor memory. The removable recording medium 31 may be, for example, a CD medium, a DVD medium, or a Blu-ray (registered trademark) medium. The removable recording medium 31 may also be, for example, a CompactFlash (registered trademark) card, a flash memory, or an SD memory card (Secure Digital memory card). The removable recording medium 31 may also be, for example, an IC card (Integrated Circuit card) equipped with a contactless IC chip, or an electronic device.

[0026] The connection port 26 is a port for directly connecting a device to the model generation device 1. Examples of the connection port 26 include a USB (Universal Serial Bus) port, an IEEE1394 port, a SCSI (Small Computer System Interface) port, an RS-232C port, and an HDMI (registered trademark) (High-Definition Multimedia Interface) port. By connecting an external device 33 to this connection port 26, the model generation device 1 can directly obtain various types of data from the external device 33 and provide various types of data to the external device 33.

[0027] The communication device 27 is, for example, a communication interface configured with a communication device or the like for connecting to a communication network 35. The communication device 27 is, for example, a communication card for a wired or wireless local area network (LAN), Bluetooth (registered trademark), or wireless USB (WUSB). The communication device 27 may also be a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various types of communication. The communication device 27 can transmit and receive signals, for example, between the Internet and other communication devices in accordance with a predetermined protocol such as TCP / IP. The communication network 35 connected to the communication device 27 is configured with a network connected by wire or wirelessly, and may be, for example, the Internet, a home LAN, an in-house LAN, infrared communication, radio wave communication, satellite communication, or the like.

[0028] The above describes an example of a hardware configuration capable of realizing the functions of the model generation device 1 according to this embodiment. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. Therefore, the hardware configuration used can be changed as appropriate depending on the technical level at the time of implementing this embodiment.

[0029] The functions of the model generation device 1 according to this embodiment will be described below. The model generation device 1 according to this embodiment is configured to be able to execute each step of the deep learning model generation method according to the present disclosure. Figure 2 is a diagram schematically illustrating an example of the functions of the model generation device 1 according to this embodiment.

[0030] 2, the model generating device 1 has functions including a cropped image generating unit 101 and an encoder learning unit 102. The cropped image generating unit 101 and the encoder learning unit 102 are realized, for example, mainly by a processor 11, a RAM 12, and a ROM 13.

[0031] The cropped image generating unit 101 generates at least one type of cropped image from an original image. The original image is extracted by the processor 11 from a database stored in the storage device 24, for example. The size, aspect ratio, number, etc. of the original image are not particularly limited. In one embodiment described below, the original image is a surface image of a steel product during or after manufacturing.

[0032] In the example shown in FIG. 2, three original images IM1, IM2, and IM3 are loaded into the cropped image generation unit 101 (FIG. 2(a)). Then, the cropped image generation unit 101 generates cropped images Cp1, Cp2, and Cp3 from the three original images IM1, IM2, and IM3, respectively (FIG. 2(b)). The cropped images Cp1, Cp2, and Cp3 are different sizes and are different types of cropped images. In this disclosure, the "size" of a cropped image is understood to include the number of pixels (area) and aspect ratio of the cropped image, and the number of pixels and aspect ratio will be separately described below where necessary.

[0033] A probability of adoption for each size of the cropped image is preset. In one embodiment, the probability of adoption of a size is set with a gradient that increases as the aspect ratio approaches 1. In one embodiment, the probability of adoption of a size is set with a gradient that increases as the number of pixels decreases. In one embodiment, the probability of adoption of a size is set with a gradient that increases as the aspect ratio approaches 1 and the number of pixels decreases.

[0034] 3A and 3B are diagrams showing an example of a table containing information on the adoption probability of a preset size for each size. FIGS. 3A and 3B show the number of vertical pixels ("H" in FIGS. 3A and 3B) and the number of horizontal pixels ("W" in FIGS. 3A and 3B) for the size of the cropped image to be generated, as well as the adoption ratio for that size. The adoption ratio is an example of information constituting the adoption probability. Regarding the adoption probability of the present disclosure, the number of cropped images of a given size to be generated may be determined so as to achieve a certain adoption ratio, or whether or not to generate them may be determined randomly based on a probability set based on the adoption ratio. In other words, in the example of FIGS. 3A and 3B, if the adoption ratio of two cropped images of a given size is 1:2, one of two options can be used: (i) generating one of the two sizes at a ratio of one to two, or (ii) randomly generating a desired number of cropped images of one of the two sizes with a 1 / 3 adoption probability for one and a 2 / 3 adoption probability for the other.

[0035] For example, in the example shown in FIG. 3A, the adoption ratio for a cropped image having a size of H×W=128×128 (pixels) is 64. Also, for example, the adoption ratio for a cropped image having a size of H×W=128×576 (pixels) is 1. Thus, in the examples shown in FIGS. 3A and 3B, the size adoption ratio is set in a skewed manner so that it increases as the aspect ratio approaches 1 and the number of pixels decreases. In the embodiment shown in FIGS. 3A and 3B, the vertical and horizontal sizes are intentionally limited to 128 or greater because learning results did not improve when the vertical and horizontal sizes were smaller than 128. The value of this lower limit (128 in FIGS. 3A and 3B) is considered to depend on the learning target. Therefore, during learning, it is preferable to adjust the lower limit of the selectable vertical and horizontal pixel counts so as to maximize the effect of improving learning results depending on the learning target. Note that the blank spaces (darkest shaded areas) in FIG. 3 indicate that cropped images of that size are not adopted because they cannot be stored in RAM 12 (VRAM). The tables including the information on the adoption probabilities exemplified in FIGS. 3A and 3B may be set manually in advance according to the capacity of the RAM 12 (VRAM), or may be set automatically by a program.

[0036] 3B, the size adoption ratio is set higher as the aspect ratio approaches 1. As a result, the data capacity that the RAM 12 (VRAM) can hold depends more on the total number of pixels (area) than on the vertical and horizontal sizes of the image, and therefore can be set closer to the upper limit capacity of the RAM 12 (VRAM).

[0037] FIG. 4 is a diagram schematically illustrating an example of the relationship between the size of a cropped image Cp and the size of an original image IM, and is a diagram illustrating an example of a method for expanding the original image IM. As shown in FIG. 4, depending on the size of the determined cropped image Cp, the number of vertical or horizontal pixels of the cropped image Cp may be larger than the number of vertical or horizontal pixels of the original image IM. In this case, when the cropped image Cp is generated from the original image IM, an undefined area NA, in which no information is contained in the vertical or horizontal pixels, is generated. Because a cropped image Cp having such an undefined area NA cannot be used for learning, the original image IM is expanded by supplementing the number of vertical or horizontal pixels using one of the following methods.

[0038] The expansion of the original image IM can be performed by at least one of the following methods: (a) enlarging and deforming the original image IM so that it is equal to or larger than the size of the cropped image Cp; (b) copying a portion of the original image IM directly or by flipping it upside down or left to right into the undefined area NA of the original image IM that is missing for generating the cropped image Cp; and (c) adding zero to the undefined area NA of the original image IM that is missing for generating the cropped image Cp.

[0039] In (a), the original image IM is enlarged and deformed so that it is equal to or larger than the size of the cropped image Cp. As shown in (a) of FIG. 4, the cropped image Cp can be generated from the enlarged and deformed original image IM.

[0040] In (b), a portion of the original image IM is copied directly or flipped upside down or flipped left to right into the undefined area NA of the original image IM that is missing to generate the cropped image Cp. Figure 4(b) shows an example of copying the original image IM by flipping it upside down and left to right. As shown in Figure 4(b), a portion of the original image IM is copied (IM') and pasted to fill in the vertical pixels of the original image IM that correspond to the undefined area NA, thereby generating a cropped image Cp from the continuous area of ​​the original image IM and the copy IM'.

[0041] In (c), the undefined area NA of the original image IM that is missing for generating the cropped image Cp is assigned a zero ("NA'" in FIG. 4(c)). "Assigning zero" means, for example, setting the lower limit (0) of the information used for learning to the undefined area NA of the cropped image Cp. For example, if the information used for learning includes color information, this means setting a pixel having (R, G, B) = (0, 0, 0) in the RGB data. Note that, as long as the undefined area NA can be defined so that it can be learned, "assigning zero" is not limited to the above example; it can also be achieved by setting a pixel having the desired information. However, from the perspective of reducing the impact of such added information on learning, it is preferable to set the lower limit (0) of certain information, as in the above example.

[0042] The generated cropped images Cp1, Cp2, and Cp3 are passed to the encoder training unit 102 and used for training the encoder.

[0043] The encoder training unit 102 according to this embodiment is configured to be able to perform training of an encoder in a deep learning model to be generated. The deep learning model according to this embodiment includes the architecture of a vision transformer. The deep learning model as a vision transformer includes an embedder that generates multiple patches from an input image and embeds position information, an encoder 110, and a classifier that outputs class classification. The encoder training unit 102 according to this embodiment performs training of the encoder 110 of the vision transformer.

[0044] In the example shown in Fig. 2, the encoder learning unit 102 divides each of the cropped images Cp1, Cp2, and Cp3 to generate a plurality of patch images P (Fig. 2(c)). For convenience, Fig. 2 focuses on the cropped image Cp1 as a representative image, and shows an example of patch generation and learning functions using the cropped image Cp1, but the same applies to the cropped images Cp2 and Cp3.

[0045] The multiple patch images P are generated at a predetermined size independent of the size of the cropped image. In one embodiment, the size of the patch image P is first determined, and then candidate sizes of the cropped images to be generated by the cropped image generation unit 101 are determined so that the sizes are integer multiples of the size of the patch image P. In the example shown in FIG. 3 , the size of the patch image P is predetermined as H×W=16×16 (pixels). Correspondingly, candidate sizes (128, 192, . . . , 1024) of the vertical and horizontal sizes of the cropped images are determined so that the number of vertical and horizontal pixels is an integer multiple of 16, respectively. As a result, an integer number of patch images P are generated from the cropped image.

[0046] The patch image P is linearly projected and vector-converted. Position information PE (Position Encoding) is embedded in each of the vectorized patch tokens Pt (Figure 2 (d)). The position information PE is also embedded in the class token Ct. Details of embedding the position information PE will be explained later. Note that a token is a term used in natural language processing, and is primarily used to divide text into units that can be processed by deep learning models. In the ViT system, patch images, which are patches of an image, are treated as tokens, and are processed in the same way as natural language processing.

[0047] The patch token Pt and class token Ct with embedded position information PE are input to the encoder 110 ((e) of FIG. 2). The encoder 110 includes, for example, a multi-head attention layer and a multi-layer perceptron (MLP). The encoder 110 outputs a predetermined feature, for example, a vector of class token Ct. The encoder learning unit 102 calculates the loss and gradient from the feature. The encoder learning unit 102 also updates each parameter of the encoder 110 based on the calculated gradient.

[0048] In Figure 2, the series of learning steps, including (a) extraction of the original image, (b) generation of a cropped image, (c) generation of a patch image, (d) embedding, (e) gradient calculation, and (f) parameter update, are referred to as one learning set.

[0049] In general, it is known that in the training of vision transformers, the larger the number of original images (hereinafter referred to as "batch size") used in one training set (one parameter update), the higher the training efficiency. However, because the original images are stored in RAM 12 (VRAM), the batch size is limited by the upper limit capacity of RAM 12. Since VRAM with a large upper limit capacity is usually expensive, it is desirable to be able to perform efficient training even when using VRAM with a smaller upper limit capacity.

[0050] The model generation device 1 of this embodiment is capable of generating cropped images of various sizes from an original image within the upper limit capacity of the RAM 12. Therefore, even if the batch size is small, the types of cropped images used as input images can be diversified, enabling efficient learning of the encoder 110.

[0051] In one embodiment, the size of the cropped image may be changed for each training set based on the above adoption probability.

[0052] The diversity of cropped images includes the diversity of cropped image sizes, and therefore the diversity of aspect ratios and the diversity of pixel counts. Therefore, in a deep learning model having the encoder 110 trained by the model generation device 1 according to this embodiment, robustness against the size of the input image can be ensured.

[0053] Note that, because the cropped images are stored in RAM 12 in the same way as the original images, the variety of cropped images is limited by the upper limit capacity of RAM 12. In one embodiment, the adoption probability of the size of the cropped image is set so as to increase the efficiency of learning within the upper limit capacity of RAM 12. In the above embodiment, the adoption probability of a size that is set so as to increase as the aspect ratio approaches 1 and / or the number of pixels decreases is preferable from the viewpoint of increasing the efficiency of learning.

[0054] In one embodiment, the encoder learning unit 102 cumulatively adds gradients calculated using multiple cropped images Cp1, Cp2, and Cp3 as inputs, and updates parameters using the accumulated gradients. That is, after calculating gradients for one input image, it further calculates gradients for another input image without updating parameters. By repeating this process multiple times (a number of gradient accumulations), gradients are cumulatively added, and the accumulated gradients are used to update parameters after the number of gradient accumulations. This allows for adjustment conditions for the correction width during parameter update that depend on the batch size. As an example, if the batch size is small, the number of gradient accumulations is increased, and if the batch size is large, the number of gradient accumulations is decreased.

[0055] Regarding gradient accumulation, for example, the number of gradient accumulations is set to 3, and the following processing is performed. 1. Perform steps (a) to (e) on a group of cropped images A extracted from the original image. 2. Perform (a) to (e) on group B of cropped images (different from group A) extracted from the original image. 3. Perform (a) to (e) on group C of cropped images (different from groups A and B) extracted from the original image. Execute (f) using the gradient accumulation obtained by, for example, averaging the gradients calculated in 4.1 to 3. A series of processes 1 to 4, in which the gradients accumulated in steps 1 to 3 are used to perform parameter update in step 4, constitutes one batch. In this disclosure, each of processes (a) to (e) in steps 1 to 3 is referred to as one micro-batch, and the number of original images used to calculate the gradient in a micro-batch is referred to as the micro-batch size. For example, when the micro-batch size is 32, in the above example, the gradient accumulation count is three, so the batch size is 3 x 32 = 96. With gradient accumulation using such micro-batches, even if the number and size of input images that can be used in one gradient calculation (micro-batch size) are limited by the upper limit capacity of RAM 12, the batch size can be increased by increasing the number of gradient accumulation counts.

[0056] Gradient accumulation can be done by accumulating all gradients calculated in a training set. Gradient accumulation allows training with a variety of batch sizes while maintaining a uniform batch size.

[0057] In one embodiment, the model generation device 1 is configured to be able to perform self-supervised learning to train the encoder 110. Fig. 5 is a diagram schematically illustrating an example of the functions of the model generation device 1 when performing training including a consistency task, as an example of self-supervised learning.

[0058] In learning that includes a consistency task, the crop image generation unit 101 generates at least two types of crop images from one original image. In the example shown in FIG. 5, the original image IM4 is loaded into the crop image generation unit 101 (FIG. 5(a)). The crop image generation unit 101 generates a global crop (global crop image CpG) and a local crop (local crop image) CpL from the original image IM4 (FIG. 5(b)). The global crop CpG and the local crop CpL have different sizes and are different types of crop images. Note that the global crop CpG is defined as having a larger number of pixels than the local crop CpL.

[0059] For each size of the global crop CpG and the local crop CpL, an adoption probability for that size is set in advance. In one embodiment, the adoption probability of the size is set with a gradient that increases as the aspect ratio of the size approaches 1. In one embodiment, the adoption probability of the size is set with a gradient that increases as the size becomes smaller. In one embodiment, the adoption probability of the size is set with a gradient that increases as the aspect ratio of the size approaches 1 and the size becomes smaller.

[0060] Fig. 6 is an example of a table containing information on the adoption probability of a preset size for each size of global crop CpG. Fig. 6 shows the number of vertical pixels ("H" in Fig. 6), the number of horizontal pixels ("W" in Fig. 6) for the size of the global crop CpG to be generated, and the adoption ratio for that size. Details of each data shown in Fig. 6 are the same as those in Fig. 3.

[0061] Fig. 7 is an example of a table containing information on the adoption probability of a preset size for each size of the local crop CpL. Fig. 7 shows the number of vertical pixels ("H" in Fig. 7) and the number of horizontal pixels ("W" in Fig. 6) for the size of the local crop CpL to be generated, as well as the adoption ratio for that size. Details of each data shown in Fig. 7 are the same as those in Fig. 3.

[0062] The adoption ratio figures in FIGS. 6 and 7 are set commonly in both FIGS. 6 and 7. The adoption ratio figures common to both FIGS. 6 and 7 may represent the ratio between the number of global crop CpGs and the number of local crop CpLs generated. That is, for example, the adoption ratio for a global crop CpG having a size of H×W=256×256 (pixels) is 2, while the adoption ratio for a local crop CpL having a size of H×W=128×128 (pixels) is 8. In an example of such a size, four times as many local crop CpLs of the same size may be generated for one global crop CpG of that size. Furthermore, because the size of the local crop CpLs follows the adoption ratio in FIG. 7, the number and size of local crop CpLs to be adopted for a global crop CpG of a certain size is determined probabilistically. Therefore, for example, for a global crop CpG having a size of H×W=256×256 (pixels), four local crop CpLs having a size of H×W=128×128 (pixels) and two local crop CpLs having a size of H×W=128×256 (pixels) may be generated. In other words, in training including a consistency task, the number of cropped images is determined based on a predetermined probability. The number of cropped images may be changed for each training set based on the above-mentioned adoption probability.

[0063] The generated global crop CpG and local crop CpL are passed to the encoder learning unit 102 and used for learning the encoder 110 ((c) to (e) of FIG. 5). As an example, the global crop CpG is divided into patch images, and the resulting image is tokenized with position information PE, which serves as training data, while the local crop CpL is divided into patch images, and the resulting image is tokenized with position information PE, which serves as student data. Learning is then performed so that the feature values ​​output from each image are similar. Learning, including the consistency task of the encoder 110, can be performed using a desired known algorithm, for example, DINO (including DINOv2, DINOv3, etc.).

[0064] It is preferable that the global crop CpG and the local crop CpL are generated so that some of the pixels in the original image overlap. This also applies when multiple local crops CpL are generated for one global crop CpG. This can improve the efficiency of learning, including consistency tasks.

[0065] In one embodiment, the model generating device 1 is configured to be able to perform training of the encoder 110 by training that includes a completion task.

[0066] In the learning including the completion task, a part of the patch token Pt is masked and a task of restoring the masked part is executed. As an example, the unmasked original data is used as training data and the masked data is used as student data, and learning is performed so that the feature quantities output from each are similar. The learning including the completion task of the encoder 110 can be performed using a desired known algorithm, such as MAE or iBOT (Image Bert pre-training with Online Tokenizer).

[0067] In the above embodiment, various learning methods including a consistency task or a completion task have been described as examples of supervised learning, but the technology of the present disclosure is not limited to this. Any known or unknown task or learning method that can be proposed in the future can be used as long as it can efficiently train the encoder 110 of the deep learning model.

[0068] Hereinafter, details of embedding the position information PE according to the above embodiment shown in FIG. 2(d) or FIG. 5(d) will be described.

[0069] The position information PE is information about the position in the original crop image (such as the crop images Cp1 to Cp3 in FIG. 2(b) or the global crop CpG or local crop CpL in FIG. 5(b)) of each patch image P. The position information PE includes, for example, coordinate information for each pixel. The position information PE can be an absolute position PE (for example, a sinusoidal PE) or a relative position PE.

[0070] In one embodiment, the position information PE is noised, which allows for greater diversity in the cropped images.

[0071] Noise can be added to the position information PE as follows: For convenience, the following description will be given taking as an example a case where the learning of the encoder 110 is performed by MAE.

[0072] The embedding of the position information PE into the patch token Pt (class token Ct) is performed by the following formula (1). X,Y is a vector of patch tokens Pt or class tokens Ct, E maedef (X, Y) is a function related to the position information PE.

number

[0073] Function E for location information PE maedef (X, Y) are defined by the following formulas (2) to (7), where (X, Y) are normalized in advance to the range of 0≦x, y≦1, D is the dimension of the feature vector, d is the component number, and P0 is the number of pixels on one side of the patch image.

number

[0074] The noise is a function E of the position information PE. maedefFor (X,Y), it is given by, for example, (a) translating the overall position by Δ, (b) translating the coordinates by ε for each patch token Pt, and (c) resizing the coordinates by λ times.

[0075] In one embodiment, the noise parameters Δ, ε, and λ are changed within a predetermined domain with a predetermined probability for each training set. For example, the domains of each parameter are Δ: -0.5≦Δ≦0.5, ε: -0.5≦ε≦0.5, and λ: 1.4. -1 < λ < 1.4. In one embodiment, the predetermined probabilities are set uniformly random within the domain for the parameters Δ, ε, and uniformly random under logarithms within the domain for the parameter λ.

[0076] In addition, the method of adding noise to the position information PE is not particularly limited, and can be performed using CAPE (Continuous Augmented Positional Encoding), FPE (Feature Positional Encoding or Fractional Positional Encoding), RoPE (Rotary Positional Encoding), or the like.

[0077] In this way, according to the present embodiment, noise in the position information PE is diversified, thereby ensuring robustness against the diversity of data related to the position coordinates of the input image in a deep learning model having the trained encoder 110.

[0078] In one embodiment, the deep learning model has a processing unit (not shown) that performs processing related to the meaning of a feature extracted from an input image (referred to as "semantic related processing" in this disclosure) based on the feature. In this case, the model generation device 1 has a processing unit training unit that trains the semantic related processing in the processing unit of the deep learning model. The "meaning" of a feature in this disclosure is not particularly limited and refers to information that should be associated with the feature so that the model can perform a desired function depending on the application of the deep learning model.

[0079] In one embodiment of the semantic association processing, the model generation device 1 includes a classifier training unit (not shown) that performs class classification training of a classifier to estimate labels for input images based on class classification. Class classification training can be performed, for example, by supervised learning using labels assigned to the input images. More specifically, in one embodiment of the model configuration, a class classification (layer) is added to the output result of an encoder (layer) previously generated by self-supervised learning. This classifier training unit leaves the encoder (layer) as is and generates only the class classification (layer) by supervised learning. This makes it possible to determine whether or not a desired feature is present in an image using a deep learning model. Note that the previously trained encoder has features that are robust to differences in the size of the input image, so the class classification can be expected to have features that are similarly robust to differences in the size of the input image.

[0080] In one embodiment of the semantic association processing, the model generating device 1 may be configured to perform segmentation learning, which assigns a label to each pixel or a region of multiple pixels in an input image.

[0081] In one embodiment of semantic association processing, the model generating device 1 may be configured to perform learning related to an object detection task in which multiple objects included in an input image are detected and rectangular regions (bounding boxes) and labels for each region are output. Examples of object detection techniques include YOLO (You Only Look Once) and Faster R-CNN (Regions with Convolutional Neural Network).

[0082] In one embodiment of semantic association processing, the model generating device 1 may be configured to perform learning related to an image captioning task that describes the content of an image in text in natural language. As an example, image features may be extracted and then input into a natural language processing model to generate explanatory text. Examples of such techniques include Show and Tell and Bootstrapping Language-Image Pre-training (BLIP).

[0083] In one embodiment related to semantic association processing, the model generating device 1 may be configured to perform learning related to an image feature extraction task in which effective feature vectors are extracted from images and utilized for classification, retrieval, clustering, etc. One example is ResNet.

[0084] From one perspective, a deep learning model generation method according to one embodiment appropriately sets the following three features, which are in a trade-off relationship with each other, depending on the learning target. In the deep learning model generation method according to one embodiment, by setting the features so that calculations can be performed within RAM 12 (VRAM), it is possible to generate a deep learning model that is efficient and robust to differences in image size. 1. Increase the batch size. This increases the amount of information, which generally improves learning efficiency. In this case, taking into account the upper limit of RAM12 (VRAM), you can increase the batch size by reducing the size of the cropped images. However, it may be better not to make the size of the cropped images too small. 2. Increasing the size of the cropped image. This increases the number of tokens with embedded position information PE, which is expected to improve PE parameters and performance relative to image size. 3. The size of the global crop CpG and the size of the local crop CpL are changed according to the original image to be learned.

[0085] In one embodiment, the deep learning model constitutes a determination unit in a surface texture inspection device. The surface texture inspection device is a device that inspects the surface texture of an object to be inspected, such as a steel product. When a surface image of the object to be inspected (hereinafter referred to as an input image) is input, the surface texture inspection device uses a deep learning model to estimate the presence or absence of a defect (normal or defective), and if a defect is present, the type and shape (area) of the defect, and outputs the estimation result. The input image is an image captured by a digital camera or the like, and is composed of, for example, 800 x 600 pixels. Therefore, in training the encoder 110 of the deep learning model, the surface image of the object to be inspected is used as the source image.

[0086] In one embodiment, the input images include not only images taken by an optical camera of a steel product during or after manufacturing, but also images obtained by a device that measures or estimates various physical quantities such as surface temperature distribution. In one embodiment, the input images include images obtained by an internal flaw detection device that uses ultrasound or magnetism to confirm the absence of internal flaws. In one embodiment, the input images include images obtained by a visual inspection device when shipping a steel product such as a coiled product.

[0087] In one embodiment, the defect may be, for example, dirt, foreign matter, a local temperature drop, or a poor coil winding shape.

[0088] While one embodiment of the present invention has been described above, the present invention is not limited to the illustrated embodiment. It is clear that a person skilled in the art can conceive of various modifications and alterations within the scope of the ideas described in the claims, and it is understood that these also naturally fall within the technical scope of the present invention.

[0089] Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that are apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects. [Industrial Applicability]

[0090] The present invention is useful in generating deep learning models. [Explanation of symbols]

[0091] 1. Model generation device 11 processors 12 RAM 13 ROM 21 Bus 22 Input Devices 23 Output Devices 24 Storage Devices 25 Drive 26 connection ports 27 Communication equipment 31 Removable Recording Media 33 Externally connected devices 35 Communication Network 101 Crop image generation unit 102 Encoder learning unit 110 Encoder Ct Class Token Cp, Cp1, Cp2, Cp3 cropped images CpG Global Crop CpL Local Crop IM, IM1, IM2, IM3, IM4 Original image NA undefined area P patch image PE location information Pt Patch Token

Claims

1. A deep learning model generation device that generates a deep learning model, The deep learning model includes at least an encoder that performs an operation based on a token in which position information of the input image is embedded in each of a plurality of patches generated from the input image; The deep learning model generation device a cropped image generating unit that generates at least one type of cropped image from an original image; an encoder learning unit that uses the cropped image as the input image and executes learning of the encoder, The cropped image generation unit The image processing system is configured to change at least one of the size of each cropped image and the number of cropped images for each learning set in the learning based on a predetermined probability within a predetermined condition. Deep learning model generator.

2. the encoder learning unit has a memory that temporarily stores at least the original image and the cropped image in one learning set; The predetermined condition is: The deep learning model generation device according to claim 1 , further comprising a condition for determining the total size of the original image and the cropped image so that the total size is within an upper limit capacity that can be stored in the memory.

3. The deep learning model generation device according to claim 1 or 2, wherein the predetermined probability regarding the size of the cropped image includes a probability that is set in a skewed manner so that the closer an aspect ratio of the size is to 1, the higher the probability.

4. The deep learning model generation device according to claim 1 or 2, wherein the predetermined probability regarding the size of the cropped image includes a probability set in a gradient such that the smaller the size, the higher the probability.

5. The encoder learning unit The deep learning model generation device according to claim 1 or 2, configured to embed position information in the input image by adding noise to each of a plurality of patches generated using the cropped image as the input image.

6. The cropped image generation unit The deep learning model generation device according to claim 5 , configured to change the noise parameters of the position information for each of the cropped images for each of the learning sets in the learning based on a predetermined probability within predetermined conditions.

7. The cropped image generation unit configured to enlarge the original image so as to be able to generate the cropped image if the original image is smaller than the size of the cropped image; Extending the original image includes: (a) enlarging and transforming the original image so that the original image is equal to or larger than the size of the cropped image; (b) copying a portion of the original image directly or by flipping it upside down or left to right into an undefined area of ​​the original image that is insufficient for generating the cropped image; and (c) filling the undefined areas of the original image that are missing for generating the cropped image with zeros; The deep learning model generation device according to claim 1 or 2, comprising at least one of the following methods.

8. The encoder learning unit The deep learning model generation device according to claim 1 or 2, configured to perform the training of the encoder using self-supervised learning.

9. The self-supervised learning 9. The deep learning model generation device according to claim 8, further comprising: executing one or both of a consistency task of matching feature amounts output from each of the different types of cropped images, and a completion task of masking a part of the cropped image and restoring the masked part.

10. The deep learning model a processing unit that performs semantic association processing related to the meaning of the feature in the input image based on the result of the calculation by the encoder, The deep learning model generation device The deep learning model generation device according to claim 1 or 2, further comprising a processing unit learning unit that executes learning of the processing unit so as to estimate the meaning for the feature amount of the input image.

11. The deep learning model generation device according to claim 1 or 2, wherein the original images include images obtained of steel products during or after manufacture.

12. A deep learning model generation method for generating a deep learning model, comprising: The deep learning model includes at least an encoder that performs an operation based on a token in which position information of the input image is embedded in each of a plurality of patches generated from the input image; The deep learning model generation method includes: generating at least one cropped image from the original image; and performing training of the encoder using the cropped image as the input image; generating the cropped image changing at least one of the size of each of the cropped images and the number of the cropped images for each learning set in the learning based on a predetermined probability within a predetermined condition; Deep learning model generation method.

Images