Information processing device, information processing method, and program
The pixel shift downsample (PSDS) process in CNNs addresses the inefficiencies of traditional downsampling methods by maintaining data integrity and reducing computational load, enhancing machine learning accuracy and efficiency.
Patent Information
- Application Number
- JP2023083430
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-19
- Publication Date
- 2024-11-29
AI Technical Summary
Convolutional neural networks (CNNs) face challenges in downsampling that result in unsuitable data for multi-resolution processing and require excessive computational resources, leading to lower machine learning performance due to inaccurate downsampling and high calculation demands.
The implementation of a pixel shift downsample (PSDS) process using a single filter and reverse pixel shuffler processing to rearrange pixel data, allowing for accurate downsampling without reducing the amount of extracted information and minimizing computational load.
This approach enables high-accuracy analysis and recognition in artificial intelligence using CNNs by maintaining data integrity and reducing processing load, facilitating efficient machine learning.
Smart Images

Figure 2024166974000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to an information processing device and an information processing method for processing data using a convolutional neural network (CNN). [Background technology]
[0002] In recent years, convolutional neural networks (CNNs) are widely used to analyze and recognize data using artificial intelligence (AI). Among these CNNs, a network called U-net is known for estimating image segmentation.
[0003] A typical configuration of U-net is as described in Non-Patent Document 1, where the encoding side performs convolution processing and pooling processing in sequence to downsample, and a part of the convolution results of each layer on the encoding side is supplied to the decoding side layer as a feature map. On the decoding side, the data is upsampled, and the data of the feature map supplied from the encoding side is cropped to perform data convolution.
[0004] Also, as described in Non-Patent Document 2, a model called pix2pix uses U-net to identify image regions. In this pix2pix, instead of the pooling process of U-net disclosed in Non-Patent Document 1, downsampling is performed using convolution. [Prior art documents] [Non-patent literature]
[0005] [Non-Patent Document 1] Olaf Ronneberger, Philipp Fischer, andThomas Brox, “U-Net: Convolutional Networks for BiomedicalImageSegmentation”, “MICCAI2015.LNCS”, Springer, vol. 9351, p. 234-241, 2015 [Non-Patent Document 2] Phillip Isola Jun-Yan Zhu Tinghui Zhou Alexei A. Efros, “Image-to-Image Translation with Conditional Adversarial Networks”, Berkeley AI Research (BAIR) Laboratory, University of California, Berkeley, 26 Nov 2018 Summary of the Invention [Problem to be solved by the invention]
[0006] Downsampling in U-net and other networks has the function of feature extraction used in recognition and other cases, as well as the aspect of multi-resolution processing. Therefore, when downsampling is performed using pooling, while the aspect of feature extraction is strong, there is a possibility that data that is not suitable for multi-resolution processing will be generated. In addition, downsampling using normal convolution processing has the issue of a large amount of calculations, and the downsampling is not accurate, which can reduce the performance of machine learning depending on the application.
[0007] The present invention has been made in consideration of the above problems, and has an objective of providing an information processing device, an information processing method, and a program that are capable of performing accurate downsampling with minimal calculations without reducing the amount of information extracted in artificial intelligence using CNN. [Means for solving the problem]
[0008] In order to solve the above problem, the invention of claim 1 is an information processing device that performs data processing on digital data using a convolutional neural network, comprising: a single filter processing means that performs single filter processing by applying a single filter to original data, which is the digital data and is a collection of a plurality of pixels, to perform an operation; a pooling processing means that performs pooling processing to extract data from the original data by performing a predetermined processing on the original data; and an inverse pixel shuffler means that performs inverse pixel shuffler processing as a process of rearranging the pixels of the processed original data output by the single filter processing means, the pooling processing means, or the convolution processing means.
[0009] The invention described in claim 2 is characterized in that, in addition to the configuration described in claim 1, the digital data is image data.
[0010] The invention described in claim 3 is an information processing method by an information processing device which performs data processing on digital data using a convolutional neural network, comprising: a single filter processing procedure in which a single filter is applied to original data, which is a collection of a plurality of pixels, and a calculation is performed; a pooling processing procedure in which a pooling process is performed to extract data from the original data by performing a predetermined process on the original data, or a convolution processing procedure in which a convolution process is performed; and an inverse pixel shuffler procedure in which an inverse pixel shuffler process is performed as a process of rearranging the pixels of the one piece of original data output in the single filter processing procedure, the pooling processing procedure, or the convolution processing procedure.
[0011] The invention described in claim 4 is a program for causing a computer to function as the information processing device described in claim 1 or 2. Effect of the Invention
[0012] According to the present invention, in artificial intelligence using CNN, it is possible to perform analysis and recognition with high accuracy while preventing the amount of data and processing load from becoming excessive. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a functional block diagram showing an example of an information processing apparatus according to an embodiment of the present invention. [Diagram 2] FIG. 2 is a functional block diagram of a pixel shift downsampling processing unit according to the embodiment. [Diagram 3] 2 is a diagram showing a schematic diagram of the stride principle and the pixel shift down sample (PSDS) principle in the information processing device according to the first embodiment; FIG. [Figure 4] 1A is a diagram showing a stride principle and a filter processing principle as an alternative process to pixel shift downsampling in the information processing device of the same as above, and FIG. 1B is a diagram showing a result of the filter processing. [Diagram 5] 3A is a diagram showing a part of the result of filter processing in the information processing device, and FIG. 3B is a diagram showing an inverse pixel shuffler process in the information processing device. [Figure 6] 4 is a flowchart showing a processing procedure of downsampling processing in the information processing device. [Figure 7] 3A to 3C are diagrams illustrating downsampling and upsampling processes in the information processing device according to the first embodiment. [Figure 8] 4 is a flowchart showing a processing procedure of an upsampling process in the information processing device. [Figure 9] 4 is a flowchart showing a procedure for setting filter coefficients in the information processing device. [Figure 10] 4 is a diagram showing a schematic diagram of setting of filter coefficients in the information processing device according to the first embodiment; FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014] 1 to 10 show an information processing device and an information processing method in the information processing device according to an embodiment of the present invention. Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
[0015] [Basic configuration] First, the configuration of the information processing device of this embodiment will be described.
[0016] 1, an information processing device 1 according to this embodiment includes artificial intelligence (AI, hereinafter simply referred to as "AI"), and performs analysis and recognition of various data by AI, and restoration of data used for the analysis and recognition. The information processing device 1 performs data processing on digital data using CNN.
[0017] For example, the information processing device 1 of this embodiment performs downsampling (processing to make the number of pixels after processing equal to or smaller than the number of pixels before processing) and upsampling (processing to make the number of pixels after processing equal to or larger than the number of pixels before processing) of image data which is a collection of pixels. However, the information processing device 1 may be used for any processing of any other data.
[0018] In the following, this embodiment will be described assuming that the information processing device 1 analyzes, recognizes, and restores image data as digital data. However, the data handled by the information processing device 1 is not limited to image data, and may be, for example, audio data as digital data or various other digital data other than audio.
[0019] As shown in FIG. 1, the information processing device 1 of this embodiment includes, as functional means, an input data acquisition unit 10, a downsampling unit 11, an upsampling unit 12, a cropping unit 13, a filter coefficient setting unit 14, a storage unit 15, and an output unit 16. The downsampling unit 11 performs a series of processes for the downsampling process described above. The downsampling unit 11 includes a convolution processing unit 11a as a "convolution processing means", a pooling processing unit 11b as a "pooling processing means", a single filter processing unit 11c as a "single filter processing means", a pixel shift downsampling processing unit 11d, and an inverse pixel shuffler processing unit 11e as an "inverse pixel shuffler means". The upsampling unit 12 performs a series of processes for the upsampling process described above. The upsampling unit 12 includes a convolution processing unit 12a and an upsampling processing unit 12b as "convolution processing means". Although these functional means are configured by hardware or by a combination of hardware and software (such as a computer program), at least one of these functional means may be configured by software only. Note that the pixel shift downsampling processing unit 11d will be referred to as the PSDS processing unit 11d hereinafter in this specification.
[0020] The input data acquisition unit 10 acquires input data, which is data that is the subject of various processes in the information processing device 1. Fig. 1 shows a state in which the input data acquisition unit 10 acquires an input image 101 as "original data." The input data acquisition unit 10 may acquire the data as input data by downloading the data via a network (not shown) or the like.
[0021] In image data processing, the convolution processing unit 11a performs convolution processing on the original data. Specifically, the convolution processing unit 11a applies a different filter to each channel of the input data and adds up the results (a bias value may also be included). In addition, the filter coefficient by which the input data is multiplied is changed for each output channel. Therefore, if the input data has three channels and the output has six channels, 18 types of filters and six bias values are required, and each filter process involves product-sum calculation processing.
[0022] When a stride is set, the pooling processing unit 11b reduces the number of data for the data, and when a stride is not set, the pooling processing is performed without reducing the number of data. The pooling processing unit 11b includes a pooling processing setting function for setting the type of pooling, and can execute the pooling processing by any method according to the setting. The pooling processing setting function sets, for example, max pooling (a process for extracting only the maximum data in a window (hereinafter simply referred to as "window"), which is a processing area, and deleting the other data), avg pooling (a process for calculating the average value of the data present in the window), etc. The pooling processing unit 11b can generate downsampling data by performing the pooling processing set by the pooling processing setting function.
[0023] The single filter processing unit 11c filters data using a single filter. The filtering method involves performing a product-sum operation on a predetermined filter coefficient and pixel data to obtain digital data as the filtering result. The difference from the convolution processing unit 11a is that the convolution processing uses multiple filter coefficients with different coefficients to generate data for, for example, 64 or 128 channels, whereas the single filter processing unit 11c performs filtering using a single filter coefficient. Therefore, if the function of the PSDS processing unit 11d described later is not used, only filtered data with the same number of channels as the input is obtained.
[0024] The PSDS processing unit 11d sets a window frame in the downsampling process, a stride value, etc., and will be described in detail later. The inverse pixel shuffler processing unit 11e will also be described later.
[0025] The cropping processor 13 performs cropping used in U-net, and cuts out the central part of the data that has been subjected to convolution, etc., to generate a feature map.
[0026] Convolution processing unit 12a of upsampling unit 12 performs convolution processing on input data using a filter, similar to convolution processing unit 11a of downsampling unit 11. Upsampling processing unit 12b of upsampling unit 12 performs processing to increase the number of pixels, such as deconvolution processing and pixel shuffler processing.
[0027] The filter coefficient setting unit 14 sets filter coefficients to be used in the convolution processing unit 11a of the downsampling unit 11, the convolution processing unit 12a of the upsampling unit 12, the single filter processing unit 11c, the upsampling processing unit 12b, etc. so that the output of the machine learning becomes the desired value.
[0028] The memory unit 15 stores design information for the machine learning network realized by the information processing device 1, and also temporarily stores information on feature maps, data from the processing steps of the downsampling unit 11 and the upsampling unit 12, etc.
[0029] The output unit 16 outputs the machine learning result, for example, location information of a specific object included in the input image when the machine learning relates to segmentation.
[0030] FIG. 2 is a diagram showing functional blocks of the PSDS processing unit 11d, which includes a window frame setting unit 111, a stride value setting unit 112, a start position number setting unit 113, a start position setting unit 114, and a processing content setting unit 115.
[0031] The window frame setting unit 111 sets the size of the window frame of each filter used in the downsampling process, and the stride value setting unit 112 sets the slide amount of the window frame in the filter process. The start position number setting unit 113 sets the number of positions at which each filtering process starts in the pixel shift downsampling process, and the start position setting unit 114 sets the position at which each filtering process starts. The processing content setting unit 115 sets which of convolution processing, pooling processing, and single filter processing is to be performed in the pixel shift downsampling process. The number of start positions may be correlated with the stride value, and is set to the square of the stride value in this embodiment. That is, when the stride value is 2, the number of start positions is 4, and when the stride value is 3, the number of start positions is 9. However, the correlation is not limited to this.
[0032] [Principle of pixel shift downsampling] 3 is a diagram showing the principle of pixel shift downsampling (PSDS) executed by the downsampling unit 11 in the information processing device of this embodiment. In this embodiment, various settings of the PSDS processing unit 11d are as follows: Window frame: 2×2, stride value: 2, number of starting positions: 4, processing content setting: single filter processing. Note that the setting of the starting position will be explained in the embodiment of FIG.
[0033] For example, consider the case where image data is subjected to various filtering processes, such as convolution processing, or a window is used in a pooling process to reduce the number of pixels, thereby obtaining down-sampled data.
[0034] By moving the filter or window to different positions on the image data multiple times and performing multiple processes, multiple processing results can be obtained.
[0035] It is also possible to prepare multiple types of processing for image data, multiple window frame size settings, and processing start positions. In this way, multiple convolution data having different information, multiple pooling data having different information, multiple filtered data by a single filtering process having different information, etc. can be generated for each processing start position. This makes it possible to generate multiple downsample data having different information.
[0036] This process of shifting the start position of processing on the original data to be processed and performing a specific process is called "pixel shift down sampling (PSDS)" (hereinafter simply referred to as "PSDS").
[0037] Figure 3 shows a schematic diagram of PSDS processing with a window frame of 2x2, a stride value of 2 (stride value setting procedure), and the number of starting positions set to 4. Here, it is assumed that a single filter process using a low-pass filter is set as the processing content. Below, the four starting positions are explained as [Process 1] to [Process 4].
[0038] [Process 1: Processing that uses the reference position as the starting position] For example, as shown in (a-1) of Fig. 3, first, a matrix operation is performed to multiply four pixels (hereinafter simply referred to as "four pixels") of two rows and two columns in image data 20 as "original data" by a two row and two column window frame 21 of filter processing (hereinafter simply referred to as "filter 21"), to obtain data as a processing result. The initial installation position of the center part of this filter 21 is called the "reference position" indicated by reference numeral 30 (hereinafter simply referred to as "reference position"), and the installation position of this filter 21 is indicated by reference numeral 31. As one of the setting information of the start position setting unit 114 in the PSDS processing unit 11d, position information of reference numeral 31, which is the start position (information that pixel numbers 1, 2, 6, and 7 fall within the window frame (filter 21)) is set (start position setting procedure).
[0039] Next, as shown in (a-2), the filter 21 is slid in the X direction of the image data 20 (the left-right direction in the figure; here, to the right in the figure) by a stride value of 2 (a position moved by two data amounts), and a matrix operation is performed on the next four pixels using the same filter 21 to obtain the filtered result data (processing content setting procedure, processing execution procedure).
[0040] [Process 2: Processing starting from the right position] Meanwhile, in parallel with the filter processing at the reference position 30 in (a-1) above, as shown in (b-1), a position shifted by one pixel in the X direction from the reference position 30 of the image data 20 is set as a new start position 32, and a matrix operation is performed on four pixels (2 rows x 2 columns) using the filter 21 with the same coefficients as in (a-1) to obtain convolution data. As one of the setting information of the start position setting unit 114 in the PSDS processing unit 11d, position information of the code 32, which is the start position (information that pixel numbers 2, 3, 7, and 8 fall within the window frame (filter 21)), is set as in the above process 1. This start position 32 in (b-1) is a position to the right of the reference position 30 (hereinafter, this start position 32 will be simply referred to as the "right position"; start position setting procedure).
[0041] Next, as shown in (b-2), the filter 21 is slid in the X direction from the right position of the image data 20 with a stride value of 2, and a matrix operation is performed by the filter 21 on the next four pixels to obtain the filtering result data (stride value setting procedure, processing content setting procedure, processing execution procedure).
[0042] [Process 3: Processing starting from the bottom position] Furthermore, in parallel with the filter processing at the reference position and the right position, as shown in (c-1), a position shifted by one pixel in the Y direction (up and down in the figure; here, downward in the figure) with respect to the reference position 30 of the image data 20 is set as a new start position 33, and a matrix operation is performed on four pixels (2 rows x 2 columns) using the filter 21 with the same coefficients as in (a-1) to obtain data of the filtering result. In this case, too, position information of the code 33, which is the start position (information that pixel numbers 6, 7, 11, and 12 fall within the window frame (filter 21)), is set as one of the setting information of the start position setting unit 114 in the PSDS processing unit 11d. This start position in (c-1) is a lower position with respect to the reference position 30 (hereinafter, this start position 33 will be simply referred to as the "lower position") (start position setting procedure).
[0043] Next, as shown in (c-2), the filter 21 is slid in the X direction (right direction) from the bottom position of the image data 20 with a stride value of 2, and a matrix operation is performed by the filter 21 on the next four pixels to obtain the filtered result data (stride value setting procedure, processing content setting procedure, processing execution procedure).
[0044] [Process 4: Processing starting from a diagonal position] Furthermore, in parallel with the filter processing at the reference position, right position, and bottom position, as shown in (d-1), a position shifted diagonally downward and to the right by one pixel with respect to the reference position 30 of the image data 20 (a position shifted by one pixel each in the X direction (right direction) and the Y direction (downward)) is set as a new start position 34. Then, a matrix operation is performed on four pixels (2 rows x 2 columns) using a filter 21 with the same coefficients as in (a-1) to obtain data of the filtering result. Position information of the code 34, which is the start position (information that pixel numbers 7, 8, 12, and 13 fall within the window frame (filter 21)), is set as one of the setting information of the start position setting unit 114 in the PSDS processing unit 11d. This start position 34 in (d-1) is a diagonal position with respect to the reference position 30 (hereinafter simply referred to as the "diagonal position") (start position setting procedure).
[0045] Next, as shown in (d-2), the filter is slid in the X direction (right direction) from the diagonal position of the image data 20 with a stride value of 2, and a matrix operation is performed on the next four pixels by the filter 21 to obtain the filtering result data (stride value setting procedure, processing content setting procedure, processing execution procedure).
[0046] When the start position is set to four (that is, four start positions are set) as described above, information on the four start positions is set in start position setting unit 114.
[0047] In the above [Process 1] to [Process 4], a single filter that operates as a reduction filter is used in the filtering process of image data 20 with starting positions 31, 32, 33, and 34 being the reference position, right position, lower position, and diagonal position. Even when a single filter is used in this way, multiple filtering results with shifted pixel centers can be obtained. This makes it possible to eliminate bias in the position of the filtering process.
[0048] After the above [Process 1] to [Process 4], the following [Process 5] is performed.
[0049] [Process 5: Repeat the process] The filtering processes with the reference position 30, right position 32, bottom position 33 and diagonal position 34 as starting positions in the above [Process 1] to [Process 4] are performed up to the right end of the image data 20 while sliding the filter 21 in the X direction (rightward) with a stride value of 2 in the same manner as above. Thereafter, the positions shifted by 2 pixels in the Y direction (downward) of the image data 20 from the initial starting positions 31, 32, 33 and 34 are set as new starting positions (not shown), and the filtering processes of the filter 21 on the image data 20 are performed with the reference position, right position, bottom position and diagonal position as starting positions in the same manner as above.
[0050] For example, the calculation in [Process 1] above with reference position 30 as start position 31 shifts the image data 20 in the Y direction (downward) by two pixels with a stride value of 2 to a new start position (not shown) that overlaps with pixels No. 11, 12, 16, and 17 shown in Fig. 3. Filtering is performed by filter 21 on these four pixels No. 11, 12, 16, and 17, and then the same filtering process by filter 21 as above is repeated.
[0051] Similarly, for filtering with starting positions 32, 33, and 34 being the right position in [Process 2] above, the bottom position in [Process 3] above, and the diagonal position in [Process 4] above, the filtering process is repeated on image data 20 by filter 21, with new starting positions (not shown) being positions shifted two pixels in the Y direction (downward) from the original starting positions 32, 33, and 34.
[0052] The above [Process 5] may be performed as a series of processes together with at least one of the above [Process 1] to [Process 4]. Also, [Process 5] may be repeatedly performed on the same image data 20. For example, by repeatedly performing [Process 5] on the image data 20 immediately after [Process 1], a filtering process with (a-1) as the starting position can be performed on the entire image data 20. Also, for example, by repeatedly performing [Process 5] on the image data 20 immediately after [Process 1] and immediately after each of [Process 2] to [Process 4], a filtering process using PSDS can be performed on the entire image data 20.
[0053] As shown in the above [Process 1] to [Process 5], by setting the pixel center based on a position shifted by one pixel to the right, down, and diagonally other than the reference position 30 and performing the convolution process, four times as much data can be obtained compared to the case where the filtering process or pooling process is performed based only on the fixed reference position 30, and learning and inference can be performed using four times as much data. Therefore, compared with the conventional filtering process or pooling process with a fixed reference position, it is possible to improve the learning efficiency and the quality of the inference result. Also, in the case of the conventional convolution process, it is necessary to perform the process using many filter coefficients, but in the present invention, many filtering process results can be obtained using a single filter coefficient, and the accuracy of the downsampling process can be improved.
[0054] It is generally said that the more types of filter coefficients used in convolution, the higher the accuracy, but in the present invention, even though a single filter coefficient is used, the accuracy can be improved. Note that the filter coefficients of the filter used in downsampling may be the same or different for each layer.
[0055] In this embodiment, a set of continuous data in two dimensions formed by filtering processing starting from a specific start position is called a “channel.” For example, in the case of the above [Process 1] to [Process 5], four sets of data are formed with start positions 31, 32, 33, and 34 being the reference position 30, the right position, the lower position, and the diagonal position, so that four channels of filtering data are generated.
[0056] In the example shown in Fig. 3, the start positions 31, 32, 33, and 34 are respectively set to the right, down, and diagonal positions shifted by one pixel from the reference position 30, and the stride value in the X direction (rightward) from each of the start positions 31, 32, 33, and 34 is set to 2 for filtering. However, the magnitude and direction of the stride are not limited to this, and any magnitude and direction of the stride may be used as long as learning and inference can be appropriately performed using the processed data.
[0057] In the above description, an example was given in which a 2×2 filter was used, the stride value was set to 2, and the image was reduced by 1 / 2 (the number of pixels was reduced). However, if a 3×3 filter was used, the stride value was set to 2, and the image was reduced by 1 / 2, the start positions of the reference position, right position, lower position, and diagonal position would be pixel positions No. 7, No. 8, No. 12, and No. 13. This is a filtering result at a position shifted by 0.5 pixels from each start position when a 2×2 filter was used, so a larger amount of data may be obtained by preparing a plurality of single filters in this way and generating filtering data for a plurality of channels. Also, instead of changing the size of the single filter, a larger amount of data may be obtained by generating filtering data for a plurality of channels using filters with different filter coefficients and biased filter coefficients. As for the type of single filter, a low-pass filter for extracting the reduced component of the image may be used as described in the above embodiment, or a high-pass filter for extracting the high-frequency component of the image may be used.
[0058] In the above embodiment, a 2x2 filter is used, and the start position is set to a position where a pixel is located in the image, but the start position may be set outside the image, and processing such as padding (see padding 51, 52 shown in FIG. 4(a)) may be performed for the portion of the filter frame where no pixel exists. Also, different stride values may be set in the X and Y directions.
[0059] In the example shown in Fig. 3, the stride value was set to 2 to reduce the reduction size by half, and as a result, four channel data was obtained, but the number of channels is not limited to four as long as learning and inference can be performed appropriately using the processed data. For example, it is possible to set the starting positions to three positions (the reference position, the right position, and the bottom position) and have three channels of the processing result, or to set the starting position to only a diagonal position and have one channel of the processing result.
[0060] 3, the filter 21 used is a square matrix of 2 rows and 2 columns, but is not limited to this. In other words, as long as it can appropriately perform learning and inference using processed data, the number of rows and columns constituting the filter 21 may be three or more, and it does not have to be a square matrix.
[0061] The principles shown in the above [Process 1] to [Process 5] can be applied not only to filtering processes using a single filter coefficient, but also to convolution processes and pooling processes. When PSDS is applied to convolution processes, that is, when "convolution" is set in the processing content setting unit 115 of the PSDS processing unit 11d, convolution itself has a large amount of processing calculation, and by using PSDS, more resources are required for processing calculations, but since the amount of information increases, more accurate machine learning can be expected. Also, when "pooling" is set in the processing content setting unit 115, pooling results of multiple channels (four channels in the above embodiment) with different starting positions can be obtained. In any of the convolution processes, pooling processes, and single filter processes, information such as the window frame size, stride value, number of starting positions, and starting position setting is not limited to only those exemplified in (a-1) to (d-2) of FIG. 3.
[0062] [Reverse pixel shuffler processing] The information processing device 1 of this embodiment can also perform processing by the inverse pixel shuffler processing unit 11e shown in FIG. 1, instead of the processing by the PSDS processing unit 11d shown in FIG.
[0063] The inverse pixel shuffler processor 11e of this embodiment divides the original channel into multiple new channels, and rearranges multiple pixel data existing in the original channel into multiple new channels. This process is called "inverse pixel shuffler process (space-to-depth)" and will be referred to as "inverse pixel shuffler process" in this specification.
[0064] [Configuration of the inverse pixel shuffler processing unit] The inverse pixel shuffler processing unit 11e of this embodiment has the same configuration as the PSDS processing unit 11d shown in Fig. 2. That is, the inverse pixel shuffler processing unit 11e has the same configuration as the window frame setting unit 111, the stride value setting unit 112, the start position number setting unit 113, the start position setting unit 114, and the processing content setting unit 115. Note that none of these configurations are shown in Fig. 1.
[0065] The component equivalent to the window frame setting unit 111 of the inverse pixel shuffler processing unit 11e functions as a size setting means for setting a filter frame (filter size) for performing a single filter process or the like on the image data 20 that is the original data.
[0066] In the inverse pixel shuffler processing unit 11e, the configuration corresponding to the stride value setting unit 112 has a function of setting the size by which the window frame performing the filtering process is moved (strided) on the original data, that is, the image data 20. However, when the inverse pixel shuffler process is used, it is preferable to perform the process by setting the stride value to 1.
[0067] In the inverse pixel shuffler processing unit 11e, the configuration corresponding to the start position setting unit 114 has a function of setting at least a partial area of the original data, that is, image data 20, as a start position for initially setting the filter frame.
[0068] In the inverse pixel shuffler processing unit 11e, a component corresponding to the start position number setting unit 113 sets a value corresponding to the start position number when rearranging image data 20, which is original data that has been subjected to a single filter process or the like, into new image data 57, 58, 59, 60. The value corresponding to the start position number here is, for example, the number of pixels present in the frame 53, and this number of pixels becomes the number of channels of the new image data 57, 58, 59, 60, and this number of channels corresponds to the start position number (details will be described later).
[0069] In the inverse pixel shuffler processing unit 11e, the configuration corresponding to the processing content setting unit 115 has a function of setting specific processing content similar to that of the processing content setting unit 115 in the inverse pixel shuffler processing.
[0070] Furthermore, the inverse pixel shuffler processing unit 11e of this embodiment can use the processing results of at least one of the convolution processing unit 11a, the pooling processing unit 11b, and the single filter processing unit 11c. In other words, in the PSDS processing, the position of the extracted data is shifted and then the single filter processing or the like is performed, whereas in the inverse pixel shuffler processing, the data to be extracted from the pixel data that has previously been subjected to the single filter processing or the like is rearranged to obtain a similar result. The convolution processing, pooling processing, and processing by the single filter processing unit 11c have already been described in the above explanations of the "convolution processing unit 11a", "pooling processing unit 11b", and "single filter processing unit 11c", so the description thereof will be omitted here.
[0071] Like the PSDS processing unit 11d, the inverse pixel shuffler processing unit 11e includes a configuration (not shown) similar to that of the processing content setting unit 115. Therefore, it is possible to appropriately select a desired data processing before performing the inverse pixel shuffler processing, and to perform the inverse pixel shuffler processing on the processed data.
[0072] [Inverse pixel shuffler processing steps (including pre-processing steps)] The inverse pixel shuffler process of this embodiment will be described below with reference to FIGS.
[0073] 4(a) and (b) are diagrams showing a filtering process as an example of a process performed before the inverse pixel shuffler process. Here, this process is performed by the single filter processor 11c filtering data using a single filter (single filter processing procedure).
[0074] Fig. 4(a) shows a schematic diagram of a filtering process for image data 20, with a window frame of 3x3 and a stride value of 1. In Fig. 4(a), padding 51 is added above the first row of the original image data 20, and padding 52 is added to the left of the first column. These paddings 51 and 52 are added in order to obtain the same number of pixel data as the original image data 20 after processing such as a single filter process. Therefore, unless there is a special need to match the number of data before and after the filtering process, there is no need to add one or both of the paddings 51 and 52.
[0075] As shown in (a) of FIG. 4, first, the single filter processing unit 11c sets the first row and first column of the image data 20 before the padding 51 and 52 are added as the reference position 30. The single filter processing unit 11c sets the filter frame 21a so that the center of the 3×3 filter 21 (i.e., the pixel in the second row and second column) is at the reference position 30. At this time, the filter frame 21a is set in a state including the padding 51 and 52. In this state, the single filter processing unit 11c performs a filtering process on the pixels of the image data in which the filter frame 21a is set. The result of the filtering process is set to pixel 54 (the pixel indicated by the circled number 1) in the first row and first column of the image data 20 shown in (b) of FIG. 4.
[0076] Next, as shown in (a) of Fig. 4, the single filter processing unit 11c shifts the filter frame 21a by one in the X direction (the left-right direction in Fig. 4; rightward in this case). The single filter processing unit 11c performs a filtering process on the pixel of the image data 20 on which the filter frame 21a is placed. The result of the filtering process is set to pixel 55 (the pixel indicated by the circled number 2) in the first row and second column of the image data 20 shown in (b) of Fig. 4.
[0077] Next, as shown in (a) of Fig. 4, the single filter processing unit 11c further shifts the filter frame 21a by one in the X direction (to the right). The single filter processing unit 11c performs a filtering process on the pixel of the image data 20 in which the filter frame 21a is placed. The result of the filtering process is set to pixel 56 (the pixel indicated by the circled number 3) in the first row and third column of the image data 20 shown in (b) of Fig. 4.
[0078] The single filter processing unit 11c repeatedly performs the same processing on all pixels of the image data 20 while moving the filter frame 21a to be filtered based on the stride value (the stride value=1 in this embodiment). After the filter frame 21a has moved to the right end of the image data 20 in the X direction, the single filter processing unit 11c moves the filter frame 21a in the Y direction (the up-down direction in FIG. 4; downward here). As a result, the filter frame 21a is moved to the left end (or the right end) of the row below in the image data 20. Then, the single filter processing unit 11c causes the single filter processing unit 11c to repeatedly perform the filtering processing while moving the filter frame 21a by one from left to right (or right to left). Then, the single filter processing unit 11c performs the filtering processing on all pixels of the image data 20.
[0079] FIG. 5(a) is a diagram showing a schematic view of a portion of image data 20 that has been subjected to the filtering process shown in FIG. 4. FIG. 5(a) shows, as an example, a case where the inverse pixel shuffler process is performed on image data 20 of 8×8=64 pixels. However, the image data 20 on which the inverse pixel shuffler process is performed may be of any size. That is, the size of the image data 20 may be larger or smaller than 8×8. Furthermore, the length of the image data 20 in the X direction (the left-right direction in the figure) and the Y direction (the up-down direction in the figure) do not have to be the same. For example, the length of the image data 20 in the X direction may be larger than the length in the Y direction.
[0080] The inverse pixel shuffler processing unit 11e performs an inverse pixel shuffler process on the filtered image data 20 (an inverse pixel shuffler procedure).
[0081] 5, as an example, a process of extracting pixels from the original image data 20 and reconstructing them into image data 57, 58, 59, and 60 is shown. In this example, the interval at which pixels are extracted from the original image data 20 is set to a predetermined value, for example, 2 (2 in the X direction and 2 in the Y direction). Therefore, for convenience, the lengths of the X direction and the Y direction of the range shown as the frame 53 are each set to 2. Also, the X direction length and the Y direction length of the new image data 57, 58, 59, and 60 are set to 1 / 2 of the X direction length and the Y direction length of the original image data 20 (4 in the X direction and 4 in the Y direction). Also, in this example, the inverse pixel shuffler process starts from the upper left of the original image data 20, and is sequentially performed in the X direction and then in the Y direction.
[0082] Here, when the original image data that has been subjected to filtering or other processing is extracted as four-channel data, the frame 53 is set to two rows and two columns.
[0083] 5A, a frame 53 is first set in the upper left 2-row, 2-column range of the original image data 20, i.e., the four pixels of the circled numbers 1, 2, 9, and 10. Then, these four pixels of the circled numbers 1, 2, 9, and 10 are set as the range of the frame 53. In other words, the size of the frame 53 (for example, 2×2, or 4 pixels) is also determined by the number of channels obtained by rearrangement.
[0084] The number of channels obtained by rearrangement (shuffling) can be set to any value other than 4. For example, by making frame 53 3 rows and 3 columns, 9 channels of data can be obtained. Also, the range of pixels to be shuffled, that is, the size of frame 53, does not need to be the same in the X and Y directions, and can be set arbitrarily according to the state and size (such as the aspect ratio) of the original image data 20. For example, if the size of the original image data 20 is 1 vertical and 3 horizontal, the size of frame 53 can be set to 3 rows and 1 column.
[0085] The size of the range of pixels to be shuffled is desirably set to a size that can improve the learning efficiency and the quality of the inference results in machine learning using image data 57, 58, 59, 60. However, the number of channels to be rearranged can also be set to a size that does not directly affect the efficiency or effectiveness of machine learning.
[0086] 5(a) and 5(b), the numbers 1 to 32 indicate the correspondence between pixels of the image data 20 before processing and the image data 57, 58, 59, and 60 after processing. Note that the numbers 33 to 64 after the number 32 in the figure have been omitted to simplify the figure.
[0087] As shown in the corresponding relationships in (a) and (b) of FIG. 5, the inverse pixel shuffler processing unit 11e rearranges pixels existing within a frame 53 set in the original image data 20 to the same positions in the new image data 57, 58, 59, and 60.
[0088] For example, in FIG. 5A, the number of channels to be rearranged is four, so when frame 53 is used as the reference, the pixel data for each pixel (second pixel) skipped one pixel in each of the X and Y directions is compiled as new image data 57, 58, 59, and 60 after rearrangement.
[0089] The inverse pixel shuffler processor 11e sets the inverse pixel shuffler process as shown in the following (Setting A) and (Setting B), thereby realizing uniform and low-load processing.
[0090] (Setting A) The interval at which pixels are extracted is the reduction ratio of the lengths in the X and Y directions of image data 20. For example, in Fig. 5, the interval at which pixels are extracted is 2, so while the length of image data 20 in the X and Y directions is 8, the lengths of image data 57, 58, 59, and 60 in the X and Y directions are 8 x 1 / 2 = 4.
[0091] (Setting B) The number of channels is the square of the interval at which pixels are extracted. For example, in FIG. 5, the interval at which pixels are extracted is 2. Therefore, the number of channels of the image data 57, 58, 59, 60 is 4, which is the square of 2. In this case, the number of channels is equivalent to the number of start positions 31, 32, 33, 34 in the above (Process 1) to (Process 4) (see (a-1) (b-1) (c-1) (d-1) in FIG. 3), and is the number of data generated by processing the original image data 20. In other words, if the interval at which pixels are extracted is determined, the number of channels of the processed image data 57, 58, 59, 60 is determined. In addition, learning and inference by machine learning can be performed based on multiple image data 57, 58, 59, 60, which is more than one image data 20. Therefore, the learning efficiency in machine learning and the quality of inference results can be improved.
[0092] The total data amount, i.e., the total number of pixels, of the image data 57, 58, 59, 60 obtained by such inverse pixel shuffler processing is the same as the total number of pixels of the original image data 20. However, by using the inverse pixel shuffler processing, it is possible to obtain image data 57, 58, 59, 60 of four channels for each of the four pixels (pixels with the numbers 1, 2, 9, 10) present in the top left two rows and two columns of the image data 20 in the reference position of the data (in (a) of FIG. 5).
[0093] In the above example, in FIG. 4, a case has been described in which the image data 20 is subjected to filtering processing by a single filter processor 11c, but in this case, any filter may be used for this filtering processing.
[0094] For example, the single filter processing unit 11c can be set for filtering processing with either the same single filter 21 as the PSDS processing unit 11d or a different single filter 21. The single filter processing unit 11c may apply multiple types of filters 21 to filtering of one image data 20 to generate filtered data of multiple channels, thereby obtaining more data. Instead of changing the size of the single filter 21, it may be possible to generate filtered data of multiple channels using filters 21 with different filter coefficients and biased filter coefficients. This makes it possible to obtain more data.
[0095] The filter 21 used here may be any type of filter, such as a low-pass filter, a high-pass filter, an edge emphasis filter, or an edge bulge filter. Furthermore, instead of the filtering process by the filter 21 shown in FIG. 4, a convolution process and a pooling process may be performed before the inverse pixel shuffler process.
[0096] [Effects of inverse pixel shuffler processing] The processing result by the inverse pixel shuffler processing unit 11e is equivalent to the state where the data has been pixel-shifted and processed by the PSDS processing unit 11d. Therefore, by the inverse pixel shuffler processing, it is possible to obtain image data 57, 58, 59, 60, the number of which depends on the number of pixels included in the frame 53, with the same total number of pixels as the total number of pixels of the image data 20 before processing.
[0097] In addition, since the number of pixels and the pixel state of the pre-processing image data 20 and the post-processing image data 57, 58, 59, 60 can be kept the same, it becomes easy to maintain the features of the pre-processing image data 20 in the post-processing image data 57, 58, 59, 60. Therefore, machine learning can be performed using the image data 57, 58, 59, 60, which has a small amount of post-processing data and contains many features of the original image data 20, and the accuracy of the machine learning can be improved.
[0098] Furthermore, since the processed image data 57, 58, 59, 60 can maintain the same number of pixels and pixel state as the unprocessed image data 20, the unprocessed image data 20 can be easily restored from the processed image data 57, 58, 59, 60.
[0099] [Another embodiment of the inverse pixel shuffler process] Furthermore, a single filtering process, a convolution process, and an inverse pixel shuffler process may be used in combination, or a single filtering process, a pooling process (pooling process procedure), and an inverse pixel shuffler process may be used in combination, or a convolution process, a pooling process, and an inverse pixel shuffler process may be used in combination.
[0100] For example, instead of the above-mentioned filtering process, the image data 20 may be subjected to pooling processing by the pooling processing unit 11b, and then the inverse pixel shuffler processing by the inverse pixel shuffler processing unit 11e may be performed on the image data 20. Also, instead of the above-mentioned filtering process, the image data 20 may be subjected to convolution processing by the convolution processing unit 11a, and then the inverse pixel shuffler processing by the inverse pixel shuffler processing unit 11e may be performed on the image data 20.
[0101] Also, for example, the image data 20 that has been processed by any two or more of the convolution processing unit 11a, the pooling processing unit 11b, and the single filter processing unit 11c may be configured to undergo inverse pixel shuffler processing by the inverse pixel shuffler processing unit 11e.
[0102] For example, instead of using a single filter by the single filter processing unit 11c, the inverse pixel shuffler processing by the inverse pixel shuffler processing unit 11e may be performed on image data 20 that has been subjected to filtering processing by a plurality of filters. Also, for example, the inverse pixel shuffler processing by the inverse pixel shuffler processing unit 11e may be performed on image data 20 that has been subjected to various processes other than filtering, convolution, and pooling.
[0103] [Processing Procedure] Fig. 6 is a flowchart showing the procedure of downsampling processing in the information processing device 1 of this embodiment. Fig. 7 is a diagram showing a schematic diagram of downsampling processing and upsampling processing in the information processing device of this embodiment. Note that it is also possible to apply the above-mentioned inverse pixel shuffler processing to the downsampling processing shown in this processing procedure. Furthermore, it is also possible to apply the inverse processing of the above-mentioned inverse pixel shuffler processing to the upsampling processing shown in this processing procedure.
[0104] FIG. 7 shows a state where the information processing device of this embodiment is applied to "U-net" as an example of a machine learning model using CNN to perform processing. In this model, data is processed in a plurality of hierarchical processing layers. Downsampling is performed on the encoder side of each layer, and upsampling is performed on the decoder side. Also, on the encoder side, convolution processing is performed in each layer 41 to 45, and in addition to convolution processing, filtering processing by the PSDS described above is performed in the second layer 42 to the fifth layer 45. The deeper the layer (the lower the layer in FIG. 6), the smaller the spatial spread of the data becomes, but the number of channels of the data increases. In FIG. 7, the first layer 41 to the fifth layer 45 are formed. However, the number of layers may be more or less than this.
[0105] The processing procedure of this embodiment will be described below based on the figure. In the example shown in Figure 7, PSDS is used instead of the pooling and convolution processing in the conventional U-net. By using a single filter process using PSDS instead of the convolution processing in the conventional U-net, processing can be performed with faster calculations, and by using a single filter process using PSDS instead of the pooling processing, data with a larger number of channels can be obtained than the normal pooling processing result.
[0106] [1. Downsampling process] [Step S1: Acquire Input Image] First, the input data acquisition unit 10 acquires an input image 101 as digital data "image data." The size (number of pixels) of this input image 101 may be any size, but as an example, the input image 101 is assumed to be 251 x 251 pixels. For ease of explanation, the input image 101 will be explained as grayscale data (data having one parameter (shade of black and white) value per pixel). If the input image 101 is color data (for example, data having three parameter (RGB) values per pixel), the input will be three channels.
[0107] [Step S2: Convolution Processing by Downsampling Unit 11] [Step 2-1] The convolution processing unit 11a performs convolution processing on the input image 101. In FIG. 7, the convolution processing is performed using filter coefficients in a number corresponding to the number of output channels. In this embodiment, in order to obtain an output of 64 channels, 64 types of filter coefficients 221, 222, ... 22 are used. 64 are used. In addition, filter coefficients 221, 222, . . . 22 64 For convenience, the position of the window where the convolution process (including the pooling process) is performed using is referred to as the filter position.
[0108] First, the convolution processing unit 11a performs convolution processing on the input image 101 in the form of a rectangular matrix of 251 × 251 pixels. The convolution processing uses, for example, filter coefficients of 3 rows × 3 columns, performs a product-and-sum operation between a 3 row × 3 column square matrix of values for each pixel and the filter coefficients, and calculates the operation result.
[0109] This convolution process is performed up to the upper right corner of the input image 101. However, since a 3 row x 3 column filter is used, the filter output in the X direction is 249.
[0110] Next, the convolution processing unit 11a shifts the filter position by one pixel in the Y direction and performs the convolution processing, and then obtains the convolution processing results sequentially while sliding the filter position by one pixel from left to right.
[0111] Convolution processing is sequentially performed on pixel data of the input image 101, and thereafter, similar convolution processing is repeated up to the bottom right corner of the input image 101.
[0112] However, just like the X direction described above, the filter output in the Y direction is also 249. Therefore, when a convolution process is performed on an input image of 251×251 pixels using a 3×3 filter, 249×249 convolution data 201 is obtained.
[0113] [Step 2-2] Next, the convolution processing unit 11a performs the same process as the above [Step 2-1] on the input image 101 using the second filter coefficient. This generates 249 x 249 pixel convolution data 202. Thereafter, the filter coefficients used in the convolution process are sequentially changed, and a total of 64 channels of 249 x 249 pixel convolution data 201, 202, ... 264 are generated.
[0114] In the following, for the sake of simplicity, the filter coefficients 221, 222, . . . 22 will be referred to as 221, 222, . . . 22 unless there is a need to distinguish them. 64 The filter coefficient is 22 100 and filter coefficient 22 100 The configuration for the filter processing using this is described as filter 22a, and the convoluted data 201, 202, ... 264 is described as convoluted data 200. In the following description, data of multiple channels generated by the same procedure will be described using the same reference numerals as in the above "convoluted data 200" unless there is a particular need to distinguish them.
[0115] [Step 2-4] The convolution processing unit 11a again convolutes the 249 x 249 x 64ch data obtained in [Step 2-3] above, thereby generating 247 x 247 pixel convolution data 200a with a total of 64 channels. For the convolution process, 642 filter coefficients and 64 bias values are used, and the 64 calculation results are added for each channel to obtain the 64ch convolution data 200a as output.
[0116] [Step 2-5] The convolution processing result obtained in step 2-4 is stored in the memory unit 15, and the cropping processing unit 13 extracts 56 × 56 pixels of data from approximately the center of the data as a feature map 35, generates feature maps for 64 channels, and stores them in the memory unit 15.
[0117] Through the above [Step 2-1] to [Step 2-5], the processing of the first hierarchical layer 41 is completed.
[0118] [Step S3: PSDS processing] [Step 3-1] The PSDS processor 11d of the downsampling unit 11 performs PSDS processing on the convolution data 200a of a total of 64 channels generated in [Step 2-1] to [Step 2-4] using the downsampling filter 22a. In this embodiment, the PSDS processing uses a window with N shifted start positions including the reference position, and by setting the stride value to 2 for a 2×2 window, for example, the 247×247 data resulting from the convolution processing of the first layer 41 is generated as 123×123×N (number of start positions) data, which is half the size.
[0119] [Step 3-2] Next, the PSDS processor 11d performs the same process as the above [Step 3-1] on the other convoluted data, thereby generating downsampled data 300 of 123×123×N×64 channels.
[0120] [Step S4: Convolution processing] [Step 4-1] Next, the convolution processing unit 11a performs convolution processing on each of the down-sampling data 300 generated in [Step 3-1] and [Step 3-2] to generate 121×121×64ch×N convolution data 371. Note that this convolution processing is also performed using the 3×3 window filter 22a, so that the data on each of the two sides, X and Y, is reduced by two.
[0121] [Step 4-2] Furthermore, the convolution processing unit 11a performs the convolution processing again, and 119×119×64ch×N pieces of convolution data 372 are generated.
[0122] [Step 4-3] The convolution processing result obtained in step 4-2 is stored in the memory unit 15. The cropping processing unit 13 extracts 32 × 32 data from approximately the center of the data as a feature map 35, generates feature maps for 64 × N channels, and stores them in the memory unit 15.
[0123] Through the above [Step 3-1], [Step 3-2], and [Step 4-1] to [Step 4-3], the processing of the second hierarchical layer 42 is completed.
[0124] Thereafter, the convolution processing unit 11a and the PSDS processing unit 11d repeat the processes of [Step 3-1], [Step 3-2], and [Step 4-1] to [Step 4-3], thereby sequentially processing the third layer 43, the fourth layer 44, and the fifth layer 45 shown in FIG. 7.
[0125] [2. Upsampling process procedure] 8 is a flowchart showing the procedure of the upsampling process according to this embodiment. The procedure of the upsampling process will be described below with reference to FIGS.
[0126] [Step S11: Obtain feature map] [Step 6-1] The feature maps for each layer extracted by the cropping unit 13 are stored in the memory unit 15, so a feature map for each layer is obtained using a skip connect (thick arrow) from the encoder side to the decoder side.
[0127] [Step S12: Upsampling] [Step 6-2] For example, the upsampling unit 12 receives the output of the fifth-layer convolution processing 381 (7×7×64ch×N 3 By upsampling the data, it becomes 14×14×64ch×N 3 The data is then compared with the 14×14×64ch×N data extracted by the crop processing unit 13. 3The feature map data of is merged with that of. Upsampling is performed by the upsampling processing unit 12b of the upsampling unit 12, and in this embodiment, the number of pixels is doubled by performing a 2×2 deconvolution process. Note that instead of the deconvolution process, the number of pixels may be increased by using a pixel shuffler process.
[0128] [Step S13: Convolution processing] [Step 6-3] The convolution processing unit 12a of the upsampling unit 12 converts the cropped and upsampled data 382 (14×14 pixels, 64×N 3 +14x14 pixels, 64chxN 3 ) is subjected to convolution processing to generate data 383.
[0129] [Step 6-4] The convolution processing unit 12a of the upsampling unit 12 performs the same convolution processing as in [Step 2-1] to [Step 2-4] above on each of the convolution data 383, and outputs a 10×10 pixel, 64ch×N 3 The convolution data 384 is generated.
[0130] With the above, the processing from the fifth hierarchical layer 45 to the fourth hierarchical layer 44 is completed.
[0131] [Steps S11 to S13: Repeat] [Step 7] The upsampling process for the third layer 43 to the first layer 41 is also performed in the same order as the above [Step 6-1] to [Step 6-4]. That is, a feature map cut out from approximately the center of the convolutional data on the encoder side is obtained, and merged with the upsampled convolutional data on the decoder side, and the convolution process is performed twice on the merged data to generate the encoder side data for the third layer to the first layer.
[0132] Although a detailed description of each layer will be omitted, decoding from the third layer 43 to the first layer 41 results in convoluted data 393 of 52×52 pixels and 64 channels.
[0133] [Step S14: Convolution processing] [Step 8] The convolution data 393 from the first layer encoder is subjected to 1×1 convolution in the convolution processing unit 12a of the upsampling unit 12 to obtain the output image 102. In this embodiment, a grayscale image is used as the input image, so a 52×52 pixel, 1ch output image is obtained, but if an RGB input image is used, the convolution output will be 3ch.
[0134] [3. Setting the filter coefficients] [Step 9] Fig. 9 is a flow chart showing the procedure for setting filter coefficients in information processing device 1 of this embodiment. Fig. 10 is a diagram showing a model of setting filter coefficients in information processing device 1 of this embodiment. In this embodiment, filter coefficient setting unit 14 sets filter coefficients used in each process in downsampling unit 11 and filter coefficients used in each process in upsampling unit 12. For the sake of convenience, the filter coefficients are denoted by the same reference numerals as in the above explanation of [Step 2-2] and the explanation of Fig. 7, and are referred to as filter coefficients 22. 100 However, the filter coefficient 22 shown in the following explanation is 100 The specific configuration (such as the values of the individual coefficients and the size of the matrix) is arbitrary and does not have to be the same as that shown in the explanation of [Step 2-2] above and the explanation of FIG.
[0135] The filter coefficient setting unit 14 determines the filter coefficient 22 by, for example, the following method. 100 Set.
[0136] [Step S21: Comparison with training data] In (Comparison 1) of FIG. 10, that is, by comparing the output image 102 with the training data 104, the filter coefficients 22 used in each convolution process on the encoding side and the decoding side are 100In this embodiment, since the output image has a size of 52×52 pixels, an example is described in which data from approximately the center of the original data 103 (data corresponding to the input image 101) is cropped and compared as teacher data. As a result, the filter coefficients 22 in each process are adjusted to be optimal for the information processing device 1 as a whole. 100 is set.
[0137] In this embodiment, the filter coefficients used in the single filter process are not learned and are fixed, but the filter coefficients may be configured to learn the filter coefficients for the single filter process.
[0138] [Step S22: Comparison with generated data] 10 (Comparison 2A), the original data 103 is downsampled by the downsampling unit 11, the original data (251 x 251 pixels) is reduced to half the size (the number of pixels is halved, the same applies below), and the approximate center of the data is cropped to generate teacher data 105. When downsampling, pixel shift downsampling processing and single filter processing are selected as the processing content to obtain channel data according to the number of starting positions of the pixel shift, and by using this as teacher data, a more detailed comparison can be made.
[0139] On the other hand, the crop & up sample data 388, convolution data 389, or convolution data 390 of the second layer 42 of the U-net are convoluted so that the number of channels of the training data obtained by the PSDS processing matches. The result of the convolution processing is compared with the training data 105, and the coefficients used for each convolution and the filter coefficients 22 used for the single filter processing are calculated. 100 , or adjust the window frame, stride value, number of start positions, start position, etc. of PSDS processing. Note that it is up to the design which of data 388, 389, and 390 is to be convoluted, and the optimal data can be used for comparison with the generated training data.
[0140] Furthermore, when any of the outputs (385, 386, 387) from the encoder side of the third hierarchical layer 43 is used for learning, the original data is reduced to 1 / 4 size (1 / 4 number of pixels, the same applies below) using PSDS processing, and approximately the center of the reduced data is cropped to create teacher data 106. By comparing this teacher data 106 with data obtained by convolution processing of any of the outputs (385, 386, 387) from the encoder side, more accurate learning can be performed.
[0141] To reduce original data to 1 / 4 size, one method is to use PSDS processing to directly obtain 1 / 4 size data from the original data as shown in the figure. In this case, the window frame used in PSDS processing is set to 4x4, the stride value to 4, and the number of starting positions to 16. Alternatively, data reduced to 1 / 2 size by PSDS processing may be further reduced to 1 / 4 size by PSDS processing. In either case, when original data is reduced to 1 / 4 size by PSDS processing, 16-channel data is generated, and by comparing this as training data, the filter coefficients 22 of each convolution process can be calculated. 100 The size of the window used in the pooling process and the calculations performed within the window are adjusted to be optimized.
[0142] [Variations of processing procedure] The above processing procedure is configured to use PSDS processing instead of pooling processing or convolution processing, which are used as the first processing from the second layer onwards of U-net, but it is also possible to apply PSDS processing to pooling processing or PSDS processing to convolution processing.
[0143] [Effects] As described above, in this embodiment, the image data 20 that has been subjected to the pooling process and filtering process is converted into new image data 57, 58, 59, 60 by performing an inverse pixel shuffler process. That is, the pixel data of the original image data 20 is rearranged into a plurality of new image data 57, 58, 59, 60. For example, as shown in FIG. 5, the pixels of the numbers 1, 2, 9, and 10 in the image data 20 are rearranged as pixel 59 in the first row and first column of the image data 57, 58, 59, 60.
[0144] This makes it possible to easily obtain image data 57, 58, 59, and 60 that are smaller than the original image data 20.
[0145] In addition, the number of pixels and the pixel state of the unprocessed image data 20 and the processed image data 57, 58, 59, 60 can be kept the same. Therefore, the accuracy of machine learning can be improved by performing machine learning using the image data 57, 58, 59, 60, which has a small amount of processed data and contains many of the characteristics of the original image data 20. This makes it possible to perform analysis and recognition with high accuracy in artificial intelligence using CNN while preventing the amount of data and processing load from becoming excessive.
[0146] Furthermore, since the processed image data 57, 58, 59, 60 can maintain the same number of pixels and pixel state as the unprocessed image data 20, the unprocessed image data 20 can be easily restored from the processed image data 57, 58, 59, 60.
[0147] In this embodiment, an inverse pixel shuffler process is performed on the original image data 20, and the data is rearranged on the new image data 57, 58, 59, and 60. Then, the rearrangement of pixels by the inverse pixel shuffler process can be performed for each small amount of data that can retain the characteristics of the original data. This makes it easier to perform analysis and recognition with high accuracy in artificial intelligence using CNN while preventing the amount of data and processing load from becoming excessive. In this embodiment, it is possible to perform various types of highly accurate machine learning based on the image data 20.
[0148] It goes without saying that the above-described embodiment is an example of the present invention, and the present invention is not limited to this embodiment. [Explanation of symbols]
[0149] 1. Information processing device 11b Pooling processing unit (pooling processing means) 11c Single filter processing unit (single filter processing means) 11e... Inverse pixel shuffler processing unit (inverse pixel shuffler means) 20 Image data (original data) 53 Extraction range 57, 58, 59, 60... Image data (new data)
Claims
1. An information processing device that performs data processing on digital data using a convolutional neural network, a single filter processing means for performing a single filter processing by applying a single filter to the original data, which is the digital data and is a collection of a plurality of pixels, a pooling processing means for performing a pooling processing for extracting data from the original data by performing a predetermined processing on the original data, or a convolution processing means for performing a convolution processing; and an inverse pixel shuffler means for performing an inverse pixel shuffler process as a process of rearranging the pixels of the processed original data output by the single filter processing means, the pooling processing means, or the convolution processing means.
2. 2. The information processing apparatus according to claim 1, wherein the digital data is image data.
3. 1. An information processing method by an information processing device that performs data processing on digital data using a convolutional neural network, comprising: a single filter processing procedure in which a single filter processing is performed to the original data, which is a collection of a plurality of pixels and is the digital data, and a calculation is performed by applying a single filter; a pooling processing procedure in which a pooling processing is performed to extract data from the original data by performing a predetermined processing on the original data; or a convolution processing procedure in which a convolution processing is performed; and an inverse pixel shuffler procedure in which an inverse pixel shuffler process is performed as a process of rearranging the pixels of the processed original data output in the single filter processing procedure, the pooling processing procedure, or the convolution processing procedure.
4. A program for causing a computer to function as the information processing device according to claim 1 or 2.
Citation Information
Patent Citations
Image processing apparatus, image processing method, program, method for manufacturing learned model, and image processing system
JP2021196951A
Processing data of a neural network
US20220138551A1
Image processing method and apparatus
US20220180480A1
System and Method for Image Super-Resolution
US20230153946A1
