Processing method and processing device using it

Abstract

The first processing section (100) generates a first feature map (720) by performing processing of a first neural network on the object image (700). The enlargement section (400) enlarges the first feature map (720). The combination section (500) generates a combined image by combining the first feature map (720) with the object image (700). The second processing section (300) generates a second feature map (724) by performing processing of a second neural network on the combined image.

Description

Technical Field

[0001] This disclosure relates to processing technology, and more particularly to a processing method for performing processing on input information and a processing apparatus using the same. Background Technology

[0002] Deep learning is used, for example, in image recognition processing. Deep learning is known as a methodology for machine learning using multi-layer neural networks, such as convolutional neural networks. Convolutional neural networks are formed by repeatedly performing convolution and pooling on local regions. Furthermore, a fully convolutional neural network has been proposed, in which the fully connected layers constituting the convolutional neural network are located within the convolutional layers (see, for example, Patent Document 1).

[0003] [Prior Technology Documents]

[0004] [Patent Literature]

[0005] Patent Document 1: International Publication No. 19 / 159419 Summary of the Invention

[0006] [The problem the invention aims to solve]

[0007] Since fully convolutional neural networks process a superposition of local regions, the positional relationships of elements within the image being processed are uncertain. To improve the accuracy of image recognition in neural networks, it is preferable to be able to determine the positional relationships of elements within the image.

[0008] This disclosure was made in view of the above circumstances, and its purpose is to provide a technique for improving the accuracy of image recognition in neural networks that do not contain fully connected layers.

[0009] [Technical solutions used to address technical problems]

[0010] To address the aforementioned issues, one processing apparatus disclosed herein includes: a first processing unit that generates a first feature map smaller than the object image by performing processing on an object image that is the object of processing using a first neural network; an enlargement unit that enlarges the first feature map generated in the first processing unit to make it the same size as the object image; a combination unit that generates a combined image by combining the enlarged first feature map in the enlargement unit with the object image; and a second processing unit that generates a second feature map smaller than the object image and larger than the first feature map by performing processing on the combined image generated in the combination unit using a second neural network. The first neural network of the first processing unit and the second neural network of the second processing unit do not contain fully connected layers. During learning, a first stage of learning is performed only for the first neural network of the first processing unit, and during learning, a second stage of learning is performed for the second neural network of the second processing unit after the first stage of learning for the first neural network has been performed.

[0011] Another aspect of this disclosure is a processing method. This method includes: generating a first feature map smaller than the object image by processing an object image (the object being processed) using a first neural network; enlarging the generated first feature map to make it the same size as the object image; generating a combined image by combining the enlarged first feature map with the object image; and generating a second feature map smaller than the object image and larger than the first feature map by processing the generated combined image using a second neural network. Neither the first nor the second neural network contains fully connected layers. During learning, a first stage of learning is performed only for the first neural network, and a second stage of learning is performed for the second neural network after the first stage of learning for the first neural network has been completed.

[0012] Furthermore, any combination of the above-mentioned constituent elements, or any manner of converting the present disclosure into a method, apparatus, system, computer program, or recording medium containing a computer program, is also valid as a form of the present disclosure.

[0013] [Invention Effects]

[0014] According to this disclosure, it is possible to improve the accuracy of image recognition in neural networks that do not contain fully connected layers. Attached Figure Description

[0015] Figure 1 (a) Figure 1 (b) is a diagram showing the configuration of the processing apparatus that serves as the comparison object in this embodiment.

[0016] Figure 2 It means Figure 1 (a) Figure 1 The diagram showing the composition of the dense blocks in (b).

[0017] Figure 3 This is a diagram illustrating the configuration of the processing apparatus in this embodiment.

[0018] Figure 4 (a) Figure 4 (b) is a diagram showing the configuration of the processing apparatus in this embodiment.

[0019] Figure 5 (a) Figure 5 (b) means Figure 3 , Figure 4 A diagram showing the processing overview in the enlarged section of (b).

[0020] Figure 6 (a) Figure 6 (c) represents Figure 3 , Figure 4 A diagram outlining the processing in the assembly section of (b).

[0021] Figure 7 It means Figure 4 (a) Figure 4 The timing diagram of the processing steps of the processing device in (b). Detailed Implementation

[0022] Before describing the specific embodiments of this disclosure, an outline of these embodiments will be provided. The embodiments relate to a processing apparatus for determining features within an image (hereinafter referred to as an "object image") by performing image recognition processing on an image that is the object of processing. A fully convolutional neural network without fully connected layers can be used for image recognition processing. In a fully convolutional neural network, convolutional and pooling layers are repeatedly processed to output an image (hereinafter referred to as a "segmentation image" or "feature map") that has been segmented based on coloring of the contained objects.

[0023] However, as mentioned above, since a fully convolutional neural network is a superposition of local region processing, the positional relationships of elements within an object image are uncertain. Since image recognition accuracy improves by utilizing the positional relationships of elements within the object image, it is desirable to determine these relationships. On the other hand, to determine the positional relationships of elements within the object image, the region to be recognized by the fully convolutional neural network needs to be expanded, but expanding the region itself reduces the accuracy of image recognition within that region. Therefore, it is necessary to balance the determination of the positional relationships of elements within the object image with the improvement of image recognition accuracy within that region.

[0024] The processing apparatus of this embodiment includes a network that identifies a wide area (hereinafter referred to as "first neural network") and a network that identifies a narrow area (hereinafter referred to as "second neural network"). A processed image is input to the first neural network. An image formed by combining a feature map generated by the first neural network (hereinafter referred to as "first feature map") with the processed image (hereinafter referred to as "combined image") is input to the second neural network. A feature map generated by the second neural network (hereinafter referred to as "second feature map") is output.

[0025] Figure 1 (a) Figure 1 (b) indicates the configuration of the processing device 10 that is the object of comparison. In particular, Figure 1 (a) represents the structure used for learning processing. Figure 1 (b) indicates the composition used for identification processing. Figure 1 The processing device 10 in (a) and Figure 1 The processing device 10 in (b) can be the same device or a different device.

[0026] like Figure 1 As shown in (a), the processing apparatus 10, configured for learning processing, includes: a convolutional layer 20, a batch normalization layer 22, dense blocks 24 and 26, a max pooling layer 28, dense blocks 30 and 32, and an average pooling layer 34. In the learning processing, a learning image 40 is used. The learning image 40 is an image prepared in advance as training data for the results of image recognition. The learning image 40 has, for example, a size of "28×28". This can be described as, for example, a portion of an image with a size of "1024×1024", i.e., a detailed image. The learning image 40 is input to the convolutional layer 20.

[0027] Convolutional layer 20 performs spatial filtering by shifting a spatial filter smaller than the size of the input image, for example, the learning image 40. In convolutional layer 20, the size of the spatial filter is defined as "5×5", and the shift magnitude, or span, of the spatial filter is defined as "2". Since spatial filtering is a well-known technique, its explanation is omitted here, but this spatial filtering is equivalent to convolution processing, which extracts features from the image. Padding and similar operations can also be performed in convolutional layer 20. Furthermore, convolutional layer 20 can use multiple spatial filters in parallel for the image, performing multiple spatial filters in parallel. Through the parallel use of multiple spatial filters, the image is enhanced. The number of spatial filters used in parallel in convolutional layer 20 is called the number of channels. Known techniques can be used in batch normalization layer 22. Intermediate information 42 is the result of processing based on convolutional layer 20 and batch normalization layer 22, representing information intermediate in the processing. Intermediate information 42 has a size of "14×14".

[0028] Dense blocks 24 Figure 2 As shown. Figure 2 This represents the structure of a dense block. As shown in the figure, it includes: intermediate information 60, convolutional layer 62, combination layer 64, intermediate information 66, convolutional layer 68, intermediate information 70, and batch normalization layer 72. Intermediate information 60, 66, and 70 are the same as intermediate information 42, representing information processed midway. For example, intermediate information 60 has 32 channels, intermediate information 66 has 64 channels, and intermediate information 70 has 32 channels. Convolutional layers 62 and 68 are the same as convolutional layer 20. The spatial filter size of convolutional layer 62 is specified as "3×3", and the spatial filter size of convolutional layer 68 is specified as "1×1". Combination layer 64 generates intermediate information 66 by combining the processing results of intermediate information 60 and convolutional layer 62. The processing results of intermediate information 60 and convolutional layer 62 are combined as different channels. Batch normalization layer 72 is the same as batch normalization layer 22. Figure 1 (a)

[0029] Dense blocks 26, 30, and 32 are constructed in the same manner as dense block 24. Dense blocks 24, 26, 30, and 32 are not limited to... Figure 2 The composition of the convolutional layers and other components can also be different. In addition, the compositions of dense blocks 24, 26, 30, and 32 can also be different from each other.

[0030] Max pooling layer 28 is constructed similarly to convolutional layer 20. Max pooling layer 28 reduces the image size by aggregating multiple pixels within any region of the image into a single pixel. To aggregate multiple pixels into a single pixel, max pooling layer 28 uses the average value of multiple pixel values ​​within the region. Pooling is performed to enhance robustness against translational shifts in the average value within the region of interest. In max pooling layer 28, the magnitude of the region shift, i.e., the span, is defined as "2".

[0031] The average pooling layer 34 performs pooling processing in the same way as the max pooling layer 28. To combine multiple pixels into one pixel, the average pooling layer 34 uses the maximum value among the multiple pixel values ​​within a region. Pooling is performed to enhance robustness against translational shifts of representative values ​​in the region of interest. The size of the region in the average pooling layer 34 is defined as "7×7".

[0032] In the learning process, the coefficients of the spatial filters of the convolutional layers 20 and the convolutional layers contained in dense blocks 24, 26, 30, and 32 are learned based on the learning image 40 and the training data. Since the learning of the spatial filter coefficients using the learning image 40 and the training data can be performed using known techniques, a description of this process is omitted here.

[0033] like Figure 1 As shown in (b), the processing apparatus 10, as a configuration for identification processing, includes a convolutional layer 20 to an average pooling layer 34. This configuration is consistent with... Figure 1 (a) is the same. Here, through Figure 1 The coefficients of the spatial filter derived from the learning process in (a) are assigned to convolutional layer 20 and the convolutional layers contained in dense blocks 24, 26, 30, and 32. Object image 50 is an image of the object to be recognized in the processing device 10, for example, having a size of "1024 × 1024". Object image 50 is input to convolutional layer 20.

[0034] Since convolutional layers 20 to average pooling layers 34 perform the same processing as before, details are omitted here. Additionally, during the recognition process, intermediate information 52 and intermediate information 54 are generated. Intermediate information 52 has a size of "512×512", and intermediate information 54 has a size of "256×256". The result of the recognition process is output from average pooling layer 34.

[0035] Since the learning image 40 has a size of 28×28 and the object image 50 has a size of 1024×1024, the learning processing using the learning image 40 can be considered as being performed on a narrow domain. Therefore, the processing device 10 can identify features in the narrow domain in detail. Here, when different features are similar in the narrow domain, it is difficult for the processing device 10 to distinguish these features. On the other hand, if these features are arranged in different positions in the overall image, they can be distinguished based on the relative positional relationships of the features. Therefore, learning processing is required for a wide domain. However, when learning processing is performed on a wide domain, the processing device 10 cannot identify features in the narrow domain in detail. Therefore, it is required to achieve both detailed identification of features in the narrow domain and identification of the positions of features in the wide domain.

[0036] Here, we will explain in the order of (1) composition, (2) learning processing, and (3) identification processing.

[0037] (1) Composition

[0038] Figure 3This describes the configuration of the processing apparatus 1000. This configuration is used for identification processing. The processing apparatus 1000 includes a first processing unit 100, an amplification unit 400, a combination unit 500, and a second processing unit 300. The first processing unit 100 includes a batch normalization layer 110, a convolutional layer 112, a dense block 114, and a max-pooling layer 118. The configuration between the dense block 114 and the max-pooling layer 118 is omitted. The second processing unit 300 includes a batch normalization layer 310, a convolutional layer 312, a dense block 314, a dense block 316, and a max-pooling layer 318. Neither the first processing unit 100 nor the second processing unit 300 includes fully connected layers.

[0039] (2) Learning Processing

[0040] In the learning process, the second stage of learning is conducted after the first stage. Figure 4 (a) Figure 4 (b) indicates the configuration of the processing device 1000. Figure 4 (a) represents the components used for learning in stage 1. Figure 4 (b) represents the composition used for learning in stage 2. Figure 4 In the first phase of learning shown in (a), only targeted Figure 3 The first processing unit 100 performs learning. In the first stage of learning, a learning image 650 is used. The learning image 650 is an image of the image recognition result prepared in advance as training data. The learning image 650 has, for example, a size of "112×112". The learning image 650 is input to the first processing unit 100.

[0041] The batch normalization layer 110, convolutional layer 112, dense block layer 114, and max pooling layer 118 in the first processing unit 100 are included in the first neural network. They are related to... Figure 1 (a)-(b) Figure 2 The same processing is performed. The size of the spatial filter of convolutional layer 112 is specified as "5×5", and the span is specified as "2". The span of max pooling layer 118 is specified as "2".

[0042] Intermediate information 652 is generated in the first processing unit 100. The intermediate information 652 has a size of 56×56. In the first stage of learning, the coefficients of the spatial filter of the convolutional layer included in the first processing unit 100 are learned based on the learning image 650 and the training data. Since the learning of the spatial filter coefficients can be performed using known techniques, a description is omitted here.

[0043] Figure 4The second stage of learning, as shown in (b), is performed on the second processing unit 300 after the first stage of learning has been performed on the first processing unit 100. Therefore, coefficients of the spatial filter derived from the first stage of learning are set in each convolutional layer included in the first processing unit 100. The learning image 650 is also used in the second stage of learning. The learning image 650 is input to the first processing unit 100 and the combination unit 500. The first processing unit 100 performs recognition processing on the learning image 650 to generate intermediate information 654. The intermediate information 654 has a "7×7" dimension. Therefore, the first processing unit 100 converts the "112×112" size to a "7×7" size.

[0044] Because the learning image 650 and the intermediate information 654 have different sizes, they cannot be combined. Here, the first processing unit 100 is connected to the magnification unit 400, and the magnification unit 400 is connected to the combination unit 500. The magnification unit 400 generates intermediate information 656 by magnifying the intermediate information 654 generated in the first processing unit 100. The intermediate information 656 has the same "112×112" size as the learning image 650.

[0045] Figure 5 (a) Figure 5 (b) indicates the processing summary in the magnification section 400. Figure 5 (a) represents an example of the processing in the magnification unit 400. Pixels 850a to 850h, collectively referred to as pixels 850, are the pixels constituting the intermediate information 654. An additional pixel 852 is added between adjacent pixels 850. For example, a third additional pixel 852c is added between pixel 850a, pixel 850c, pixel 850d, and pixel 850e. The same applies to additional pixels 852a, 852d, 852e, 852g, and 852h. By adding additional pixels 852 to pixels 850 in this way, the intermediate information 654 is magnified. The magnification unit 400 copies the value of pixel 850 to the adjacent additional pixels 852. For example, the value of pixel 850c is copied to the third additional pixel 852c. As a result, intermediate information 656 is generated.

[0046] Figure 5 (b) shows another example of the processing in the magnification section 400. Pixel 850 and appended pixel 852 with Figure 5The same as (a). The magnifying unit 400 uses the value obtained by interpolating the values ​​of multiple pixels 850 surrounding the additional pixel 852 for the additional pixel 852. For example, the value obtained by interpolating the value of the first pixel 850a, the value of the third pixel 850c, the value of the fourth pixel 850d, and the value of the fifth pixel 850e is used for the third additional pixel 852c. Since the interpolation can be performed using well-known techniques, the explanation is omitted here. Return Figure 4 (b)

[0047] The combination unit 500 combines the learning image 650 with the intermediate information 656. Figure 6 (a) Figure 6 (c) represents a processing summary in the assembly unit 500. The first input information 800a and the second input information 800b are two pieces of information input to the assembly unit 500, corresponding to the learning image 650 and intermediate information 656. Figure 6 In (a), the first input information 800a and the second input information 800b are combined as different channels. Figure 6 In (b), the first input information 800a and the second input information 800b are arranged horizontally to form the information of one channel. Figure 6 In (c), the first input information 800a and the second input information 800b are arranged along the vertical axis to form the information of one channel. Return Figure 4 (b). The combination unit 500 can also combine the two inputs by performing addition, subtraction, and other arithmetic operations on each pixel.

[0048] The extraction unit 200 extracts a portion, such as the central portion, of the image combined in the combination unit 500 (hereinafter referred to as the "combined image"). The combined image has a size of "112×112", and the combined image with the extracted portion (hereinafter also referred to as the "learning image") has a size of "28×28". The learning image is input to the second processing unit 300.

[0049] The batch normalization layer 310, convolutional layer 312, dense block layer 314, dense block layer 316, and max pooling layer 318 in the second processing unit 300 are included in the second neural network. They are related to... Figure 1 (a)-(b) Figure 2 The same processing is performed. The size of the spatial filter of convolutional layer 312 is specified as "5×5", and the span is specified as "2". The span of max pooling layer 318 is specified as "2".

[0050] Intermediate information 662 is generated in the second processing unit 300. The intermediate information 662 has a size of "14×14". Therefore, the second processing unit 300 converts the "28×28" size to a "7×7" size. In the first stage of learning, while the first stage of learning has been performed, the coefficients of the spatial filter of the convolutional layer included in the second processing unit 300 are learned based on the learning image 650 and the training data. Since the learning of the spatial filter coefficients can be performed using known techniques, an explanation is omitted here.

[0051] The 112×112 learning image 650 is used for learning in the first processing unit 100, and the 28×28 learning image is used for learning in the second processing unit 300. Since the size of the learning image 650 is larger than the size of the learning image, the first network of the first processing unit 100 uses the wide area as the recognition object, and the second network of the second processing unit 300 uses the narrow area as the recognition object.

[0052] (3) Identification Processing

[0053] pass Figure 4 The coefficients of the spatial filter derived from the learning process in (a)–(b) are set to Figure 3 Each convolutional layer in the [process]. In [the context of]... Figure 3 The recognition process shown uses an object image 700. The object image 700 is an image of the object to be recognized in the processing apparatus 1000, and has, for example, a size of "1024×1024". The object image 700 is input to the first processing unit 100 and the combination unit 500. The extraction unit 200 is not used in the recognition process.

[0054] Since the batch normalization layer 110 to the max pooling layer 118 in the first processing unit 100 perform the same processing as before, their description is omitted here. Additionally, intermediate information 702 is generated during the recognition process. The intermediate information 702 has a size of "512×512". The first processing unit 100 generates a first feature map 720 by performing processing on the object image 700 using the first neural network. The first feature map 720 has a size of "64×64".

[0055] In order to make the first feature image 720 the same size as the object image 700, the magnification unit 400 magnifies the first feature image 720 to generate a first feature image 722. The first feature image 722 has a size of "1024×1024". The combination unit 500 generates a combined image by combining the magnified first feature image 722 in the magnification unit 400 with the object image 700. The combination process is performed as described above. The combination unit 500 outputs the combined image to the second processing unit 300.

[0056] Since the batch normalization layer 310 to the max pooling layer 318 in the second processing unit 300 perform the same processing as before, the description is omitted here. Additionally, intermediate information 712 is generated during the recognition process. The intermediate information 712 has a size of "512×512". The second processing unit 300 generates a second feature map 724 by performing processing on the combined image using the second neural network. The second feature map 724 has a size of "256×256", which is smaller than the object image 700 and larger than the first feature map 720.

[0057] The main body of the apparatus, system, or method disclosed herein includes a computer. The computer executes a program to achieve the functions of the main body of the apparatus, system, or method disclosed herein. As a primary hardware component, the computer includes a processor that operates according to a program. The processor is not limited in type, as long as it can perform the function by executing the program. The processor consists of one or more electronic circuits, including semiconductor integrated circuits (ICs) or LSIs (Large Scale Integration). Multiple electronic circuits can be integrated into one chip or disposed on multiple chips. Multiple chips can be concentrated in one device or disposed in multiple devices. The program is recorded in a non-transitory recording medium such as a computer-readable ROM, optical disk, or hard disk drive. The program can also be pre-stored in the recording medium or supplied to the recording medium via a wide area communication network including the Internet.

[0058] The operation of the processing device 1000 configured above will be explained. Figure 7 This is a timing diagram showing the processing steps of the processing device 1000. The processing device 1000 performs a first stage of learning (S10) on the first processing unit 100. While using the learning results of the first processing unit 1000, the processing device 1000 performs a second stage of learning on the extraction unit 200 (S12).

[0059] According to this embodiment, during learning, since the learning image 650 is input to the first processing unit 100 and a portion of the combined image is input to the second processing unit 300, a network for recognizing objects in a wide area and a network for recognizing objects in a narrow area can coexist. Furthermore, since the networks for recognizing objects in a wide area and a network for recognizing objects in a narrow area coexist, it is possible to achieve both the determination of the positional relationships of elements within the object image and the improvement of image recognition accuracy within the region. Additionally, since the combined image obtained by combining the object image 700 and the first feature map 722 is input to the second processing unit 300, it is possible to achieve both the determination of the positional relationships of elements within the object image and the improvement of image recognition accuracy within the region. Moreover, since it is possible to achieve both the determination of the positional relationships of elements within the object image and the improvement of image recognition accuracy within the region, it is possible to improve the image recognition accuracy in neural networks that do not contain fully connected layers.

[0060] Furthermore, since the size of the first feature map is changed by the magnification section 400, the conversion magnification in the first processing unit 100 and the conversion magnification in the second processing unit 300 can be set independently. Also, since the conversion magnification in the first processing unit 100 and the conversion magnification in the second processing unit 300 are set independently, processing adapted to the first processing unit 100 and the second processing unit 300 respectively can be performed. Furthermore, since processing adapted to the first processing unit 100 and the second processing unit 300 respectively is performed, recognition accuracy can be improved. Furthermore, since the two inputs are combined as different channels, combination can be easily performed. Furthermore, since arithmetic operations are performed on the two inputs for each pixel and then combined, combination can be easily performed.

[0061] An outline of one aspect of this disclosure is described below. A processing apparatus (1000) of one aspect of this disclosure includes: a first processing unit (100) that generates a first feature map (720) smaller than the object image (700) by performing processing on an object image (700) that is the object of processing using a first neural network; an enlargement unit (400) that enlarges the first feature map (720) generated in the first processing unit (100) to make it the same size as the object image (700); a combination unit (500) that generates a combined image by combining the enlarged first feature map (722) in the enlargement unit (400) with the object image (700); and a second processing unit (300) that generates a second feature map (724) smaller than the object image (700) and larger than the first feature map (720) by performing processing on the combined image generated in the combination unit (500) using a second neural network. The first neural network of the first processing unit (100) and the second neural network of the second processing unit (300) do not contain fully connected layers. During learning, a first stage of learning is performed only for the first neural network of the first processing unit (100). During learning, while the first stage of learning for the first neural network has been performed, a second stage of learning for the second neural network of the second processing unit (300) is performed.

[0062] The combining unit (500) can combine two inputs as different channels.

[0063] Another aspect of this disclosure is a processing method. This method includes: generating a first feature map (720) smaller than the object image (700) by processing an object image (700) that is the object of processing using a first neural network; enlarging the generated first feature map (720) to make it the same size as the object image (700); generating a combined image by combining the enlarged first feature map (722) with the object image (700); and generating a second feature map (724) smaller than the object image (700) and larger than the first feature map (720) by processing the generated combined image using a second neural network. Neither the first nor the second neural network contains fully connected layers. During learning, a first stage of learning is performed only for the first neural network, and during learning, a second stage of learning is performed for the second neural network after the first stage of learning for the first neural network has been completed.

[0064] The present disclosure has been described above based on embodiments. Those skilled in the art should understand that these embodiments are merely illustrative, and the combination of each component element or processing step can have various modifications, and such modifications are also within the scope of this disclosure.

[0065] [Industrial Availability]

[0066] According to this disclosure, it is possible to improve the accuracy of image recognition in neural networks that do not contain fully connected layers.

[0067] [Explanation of reference numerals in the attached figures]

[0068] 10 Processing unit, 20 Convolutional layers, 22 Batch normalization layers, 24 and 26 Dense blocks, 28 Max pooling layers, 30 and 32 Dense blocks, 34 Average pooling layers, 62 Convolutional layers, 64 Combined layers, 68 Convolutional layers, 72 Batch normalization layers, 100 First processing unit, 110 Batch normalization layers, 112 Convolutional layers, 114 Dense blocks, 118 Max pooling layers, 200 Extraction unit, 300 Second processing unit, 310 Batch normalization layers, 312 Convolutional layers, 314 and 316 Dense blocks, 318 Max pooling layers, 400 Amplification unit, 500 Combined unit, 1000 Processing unit.

Claims

1. A processing apparatus, comprising: The first processing unit generates a first feature map smaller than the object image by performing processing on the object image of the object image using a first neural network. The magnification section enlarges the first feature map generated in the first processing section to make it the same size as the object image. The combining section generates a combined image by combining the first feature image, magnified in the magnifying section, with the object image, and... The second processing unit generates a second feature map with a size smaller than the object image and larger than the first feature map by performing a second neural network processing on the combined image generated in the combination unit; The first processing unit converts the object image to the first feature map at a first magnification, and the second processing unit converts the combined image to the second feature map at a second magnification, which are set independently of each other; The first neural network of the first processing unit and the second neural network of the second processing unit do not contain fully connected layers; The processing device performs learning processing including a first stage of learning and a second stage of learning. In the first stage of learning, based on the learning image, the coefficients of the spatial filter of the convolutional layer in the first processing unit are learned only. After the first stage of learning is performed, the second stage of learning is performed. In the second stage of learning, based on the learning image, the coefficients of the spatial filters of the convolutional layers in the second neural network of the second processing unit are learned.

2. The processing apparatus as described in claim 1, characterized in that, The combining unit merges the two inputs as different channels.

3. A processing method, comprising: The step of generating a first feature map smaller than the object image by performing processing on the object image that is the object to be processed by the first neural network. The step of enlarging the generated first feature map to make it the same size as the object image. The step of generating a combined image by combining the magnified first feature map with the object image, and The step of generating a second feature map with a size smaller than the object image and larger than the first feature map by performing the second neural network processing on the generated combined image; The first magnification rate for converting the object image into the first feature map and the second magnification rate for converting the combined image into the second feature map are set independently of each other; Neither the first neural network nor the second neural network contains fully connected layers; The learning process includes two phases: Phase 1 and Phase 2. In the first stage of learning, based on the learning image, the coefficients of the spatial filters of the convolutional layers in the first neural network are learned only. After the first stage of learning is performed, the second stage of learning is performed. In the second stage of learning, based on the learning image, the coefficients of the spatial filters of the convolutional layers in the second neural network are learned while the first stage of learning for the first neural network has been performed.

Images