IMAGE PROCESSING APPARATUS, IMAGE PROCESSING METHOD, AND IMAGE PROCESSING PROGRAM
The image processing device enhances deep learning model compression by reducing unnecessary data based on pixel influence, improving the compression rate and efficiency of data transmission.
Patent Information
- Application Number
- JP2022569366
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-12-15
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2040-12-15
AI Technical Summary
Existing deep learning models compress intermediate information that includes unnecessary data, leading to suboptimal compression rates.
An image processing device that includes an image reduction unit to reduce the amount of information in image data based on the influence of each pixel on the processing result, followed by a compression unit to compress the reduced data, using techniques like quantization and encoding.
Improves the compression rate of intermediate information extracted from deep learning models by selectively reducing unnecessary data, enhancing the efficiency of data transmission.
Smart Images

Figure 0007589752000001 
Figure 0007589752000002 
Figure 0007589752000003
Abstract
Description
[Technical field]
[0001] The present invention relates to an image processing device, an image processing method, and an image processing program. [Background technology]
[0002] As a technique for compressing and transmitting image data used in image analysis processing by a deep learning model, for example, a technique is known in which image data is input in advance to a deep learning model, and intermediate information (feature map) extracted from an intermediate layer is compressed and transmitted. This compression technique can achieve a higher compression rate than when image data is directly compressed and transmitted, and can output appropriate processing results in the output layer of the deep learning model at the transmission destination, as in the case of directly compressing and transmitting image data. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2018-195231 A [Patent Document 2] JP 2019-036899 A [Patent Document 3] JP 2018-097662 A [Patent Document 4] JP 2019-029938 A Summary of the Invention [Problem to be solved by the invention]
[0004] However, the intermediate information extracted from the intermediate layer of the deep learning model includes not only information necessary to output appropriate processing results in the output layer, but also information that is not necessary to output appropriate processing results.
[0005] In one aspect, an object of the present invention is to improve the compression rate when compressing intermediate information extracted from a deep learning model. [Means for solving the problem]
[0006] According to one aspect, an image processing device includes: A calculation unit that calculates an influence degree of each pixel that affects a processing result by the deep learning model in the image data by inputting image data to an input layer of the deep learning model and back-propagating an error based on a processing result output from an output layer and a reference result; Before being input to the input layer Each pixel of the image data Of which, The degree of impact By processing pixels whose pixel value is equal to or smaller than a predetermined threshold value, Processing to reduce the amount of information of the image data Generate reduced image data A processing unit for performing the processing; After the reduction Image data is input to the input layer of the deep learning model, and a compression unit compresses intermediate information extracted from the intermediate layer. Effect of the Invention
[0007] It is possible to improve the compression rate when compressing intermediate information extracted from a deep learning model. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration of an image processing system. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an edge device. [Diagram 3] FIG. 3 is a first diagram illustrating an example of functional configurations of an image reduction unit, an important part extraction unit, and a compression unit of the edge device. [Figure 4] FIG. 4 is a first diagram showing a specific example of the processing by the image reduction unit and the important part extraction unit. [Diagram 5] FIG. 5 is a first flowchart showing the flow of the compression process by the edge device. [Figure 6] FIG. 6 is a second diagram illustrating an example of a functional configuration of the image reduction unit and the compression unit of the edge device. [Figure 7]FIG. 7 is a second diagram showing a specific example of the processing by the image reduction unit and the important part extraction unit. [Figure 8] FIG. 8 is a second flowchart showing the flow of the compression process by the edge device. [Figure 9] FIG. 9 is a third diagram illustrating an example of functional configurations of the image reduction unit, the important part extraction unit, and the compression unit of the edge device. [Figure 10] FIG. 10 is a third diagram showing a specific example of the processing by the image reduction unit and the important part extraction unit. [Figure 11] FIG. 11 is a third flowchart showing the flow of the compression process by the edge device. [Figure 12] FIG. 12 is a fourth diagram illustrating an example of functional configurations of the image reduction unit, the important part extraction unit, and the compression unit of the edge device. [Figure 13] FIG. 13 is a fourth diagram showing a specific example of the processing by the image reduction section and the important part extraction section. [Figure 14] FIG. 14 is a fourth flowchart showing the flow of the compression process by the edge device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and the drawings, components having substantially the same functional configurations are denoted by the same reference numerals, and redundant description will be omitted.
[0010] [First embodiment] <Image processing system configuration> First, a description will be given of the system configuration of an entire image processing system including an edge device which is an example of an image processing apparatus according to the first embodiment. Fig. 1 is a diagram showing an example of the system configuration of the image processing system.
[0011] As shown in FIG. 1, the image processing system 100 includes an imaging device 110, an edge device 120, and a server device .
[0012] The imaging device 110 captures images at a predetermined frame period and transmits the image data to the edge device 120. Note that the image data may contain an object that is a target of image analysis processing by a deep learning model, which will be described later. If the image data does not contain an object that is a target of image analysis processing by a deep learning model, which will be described later, the entire image data will be invalidated, for example, by image processing, which will be described later.
[0013] An image processing program is installed in the edge device 120, and the edge device 120 functions as an image reduction unit 121, an important part extraction unit 122, and a compression unit 123 by executing the program.
[0014] The image reduction unit 121 is an example of a reduction unit, and includes a deep learning model 140. As shown in Fig. 1, in this embodiment, in the deep learning model 140, each layer from the input layer to an intermediate layer (e.g., the second layer) from which intermediate information ("feature map") is extracted is referred to as a front-stage part. In addition, in the deep learning model 140, each layer from the layer next to the intermediate layer from which the feature map is extracted to the output layer is referred to as a rear-stage part.
[0015] The image reduction unit 121 reduces the amount of information of the feature map extracted from the intermediate layer (e.g., the second layer) located at the backmost position in the front-stage unit by reducing the amount of information of the image data input to the front-stage unit. In this way, the image reduction unit 121 generates a "reduced feature map." In addition, the image reduction unit 121 notifies the compression unit 123 of the generated reduced feature map.
[0016] The important part extraction unit 122 is an example of a calculation unit, and generates an "important feature map" that indicates the degree of influence of each pixel in the image data that affects the processing result by the deep learning model 140. The generated important feature map is notified to the image reduction unit 121, and is used when reducing the amount of information of the image data input to the previous stage.
[0017] The compression unit 123 compresses the reduced feature map notified by the image reduction unit 121 by performing quantization and / or encoding processing, thereby generating a “compressed feature map.” In addition, the compression unit 123 transmits the compressed feature map to the server device 130.
[0018] As described above, in the first embodiment, when compressing a feature map extracted from the intermediate layer of the deep learning model 140, the amount of information of the feature map is reduced by reducing the amount of information of the image data, and the reduced feature map is generated and then compressed. As a result, according to the first embodiment, it is possible to improve the compression rate when compressing the feature map.
[0019] An image analysis processing program is installed in the server device 130, and the server device 130 functions as a decoding unit 131 and an image analysis unit 132 by executing the program.
[0020] The decoding unit 131 receives the compressed feature map transmitted from the edge device 120, and generates a reduced feature map by performing inverse quantization and / or decoding processing on the received compressed feature map. In addition, the decoding unit 131 notifies the image analysis unit 132 of the generated reduced feature map.
[0021] The image analysis unit 132 has a rear stage of the deep learning model 140, and receives the reduced feature map notified by the decoding unit 131 as input, and outputs the processing result from the output layer.
[0022] <Edge device hardware configuration> Next, a hardware configuration of the edge device 120 will be described. Fig. 2 is a diagram showing an example of a hardware configuration of the edge device. The edge device 120 has a processor 201, a memory 202, an auxiliary storage device 203, an I / F (Interface) device 204, a communication device 205, and a drive device 206. The hardware components of the edge device 120 are connected to each other via a bus 207.
[0023] The processor 201 has various arithmetic devices such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), etc. The processor 201 reads various programs (for example, an image processing program, etc.) onto the memory 202 and executes them.
[0024] The memory 202 has a main storage device such as a ROM (Read Only Memory) and a RAM (Random Access Memory). The processor 201 and the memory 202 form a so-called computer, and the computer realizes various functions (an image reduction unit 121, an important part extraction unit 122, and a compression unit 123) by the processor 201 executing various programs read onto the memory 202. The functional configurations of the various functions will be described in detail later.
[0025] The auxiliary storage device 203 stores various programs and various data used when the processor 201 executes the various programs.
[0026] The I / F device 204 is a connection device that connects an operation device 210, which is an example of an external device, a display device 211, and the edge device 120. The I / F device 204 accepts operations for the edge device 120 via the operation device 210. The I / F device 204 also outputs results of internal processing by the edge device 120 and displays them via the display device 211.
[0027] The communication device 205 is a communication device for communicating with other devices. In the case of the image processing system 100, the edge device 120 communicates with the imaging device 110 and the server device 130 via the communication device 205.
[0028] The drive device 206 is a device for setting the recording medium 212. The recording medium 212 here includes media that record information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, a magneto-optical disk, etc. The recording medium 212 may also include semiconductor memories that record information electrically, such as ROM, flash memory, etc.
[0029] The various programs to be installed in the auxiliary storage device 203 are installed, for example, by setting the distributed recording medium 212 in the drive device 206 and reading out the various programs recorded in the recording medium 212 by the drive device 206. Alternatively, the various programs to be installed in the auxiliary storage device 203 may be installed by being downloaded from a network via the communication device 205.
[0030] <Functional configuration of the image reduction unit, important part extraction unit and compression unit> Next, a detailed description will be given of the functional configuration of various functions (image reduction unit 121, important part extraction unit 122, and compression unit 123) realized by executing the image processing program in edge device 120. Fig. 3 is a first diagram showing an example of the functional configuration of the image reduction unit, important part extraction unit, and compression unit of the edge device.
[0031] As shown in FIG. 3, the image reduction unit 121 includes a front-stage unit 301 , a rear-stage unit 302 , an error calculation unit 303 , and an image processing unit 304 .
[0032] The front-stage unit 301 has each layer of the deep learning model 140, from the input layer to the intermediate layer from which a feature map is extracted. Upon receiving image data, the front-stage unit 301 extracts a feature map from the intermediate layer and notifies the rear-stage unit 302. Furthermore, upon receiving "reduced image data", the front-stage unit 301 extracts a reduced feature map from the intermediate layer and notifies the compression unit 123. Note that the reduced image data is an image generated by processing image data based on an important feature map, and is generated by an image processing unit 304 (details will be described later).
[0033] The rear stage unit 302 has each layer of the deep learning model 140 from the layer next to the intermediate layer from which the feature map is extracted to the output layer. The rear stage unit 302 receives the feature map as input and outputs a processing result from the output layer. The rear stage unit 302 also notifies the error calculation unit 303 of the processing result output from the output layer.
[0034] The error calculation unit 303 calculates the error between the processing result notified from the subsequent unit 302 and a reference result. The reference result refers to a predetermined classification probability for an object (correct answer data) included in the image data. For example, if the image processing system 100 is a system intended to provide a processing result used for analyzing the behavior of a person appearing in the image data, the image reduction unit 121 calculates, for example, A given area of image data (x 1 , y 1 , h 1 , w 1 ) object as a human, with a classification probability of 0.8. A given area of image data (x 1 , y 1 , h 1 , w 1 ) object as a non-human object with a classification probability of 0.1, The data set etc. that satisfies the criteria is specified as the reference result.
[0035] Furthermore, the error between the processing result and the reference result refers to, for example, the difference between the classification probability of each object in the processing result notified by the subsequent unit 302 and the classification probability of each object in the reference result. Note that the error may include, in addition to the difference in classification probability, an index (for example, IoU (Intersection Over Union)) indicating the amount of deviation between a predetermined area included in the processing result notified by the subsequent unit 302 and a predetermined area included in the reference result.
[0036] Furthermore, the error calculation unit 303 back-propagates the calculated error. This allows the important part extraction unit 122 to generate an important feature map that indicates the degree of influence of each pixel that affects the processing result by the deep learning model 140 in the image data.
[0037] The method by which the error calculation unit 303 backpropagates the error includes a number of methods such as "normal backpropagation", "guided backpropagation", "selective backpropagation", and "extended selective backpropagation".
[0038] The normal backpropagation is a method of backpropagating all the errors of the processing results notified by the rear stage unit 302. The guided backpropagation is a method of backpropagating the errors using only the positive gradients among the gradients calculated in each layer in the front stage unit 301 and the rear stage unit 302.
[0039] Selective backpropagation is a method of backpropagating only the error of the correct processing result among the processing results notified from the subsequent stage 302 by "normal backpropagation" or "guided backpropagation".
[0040] Extended selective backpropagation is a method of backpropagating large and small errors obtained by performing a specified operation on the processing result notified by the downstream unit 302 using "normal backpropagation" or "guided backpropagation."
[0041] The image processing unit 304 processes the image data using an important feature map notified by the important part extraction unit 122 described below to reduce the amount of information in the image data and generate reduced image data. Specifically, the image processing unit 304 processes the image data based on the influence degree of each pixel in the important feature map notified by the important part extraction unit 122 to reduce the amount of information in the image data and generate reduced image data.
[0042] The image processing unit 304 may process the image data in any manner. For example, pixels whose influence is equal to or less than a predetermined threshold in the important feature map may be identified, and the pixel value of the identified pixel in the image data may be set to zero (the identified pixel may be invalidated). Alternatively, pixels whose influence is equal to or less than a predetermined threshold in the important feature map may be identified, and a low-pass filter may be applied to the image data of the identified pixel. Alternatively, pixels whose influence is equal to or less than a predetermined threshold in the important feature map may be identified, and the color of the image data of the identified pixel may be reduced. In other words, processing the image data means processing the image data so that the deep learning model 140 does not recognize unnecessary features as features, and any processing method that achieves this purpose is acceptable.
[0043] Furthermore, the image processing unit 304 notifies the generated reduced image data to the front-stage unit 301. As described above, the front-stage unit 301 that has been notified of the reduced image data extracts a reduced feature map from the intermediate layer and notifies the compression unit 123.
[0044] The important part extraction unit 122 generates an important feature map using the back-propagated error. As described above, the important feature map represents the degree of influence of each pixel of the image data on the processing result. The important part extraction unit 122 notifies the image processing unit 304 of the generated important feature map.
[0045] As shown in FIG. 3, the compression unit 123 includes a quantization unit 311 and an encoding unit 312 .
[0046] The quantization unit 311 quantizes the reduced feature map notified by the front-stage unit 301 of the image reduction unit 121 , and notifies the encoding unit 312 of the quantization result.
[0047] The encoding unit 312 generates a compressed feature map by performing, for example, entropy encoding processing or any other compression processing on the quantized reduced feature map notified by the quantization unit 311. In addition, the encoding unit 312 transmits the generated compressed feature map to the server device 130.
[0048] <Specific examples of the image reduction unit and important part extraction unit> Next, a specific example of processing by the image reduction unit 121 and important part extraction unit 122 of the edge device 120 will be described. Fig. 4 is a first diagram showing a specific example of processing by the image reduction unit and important part extraction unit. As shown in Fig. 4, when image data 410 is inputted into the image reduction unit 121, the front-stage unit 301 and rear-stage unit 302 operate and output a processing result. Next, in the image reduction unit 121, the error calculation unit 303 operates to calculate an error between the processing result and a reference result, and then back-propagates the calculated error.
[0049] Next, the important part extraction unit 122 operates and uses the back-propagated error to generate an important feature map 420. In the important feature map 420 shown in Fig. 4, pixels that have a large influence on the processing result are shown in white, and pixels that have a small influence are shown in black.
[0050] Next, in the image reduction section 121, the image processing section 304 operates to invalidate pixels in the image data 410 whose influence level is equal to or less than a predetermined threshold in the important feature map 420, thereby generating reduced image data 430.
[0051] Next, the image reduction unit 121 inputs the reduced image data 430 to the front-stage unit 301 to operate the front-stage unit 301 again, and extracts a feature map from the intermediate layer (the second layer in the example of FIG. 4) of the front-stage unit 301. Furthermore, the image reduction unit 121 notifies the compression unit 123 of the extracted feature map as a reduced feature map.
[0052] <Flow of compression processing by edge devices> Next, a description will be given of the flow of compression processing by the edge device 120. Fig. 5 is a first flowchart showing the flow of compression processing by the edge device.
[0053] In step S501, each section (here, the front-stage section 301 and the rear-stage section 302) of the image reduction section 121 of the edge device 120 and the important part extraction section 122 are initialized.
[0054] In step S502, the image reduction unit 121 of the edge device 120 operates the front-stage unit 301. The front-stage unit 301 receives image data and extracts a feature map.
[0055] In step S503, the image reduction unit 121 of the edge device 120 operates the subsequent stage unit 302. The subsequent stage unit 302 receives the feature map and outputs a processing result.
[0056] In step S504, the image reduction unit 121 of the edge device 120 operates the error calculation unit 303. The error calculation unit 303 calculates the error between the processing result and the reference result, and back-propagates the calculated error.
[0057] In step S505, the important part extraction unit 122 of the edge device 120 generates an important feature map using the back-propagated error.
[0058] In step S506, the image reduction unit 121 of the edge device 120 operates the image processing unit 304. The image processing unit 304 processes the image data based on the important feature map to reduce the amount of information in the image data, and generates reduced image data.
[0059] In step S507, the image reduction unit 121 of the edge device 120 again operates the front-stage unit 301. The front-stage unit 301 receives the reduced image data and extracts a reduced feature map.
[0060] In step S508, the compression unit 123 of the edge device 120 operates the quantization unit 311 and / or the encoding unit 312. The quantization unit 311 and / or the encoding unit 312 performs quantization and / or encoding processing on the reduced feature map to generate a compressed feature map.
[0061] In step S509, the compression unit 123 of the edge device 120 transmits the compressed feature map to the server apparatus 130.
[0062] In step S510, the image reducer 121 of the edge device 120 determines whether or not to end the compression process, and if it is determined to continue (No in step S510), the process returns to step S502.
[0063] On the other hand, if it is determined in step S510 that the compression process is to be ended (Yes in step S510), the compression process is ended.
[0064] As is clear from the above description, the image processing device (edge device 120) according to the first embodiment calculates the influence of each pixel of image data, which affects the processing result when the image data is input to the deep learning model 140, and generates an important feature map. Furthermore, the image processing device (edge device 120) according to the first embodiment processes the image data based on the important feature map, thereby reducing the amount of information in the image data. Furthermore, the image processing device (edge device 120) according to the first embodiment reduces the amount of information in the feature map extracted from the intermediate layer of the deep learning model by inputting the reduced image data to the deep learning model. Furthermore, the image processing device (edge device 120) according to the first embodiment compresses the reduced feature map with the reduced amount of information.
[0065] As a result, according to the first embodiment, it is possible to improve the compression rate when compressing a feature map extracted from a deep learning model.
[0066] [Second embodiment] In the above first embodiment, the back-propagated error is used to generate an important feature map. In contrast, in the second embodiment, each feature map extracted from each layer in the previous stage is used to generate an important feature map. The second embodiment will be described below, focusing on the differences from the above first embodiment.
[0067] <Functional configuration of the image reduction unit, important part extraction unit and compression unit> First, a detailed description will be given of the functional configurations of an image reduction unit, an important part extraction unit, and a compression unit of an edge device 120, which is an example of an image processing device according to the second embodiment. Fig. 6 is a second diagram showing an example of the functional configurations of an image reduction unit, an important part extraction unit, and a compression unit of the edge device.
[0068] As shown in FIG. 6, an image reduction unit 600 is another example of the reduction unit, and includes a front-stage unit 601 and an image processing unit 304 .
[0069] The front-stage unit 601 has each layer from the input layer to the intermediate layer of the deep learning model 140. When image data is input to the front-stage unit 601, the front-stage unit 601 notifies the important part extraction unit 610 of the feature maps extracted from each layer (for example, feature map 1 extracted from the first layer, feature map 2 extracted from the second layer, ...).
[0070] Furthermore, when the reduced image data is input to the front-stage unit 601, the front-stage unit 601 notifies the compression unit 123 of the reduced feature map extracted from the hidden layer located at the backmost position in the front-stage unit 601.
[0071] The image processing unit 304 reduces the amount of information in the image data and generates reduced image data by processing the image data using the important feature map notified by the important part extraction unit 610. Specifically, the image processing unit 304 processes the image data according to the attention level of each pixel in the important feature map notified by the important part extraction unit 610, thereby reducing the amount of information in the image data and generating reduced image data.
[0072] Furthermore, the image processing unit 304 notifies the generated reduced image data to the front-stage unit 601. As described above, the front-stage unit 601 that has been notified of the reduced image data extracts a reduced feature map from the intermediate layer and notifies the compression unit 123.
[0073] The important part extraction unit 610 is another example of a calculation unit, and generates an important feature map by performing weighted addition of the feature maps of each layer notified by the previous stage unit 601. Note that in the second embodiment, the important feature map indicates the degree of attention to which pixels are focused when each layer of the previous stage unit 601 processes image data. The important part extraction unit 610 notifies the image processing unit 304 of the generated important feature map.
[0074] 6 is the same as the compression unit 123 shown in FIG. 3, and therefore will not be described here.
[0075] <Specific examples of the image reduction unit and important part extraction unit> Next, a specific example of processing by the image reduction unit 600 and important part extraction unit 610 of the edge device 120 will be described. Fig. 7 is a second diagram showing a specific example of processing by the image reduction unit and important part extraction unit. As shown in Fig. 7, in the image reduction unit 600, when image data 410 is input, the front-stage unit 601 operates and a feature map is extracted from each layer. The example in Fig. 7 shows a state in which the front-stage unit 601 has an input layer, a first layer, and a second layer, and a feature map 1 is extracted from the first layer and a feature map 2 is extracted from the second layer.
[0076] Next, the important part extraction unit 610 operates to perform weighted addition of each feature map extracted by the preceding unit 601 to generate an important feature map 710. In the example of Fig. 7, in the important feature map 710, pixels with a high degree of attention are shown in white, and pixels with a low degree of attention are shown in black.
[0077] Next, in the image reduction unit 121, the image processing unit 304 operates to invalidate pixels in the image data 410 whose attention level is equal to or lower than a predetermined threshold in the important feature map 710, thereby generating reduced image data 720.
[0078] Next, the image reduction unit 600 inputs the reduced image data 720 to the front-stage unit 601 to operate the front-stage unit 601 again, and extracts a feature map from the hidden layer (the second layer in the example of FIG. 7) located at the back end in the front-stage unit 601. Furthermore, the image reduction unit 600 notifies the compression unit 123 of the extracted feature map as a reduced feature map.
[0079] <Flow of compression processing by edge devices> Next, a flow of compression processing by the edge device 120 will be described. Fig. 7 is a second flowchart showing the flow of compression processing by the edge device. The difference from the first flowchart described using Fig. 5 is steps S801 and S802.
[0080] In step S801, the image reduction unit 600 of the edge device 120 operates the front-stage unit 601. The front-stage unit 601 receives image data and extracts a feature map from each layer.
[0081] In step S802, the important part extraction unit 610 of the edge device 120 generates an important feature map by performing weighted addition of each feature map extracted from each layer of the front-stage unit 601.
[0082] As is clear from the above description, the image processing device (edge device 120) according to the second embodiment calculates the attention level of each pixel of image data that each layer focuses on when the image data is input to the deep learning model 140, and generates an important feature map. Furthermore, the image processing device (edge device 120) according to the second embodiment reduces the amount of information in the image data by processing the image data based on the important feature map. Furthermore, the image processing device (edge device 120) according to the second embodiment reduces the amount of information in the feature map extracted from the intermediate layer of the deep learning model by inputting the reduced image data to the deep learning model. Furthermore, the image processing device (edge device 120) according to the second embodiment compresses the reduced feature map with the reduced amount of information.
[0083] As a result, according to the second embodiment, it is possible to improve the compression rate when compressing a feature map extracted from a deep learning model.
[0084] [Third embodiment] In the above first embodiment, a case has been described in which the amount of information in the image data is reduced by processing the image data based on an important feature map, and the reduced image data is input to a deep learning model, thereby reducing the amount of information in the feature map extracted from the intermediate layer of the deep learning model.
[0085] In contrast, in the third embodiment, a case will be described in which the amount of information of a feature map extracted from an intermediate layer of a deep learning model is directly reduced based on an important feature map. The third embodiment will be described below, focusing on the differences from the first embodiment.
[0086] <Functional configuration of the image reduction unit, important part extraction unit and compression unit> First, detailed descriptions will be given of the functional configurations of the image reduction unit, the important part extraction unit, and the compression unit of the edge device 120, which is an example of an image processing device according to the third embodiment. Fig. 9 is a third diagram showing an example of the functional configurations of the image reduction unit, the important part extraction unit, and the compression unit of the edge device.
[0087] As shown in FIG. 9, the image reduction unit 900 is another example of the reduction unit, and includes a front-stage unit 901 , a rear-stage unit 302 , an error calculation unit 303 , and a feature map processing unit 902 .
[0088] The front-stage unit 901 has each layer of the deep learning model 140, from the input layer to the intermediate layer from which a feature map is extracted. When image data is input to the front-stage unit 901, the front-stage unit 901 extracts a feature map from the intermediate layer and notifies the rear-stage unit 302 and the feature map processing unit 902.
[0089] The rear stage unit 302 and the error calculation unit 303 are the same as the rear stage unit 302 and the error calculation unit 303 described with reference to FIG. 3 in the first embodiment, and therefore a description thereof will be omitted here.
[0090] The feature map processing unit 902 reduces the amount of information in the feature map by processing the feature map based on the important feature map notified by the important part extraction unit 910, and generates a reduced feature map. Specifically, the feature map processing unit 902 processes the feature map based on the influence degree of each pixel in the important feature map notified by the important part extraction unit 910, thereby reducing the amount of information in the feature map and generating a reduced feature map.
[0091] The feature map processing unit 902 may process the feature map in any manner. For example, pixels whose influence level is equal to or lower than a predetermined threshold value in the important feature map may be identified, and the pixel values of the identified pixels in the feature map may be set to zero (the identified pixels may be invalidated). Alternatively, pixels whose influence level is equal to or lower than a predetermined threshold value in the important feature map may be identified, and a low-pass filter may be applied to the feature map targeting the identified pixels.
[0092] Furthermore, the feature map processing unit 902 notifies the compression unit 123 of the generated reduced feature map.
[0093] The important part extraction unit 910 is another example of a calculation unit, and generates an important feature map using the back-propagated error. As described in the first embodiment, the important feature map represents the degree of influence of each pixel of the image data on the processing result. The important part extraction unit 910 notifies the generated important feature map to the feature map processing unit 902.
[0094] 9 is the same as the compression unit 123 shown in FIG. 3, and therefore will not be described here.
[0095] <Specific examples of the image reduction unit and important part extraction unit> Next, a specific example of processing by the image reduction unit 900 and important part extraction unit 910 of the edge device 120 will be described. Fig. 9 is a third diagram showing a specific example of processing by the image reduction unit and important part extraction unit. As shown in Fig. 10, in the image reduction unit 900, when image data 410 is input, the front-stage unit 301 operates to extract a feature map, and the rear-stage unit 302 operates to output the processing result.
[0096] Next, in the image reduction unit 900, the error calculation unit 303 operates to calculate the error between the processing result and the reference result, and then back-propagates the calculated error.
[0097] The interest extractor 910 then operates to generate an interest feature map 420 using the back-propagated error.
[0098] Next, in the image reduction unit 900, the feature map processing unit 902 operates to invalidate pixels in the important feature map 420 whose influence level is equal to or less than a predetermined threshold value for the feature map extracted by the previous stage unit 901, thereby generating a reduced feature map.
[0099] <Flow of compression processing by edge devices> Next, a flow of compression processing by the edge device 120 will be described. Fig. 11 is a third flowchart showing the flow of compression processing by the edge device. The difference from the first flowchart described using Fig. 5 is step S1101.
[0100] In step S1101, the image reduction unit 900 of the edge device 120 operates the feature map processing unit 902. The feature map processing unit 902 processes the feature map based on the important feature map to reduce the amount of information in the feature map and generate a reduced feature map.
[0101] As is clear from the above description, the image processing device (edge device 120) according to the third embodiment calculates the degree of influence of each pixel of image data that affects the processing result when the image data is input to the deep learning model 140, and generates an important feature map. In addition, the image processing device (edge device 120) according to the third embodiment processes the feature map extracted from the intermediate layer of the deep learning model based on the important feature map, thereby reducing the amount of information in the feature map. Furthermore, the image processing device (edge device 120) according to the third embodiment compresses the reduced feature map with the reduced amount of information.
[0102] As a result, according to the third embodiment, it is possible to improve the compression rate when compressing a feature map extracted from a deep learning model.
[0103] [Fourth embodiment] In the above second embodiment, a case has been described in which the amount of information in the image data is reduced by processing the image data based on an important feature map, and the reduced image data is input to a deep learning model, thereby reducing the amount of information in the feature map extracted from the intermediate layer of the deep learning model.
[0104] In contrast, in the fourth embodiment, a case will be described in which the amount of information of a feature map extracted from an intermediate layer of a deep learning model is directly reduced based on an important feature map. The fourth embodiment will be described below, focusing on the differences from the second embodiment.
[0105] <Functional configuration of the image reduction unit, important part extraction unit and compression unit> First, detailed functional configurations of an image reduction unit, an important part extraction unit, and a compression unit of an edge device 120, which is an example of an image processing device according to the fourth embodiment, will be described. Fig. 12 is a fourth diagram showing an example of the functional configurations of an image reduction unit, an important part extraction unit, and a compression unit of the edge device.
[0106] As shown in FIG. 12, an image reduction unit 1200 is another example of the reduction unit, and includes a front-stage unit 601 and a feature map processing unit 1201 .
[0107] The front stage 601 is the same as the front stage 601 described in the second embodiment with reference to FIG. 6, and therefore a description thereof will be omitted here.
[0108] The feature map processing unit 1201 processes the feature map using the important feature map notified by the important part extraction unit 1210 to reduce the amount of information in the feature map and generate a reduced feature map. Specifically, the feature map processing unit 1201 processes the feature map according to the attention level of each pixel in the important feature map notified by the important part extraction unit 1210 to reduce the amount of information in the feature map, and notifies the compression unit 123 of the reduced feature map.
[0109] The important part extraction unit 1210 is another example of a calculation unit, and generates an important feature map by performing weighted addition of the feature maps of each layer notified by the front-stage unit 601. As described in the above second embodiment, the important feature map indicates the degree of attention given to which pixels when each layer of the front-stage unit 601 processes image data. The important part extraction unit 1210 notifies the feature map processing unit 1201 of the generated important feature map.
[0110] 12 is the same as the compression unit 123 shown in FIG. 3, and therefore will not be described here.
[0111] <Specific examples of the image reduction unit and important part extraction unit> Next, a specific example of processing by the image reduction unit 1200 and important part extraction unit 1210 of the edge device 120 will be described. Fig. 13 is a fourth diagram showing a specific example of processing by the image reduction unit and important part extraction unit. As shown in Fig. 13, in the image reduction unit 1200, when image data 410 is input, the front-stage unit 601 operates and a feature map is extracted from each layer. The example in Fig. 13 shows a state in which the front-stage unit 601 has an input layer, a first layer, and a second layer, and a feature map 1 is extracted from the first layer and a feature map 2 is extracted from the second layer.
[0112] Next, the important part extraction unit 1210 operates to generate an important feature map 710 by performing weighted addition on each feature map extracted from the previous stage unit 601 .
[0113] Next, in the image reduction unit 1200, the feature map processing unit 1201 operates. The feature map processing unit 1201 acquires a feature map extracted from the front-stage unit 601 (a feature map extracted from the intermediate layer (the second layer in the example of FIG. 13) located at the backmost position in the front-stage unit 601). In addition, the feature map processing unit 1201 generates a reduced feature map by invalidating pixels in the important feature map 710, the pixels having an attention level equal to or lower than a predetermined threshold, from the acquired feature map.
[0114] <Flow of compression processing by edge devices> Next, a flow of compression processing by the edge device 120 will be described. Fig. 14 is a fourth flowchart showing the flow of compression processing by the edge device. The difference from the second flowchart described using Fig. 8 is step S1401.
[0115] In step S1401, the image reduction unit 1200 of the edge device 120 operates the feature map processing unit 1201. The feature map processing unit 1201 processes the feature map based on the important feature map to reduce the amount of information in the feature map and generate a reduced feature map.
[0116] As is clear from the above description, the image processing device (edge device 120) according to the fourth embodiment calculates the attention level of each pixel of image data that each layer focuses on when the image data is input to the deep learning model 140, and generates an important feature map. In addition, the image processing device (edge device 120) according to the fourth embodiment processes the feature map extracted from the intermediate layer of the deep learning model based on the important feature map, thereby reducing the amount of information in the feature map. Furthermore, the image processing device (edge device 120) according to the fourth embodiment compresses the reduced feature map with the reduced amount of information.
[0117] As a result, according to the fourth embodiment, it is possible to improve the compression rate when compressing a feature map extracted from a deep learning model.
[0118] [Other embodiments] In the above first and second embodiments, the image data used to generate the important feature map and the image data processed based on the important feature map are described as being the same image data. However, the image data used to generate the important feature map and the image data processed based on the important feature map may be image data captured at different times. However, in the case of image data captured at different times, the important feature map is converted according to the time interval between the two image data, and the image data is processed based on the converted important feature map.
[0119] Similarly, in the above third and fourth embodiments, the image data used to generate the important feature map and the image data when the feature map to be processed based on the important feature map is extracted are described as being the same image data. However, the image data used to generate the important feature map and the image data when the feature map to be processed based on the important feature map is extracted may be image data captured at different times. However, in the case of image data captured at different times, the important feature map is converted according to the time interval between the two image data, and the feature map is processed based on the converted important feature map.
[0120] Although not mentioned in the above first to fourth embodiments, the image data used to generate the important feature map and the image data processed based on the important feature map may be captured at different times. Alternatively, the image data used to generate the important feature map and the image data when the feature map processed based on the important feature map is extracted may be image data captured at different times.
[0121] Moreover, each component in the image reduction units 121, 600, 900, 1200 described in the first to fourth embodiments may not be arranged at the locations exemplified in the first to fourth embodiments. Similarly, each component in the important part extraction units 122, 610, 910, 1210 described in the first to fourth embodiments may not be arranged at the locations exemplified in the first to fourth embodiments. For example, each component may be arranged in another device connected via a network. Also, each component may be arranged in a plurality of devices.
[0122] The true meaning of this disclosure is that when the deep learning model 140 performs image analysis processing, Extracting the importance of each pixel for extracting the target object from information at any point of the deep learning model 140; Based on the extracted information, reducing the amount of information at any point in the processing of the deep learning model 140 including the image data (at a point where it is effective to reduce the amount of intermediate information); The information extraction method that satisfies the purpose may be a method other than the extraction methods shown in the above embodiments.
[0123] Furthermore, as exemplified in each of the above embodiments, information extraction may be performed in a portion necessary for information extraction, such as a front-stage portion or a rear-stage portion of the deep learning model 140. The portion necessary for information extraction may be the portion exemplified in each of the above embodiments, or a part of it, or another portion. In other words, it is sufficient if the purpose of the above information extraction method is satisfied.
[0124] In addition, when performing the extended selective back-propagation described in the first embodiment, an error at any point of the deep learning model 140 may be used. For example, in the first embodiment, when deriving an important feature map by the extended selective back-propagation, the latter stage may not be required.
[0125] Furthermore, the compression unit 123 shown in each of the above embodiments compresses the reduced feature map notified by the image reduction unit 121 by performing quantization and / or encoding processing, but may compress a single reduced feature map by performing quantization and / or encoding processing. Alternatively, compression may be performed by performing quantization and / or encoding processing using the correlation between multiple reduced feature maps. An example in which the correlation between multiple reduced feature maps is used is a moving image.
[0126] The present invention is not limited to the configurations shown in the above embodiments, combinations with other elements, etc. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form. [Explanation of symbols]
[0127] 100: Image processing system 110: Imaging device 120: Edge devices 121: Image reduction unit 122: Important part extraction section 123: Compression section 130: Server device 131: Decoding section 132: Image analysis unit 140: Deep Learning Models 201: Processor 202: Memory 203: Auxiliary storage device 204 :I / F device 205: Communication equipment 206: Drive device 210: Operating device 211:Display device 212: Recording media 301: Front section 302: latter part 303:Error calculation section 304: Image Processing Department 310: Compression section 311: Quantization section 312: Encoding section 410: Image data 420: Important feature map 430: Image data after reduction 600: Image reduction section 601: Front section 610: Important part extraction section 710: Important feature map 720: Image data after reduction 900: Image reduction section 901: Front section 902: Feature map processing section 910: Important part extraction section 1200: Image reduction section 1201: Feature map processing section 1210: Important part extraction section
Claims
1. A calculation unit that calculates an influence degree of each pixel that affects a processing result by the deep learning model in the image data by inputting image data to an input layer of the deep learning model and back-propagating an error based on a processing result output from an output layer and a reference result; a processing unit that processes pixels of the image data before being input to the input layer, the pixels having the degree of influence equal to or less than a predetermined threshold value, thereby generating reduced image data that has been processed to reduce an amount of information of the image data; a compression unit that inputs the generated reduced image data to the input layer of the deep learning model and compresses intermediate information extracted from an intermediate layer; An image processing device comprising:
2. A calculation unit that calculates an influence degree of each pixel that affects a processing result by the deep learning model in the image data by inputting image data to an input layer of the deep learning model and back-propagating an error based on a processing result output from an output layer and a reference result; a reduction unit that generates reduced intermediate information by processing the intermediate information extracted from a predetermined layer of the intermediate layer, the image data being input to an input layer of the deep learning model, and processing pixels of the intermediate information having the degree of influence equal to or less than a predetermined threshold value, among the pixels of the intermediate information extracted from a predetermined layer of the intermediate layer, to reduce the amount of information of the intermediate information; a compression unit that compresses the generated reduced intermediate information; An image processing device comprising:
3. A calculation unit that calculates, for each pixel of the image data, an attention level indicating which pixel each layer will pay attention to when processing the image data, based on intermediate information extracted from each layer of the intermediate layer when image data is input to an input layer of the deep learning model; a processing unit that processes pixels of the image data before being input to the input layer, the pixels having the degree of attention equal to or less than a predetermined threshold, thereby generating reduced image data that has been processed to reduce the amount of information of the image data; a compression unit that inputs the generated reduced image data to the input layer of the deep learning model and compresses intermediate information extracted from a predetermined layer of the intermediate layer; An image processing device comprising:
4. A calculation unit that calculates, for each pixel of the image data, an attention level indicating which pixel each layer will pay attention to when processing the image data, based on intermediate information extracted from each layer of the intermediate layer when image data is input to an input layer of the deep learning model; a reduction unit that generates reduced intermediate information by processing the intermediate information extracted from a predetermined layer of the intermediate layer, the image data being input to an input layer of the deep learning model, and processing pixels of the intermediate information having a degree of attention equal to or less than a predetermined threshold value, among the pixels of the intermediate information extracted from a predetermined layer of the intermediate layer, to reduce the amount of information of the intermediate information; a compression unit that compresses the generated reduced intermediate information; An image processing device comprising:
5. An image processing device as described in any one of claims 1 to 4, wherein performing a process to process the pixel means invalidating the pixel.
6. Image data is input to an input layer of a deep learning model, and an error based on the processing result output from the output layer and a reference result is back-propagated to calculate the influence of each pixel of the image data that affects the processing result by the deep learning model; generating reduced image data that has been processed to reduce an amount of information of the image data by performing a process of processing pixels of the image data before being input to the input layer, the pixels having the degree of influence equal to or less than a predetermined threshold value; The generated reduced image data is input to the input layer of the deep learning model, and intermediate information extracted from the intermediate layer is compressed. An image processing method in which the processing is performed by a computer.
7. Image data is input to an input layer of a deep learning model, and an error based on the processing result output from the output layer and a reference result is back-propagated to calculate the influence of each pixel of the image data that affects the processing result by the deep learning model; The image data is input to an input layer of the deep learning model, and among the pixels of intermediate information extracted from a predetermined layer of the intermediate layer, a process is performed to process the pixels of the intermediate information whose influence degree is equal to or less than a predetermined threshold, thereby generating reduced intermediate information that has been processed to reduce the amount of information of the intermediate information; compressing the generated reduced intermediate information; An image processing method in which the processing is performed by a computer.
8. When image data is input to an input layer of a deep learning model, an attention level indicating which pixel each layer will pay attention to when processing the image data is calculated for each pixel of the image data based on intermediate information extracted from each layer of the intermediate layer; generating reduced image data by processing the pixels of the image data before being input to the input layer, the pixels having the degree of attention equal to or less than a predetermined threshold value; and The generated reduced image data is input to the input layer of the deep learning model, and intermediate information extracted from a predetermined layer of the intermediate layer is compressed. An image processing method in which the processing is performed by a computer.
9. When image data is input to an input layer of a deep learning model, an attention level indicating which pixel each layer will pay attention to when processing the image data is calculated for each pixel of the image data based on intermediate information extracted from each layer of the intermediate layer; The image data is input to an input layer of the deep learning model, and among the pixels of intermediate information extracted from a predetermined layer of the intermediate layer, a process is performed to process pixels of the intermediate information whose attention level is equal to or less than a predetermined threshold, thereby generating reduced intermediate information that has been processed to reduce the amount of information of the intermediate information; compressing the generated reduced intermediate information; An image processing method in which the processing is performed by a computer.
10. Image data is input to an input layer of a deep learning model, and an error based on the processing result output from the output layer and a reference result is back-propagated to calculate the influence of each pixel of the image data that affects the processing result by the deep learning model; generating reduced image data that has been processed to reduce an amount of information of the image data by performing a process of processing pixels of the image data before being input to the input layer, the pixels having the degree of influence equal to or less than a predetermined threshold value; The generated reduced image data is input to the input layer of the deep learning model, and intermediate information extracted from the intermediate layer is compressed. An image processing program that causes a computer to carry out the processing.
11. Image data is input to an input layer of a deep learning model, and an error based on the processing result output from the output layer and a reference result is back-propagated to calculate the influence of each pixel of the image data that affects the processing result by the deep learning model; The image data is input to an input layer of the deep learning model, and among the pixels of intermediate information extracted from a predetermined layer of the intermediate layer, a process is performed to process the pixels of the intermediate information whose influence degree is equal to or less than a predetermined threshold, thereby generating reduced intermediate information that has been processed to reduce the amount of information of the intermediate information; compressing the generated reduced intermediate information; An image processing program that causes a computer to carry out the processing.
12. When image data is input to an input layer of a deep learning model, an attention level indicating which pixel each layer will pay attention to when processing the image data is calculated for each pixel of the image data based on intermediate information extracted from each layer of the intermediate layer; generating reduced image data by performing a process of processing pixels of the image data before being input to the input layer, the pixels having the degree of attention equal to or less than a predetermined threshold, the reduced image data being processed to reduce the amount of information of the image data; The generated reduced image data is input to the input layer of the deep learning model, and intermediate information extracted from a predetermined layer of the intermediate layer is compressed. An image processing program that causes a computer to carry out the processing.
13. When image data is input to an input layer of a deep learning model, an attention level indicating which pixel each layer will pay attention to when processing the image data is calculated for each pixel of the image data based on intermediate information extracted from each layer of the intermediate layer; The image data is input to an input layer of the deep learning model, and among the pixels of intermediate information extracted from a predetermined layer of the intermediate layer, a process is performed to process pixels of the intermediate information whose attention level is equal to or less than a predetermined threshold, thereby generating reduced intermediate information that has been processed to reduce the amount of information of the intermediate information; compressing the generated reduced intermediate information; An image processing program that causes a computer to carry out the processing.
Citation Information
Patent Citations
Control device, control program and control method
JP2018097662A
Learning model creation device, learning model creation method, and learning model creation program
JP2018195231A
Error calculator and its program
JP2019029938A
Information processing unit, information processing method and program
JP2019036899A
Image processing device, learning device, image processing method, learning method, and program
JP2020092329A
