Image processing device, image processing method, and image processing program

Abstract

This image processing device comprises: an acquisition unit that acquires a moving image to be processed; a difference determination unit that determines a difference area from a past frame for a frame which is one frame among a plurality of frames and other than a key frame; a block setting unit that sets, for the frame which is one among the plurality of frames and other than the key frame, an update block, which is one among a plurality of blocks obtained by dividing an output feature map and includes an update area corresponding to the difference area to each of a plurality of layers which executes convolution processing of a neural network; and a processing unit which uses, for the key frame of the plurality of frames, the neural network to process the frame and stores the output feature map for each layer and which performs, for each frame which is of the plurality of frames and is other than the key frame, the processing using the neural network for the update block and overwrites the stored output feature map. The block setting unit sets, for each layer which executes the convolution processing, the difference area so that the difference area expands toward the periphery compared to the previous layer in accordance with a parameter of the convolution processing and sets the update block including the update area corresponding to the difference area.

Description

[Technical Field]

[0001] The technology of the present disclosure relates to an image processing device, an image processing method, and an image processing program. [Background technology]

[0002] Inference processes such as object detection, pose estimation, and segmentation using a convolutional neural network (CNN) basically target a single image, and applying the process to each frame of video requires a computational load proportional to the number of frames.

[0003] On the other hand, inference processing for video data, such as video scene understanding and object tracking, uses the inference processing for image data described above, but limits the number of frames to which it can be applied, and uses other information that can be derived with less computation to reduce the amount of computation. However, for video with rapid frame-to-frame changes, it is desirable to perform inference processing on a larger number of frame images.

[0004] One method to reduce the amount of computation in this case is to determine changes between frames for each subregion of the video and perform CNN inference processing only on the subregions where changes exist. However, this method has the problem of making inference across subregions difficult.

[0005] Also, Non-Patent Document 1 proposes a method for reducing the amount of calculation by taking the inter-frame difference for each pixel in each layer and performing a convolution calculation. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Z, Yuan, et al. Seikadai, “A 65nm 24.7μJ / Frame 12.3mW Activation-Similarity-Aware Convolutional Neural Network Video Processor Using Hybrid Precision, Inter-Frame Data Reuse and Mixed-Bit-Width Difference-Frame Data Codec,” ISSCC 2020 Summary of the Invention [Problem to be solved by the invention]

[0007] The technique described in Non-Patent Document 1 above has a problem in that it requires a complex calculation and control mechanism.

[0008] The disclosed technology has been made in consideration of the above points, and aims to provide an image processing device, an image processing method, and an image processing program that have a simple configuration and can reduce the amount of calculation required for processing using a neural network, including convolution processing. [Means for solving the problem]

[0009] a difference determination unit that determines a difference region from a previous frame for frames other than key frames among the plurality of frames; a block setting unit that sets, for each of a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map for frames other than key frames among the plurality of frames; and a processing unit that processes key frames among the plurality of frames using the neural network to save an output feature map for each layer, and processes frames other than key frames among the plurality of frames using the neural network to overwrite the output feature map saved for the update block, wherein the block setting unit sets, for each layer that performs convolution processing, the difference region so as to expand the difference region from that of a previous layer according to parameters of the convolution processing, and sets an update block including the update region corresponding to the difference region.

[0010] A second aspect of the present disclosure is an image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, the image processing device including an acquisition unit that acquires the moving image to be processed, a difference determination unit that determines a difference region from a previous frame for frames other than key frames among the plurality of frames, and a unit that sets, for each of a predetermined storage layer among a plurality of layers that perform convolution processing of the neural network for the frames other than key frames among the plurality of frames, an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map, and sets, for each of the plurality of layers, a processing target block including a processing target region corresponding to the difference region. and a processing unit that processes key frames from among the plurality of frames using the neural network to store an output feature map for each of the storage layers, and that processes the blocks to be processed for each of the plurality of layers for frames other than key frames from among the plurality of frames using the neural network and overwrites the update block of the stored output feature map for each of the storage layers, wherein the block setting unit sets the difference region for each of the storage layers in accordance with parameters of the convolution process so that the difference region is expanded outward from the previous layer, and sets an update block including an update region corresponding to the difference region.

[0011] A third aspect of the present disclosure is an image processing method in an image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, the method including: an acquisition unit acquiring a moving image to be processed; a difference determination unit determining a difference region from a previous frame for frames other than key frames among the plurality of frames; a block setting unit setting, for each of a plurality of layers that perform convolution processing of the neural network for frames other than key frames among the plurality of frames, an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map; a processing unit processing key frames among the plurality of frames using the neural network and saving output feature maps of each layer; and processing frames other than key frames among the plurality of frames using the neural network and overwriting the output feature map saved for the update block; the setting by the block setting unit including: for each layer that performs convolution processing, setting the difference region so that the difference region is expanded outward from a previous layer according to parameters of the convolution processing, and setting an update block including an update region corresponding to the difference region.

[0012] A fourth aspect of the present disclosure is an image processing method in an image processing device including a neural network that includes convolution processing for a moving image including a plurality of frames, wherein an acquisition unit acquires a moving image to be processed, a difference determination unit determines a difference region from a previous frame for a frame other than a key frame among the plurality of frames, and a block setting unit sets, for each of a predetermined storage layer among a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map for the frame other than the key frame among the plurality of frames, and sets, for each of the plurality of layers, a processing target region including a processing target region corresponding to the difference region. The method includes setting a block, and a processing unit processes a key frame from among the plurality of frames using the neural network to save an output feature map for each of the storage layers, and for frames other than the key frames from among the plurality of frames, processing the block to be processed for each of the plurality of layers using the neural network, and overwriting the update block of the saved output feature map for each of the storage layers, wherein the setting by the block setting unit includes setting the differential region for each of the storage layers in accordance with parameters of the convolution processing so that the differential region is expanded outward from the previous layer, and setting an update block including an update region corresponding to the differential region.

[0013] A fifth aspect of the present disclosure is an image processing program for causing a computer to function as the image processing device of the first or second aspect. [Effects of the Invention]

[0014] According to the disclosed technology, it is possible to reduce the amount of calculation required for processing using a neural network including convolution processing with a simple configuration. [Brief explanation of the drawings]

[0015] [Figure 1]FIG. 1 is a schematic block diagram of an example of a computer that functions as an image processing apparatus according to the first and second embodiments. [Figure 2] FIG. 1 is a block diagram illustrating a functional configuration of an image processing apparatus according to a first embodiment and a second embodiment. [Figure 3] FIG. 2 is a block diagram showing the functional configuration of a learning unit of the image processing device according to the first and second embodiments. [Figure 4] FIG. 2 is a block diagram showing the functional configuration of an inference unit of the image processing device according to the first and second embodiments. [Figure 5] FIG. 10 is an image diagram of a differential region set for each layer. [Figure 6] FIG. 2 is a diagram illustrating a differential area, an update area, and an update block. [Figure 7] FIG. 10 is an image diagram of a differential region set for each layer. [Figure 8] 4 is a flowchart showing the flow of a learning process according to the first and second embodiments. [Figure 9] 1 is a flowchart showing the flow of image processing according to the first and second embodiments. [Figure 10] 4 is a flowchart showing the flow of convolution processing in image processing according to the first embodiment. [Figure 11] 10 is a flowchart illustrating a process flow for setting an update block according to the first embodiment. [Figure 12] FIG. 10 is a conceptual diagram of a differential area set for each layer and an update block set for each storage layer. [Figure 13] 10A and 10B are diagrams for explaining a differential region, an update region, an update block, a processing target region, and a processing target block. [Figure 14] 10 is a flowchart showing the flow of convolution processing in image processing according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of the disclosed technology will be described below with reference to the drawings. Note that the same or equivalent components and parts in each drawing are given the same reference numerals. Also, the dimensional proportions in the drawings are exaggerated for the convenience of explanation and may differ from the actual proportions.

[0017] <Summary of the Disclosed Technology> The disclosed technology reduces the amount of computation required for CNN inference processing for each frame of video by following the steps below.

[0018] First, the presence or absence of differences between the input image of the previous frame and the current frame is determined in blocks of a few pixels by a few pixels, and for blocks containing difference regions, normal CNN processing is performed for one layer, with the processing result of the first layer being used. For other blocks that do not contain difference regions, the processing result of the first layer of the previous frame is read and used as the processing result of the first layer. In subsequent layers, the difference region is expanded to the range affected by the difference region of the first layer, and normal CNN processing is performed on blocks containing the expanded difference region. For blocks that do not contain difference regions, CNN processing is skipped, and the processing result of the same layer of the previous frame is read and used as the processing result of that layer. In this case, the difference region is updated based on criteria such as expanding it by one pixel to the surrounding area in layers using a 3x3 pixel kernel, and not expanding it in layers using a 1x1 pixel kernel. In addition, efficient implementation is possible by determining whether to perform CNN processing or skip it on a pre-determined block basis.

[0019] The following methods can be used in combination with the above.

[0020] The first method limits the storage of output feature maps of past frames to one out of several layers that perform convolutional processing. This reduces data transfer bandwidth and memory capacity. Outside of the storage layers, feature maps outside the differential region do not exist, and convolution processing affects invalid data from the surrounding areas. Therefore, normal CNN processing is performed over a correspondingly wider range. Furthermore, processing results affected by invalid data are discarded, and only the processing results for unaffected regions are overwritten over the results of past frames. Specifically, for each storage layer, the pixel width N by which the influence of the differential region expands up to the next storage layer is calculated. The block containing at least a portion of the update region, where the differential region is expanded by N pixels, is designated as the update block, and the feature map of the past frame is overwritten only for the update block. Furthermore, the block containing at least a portion of the target processing region, where the update region is expanded by N pixels, is designated as the target processing block, and CNN processing is performed on the target processing block.

[0021] Another method is to determine in advance the range in which the difference region of the first layer will affect the final inference result from the inference results of reduced images or past frames, and by not expanding the difference region beyond that range, CNN processing is skipped outside that range and the processing results of past frames are read. This method effectively limits the area in which CNN processing is performed, thereby reducing the amount of calculation.

[0022] [First embodiment] <Configuration of image processing device according to first embodiment> FIG. 1 is a block diagram showing the hardware configuration of an image processing apparatus 10 according to the first embodiment.

[0023] 1, the image processing device 10 includes a CPU (Central Processing Unit) 11, a ROM (Read Only Memory) 12, a RAM 13, a storage 14, an input unit 15, a display unit 16, and a communication interface (I / F) 17. Each component is connected to each other via a bus 19 so as to be able to communicate with each other.

[0024] The CPU 11 is a central processing unit that executes various programs and controls each component. That is, the CPU 11 reads programs from the ROM 12 or storage 14 and executes the programs using the RAM 13 as a work area. The CPU 11 controls the above components and performs various arithmetic processing in accordance with the programs stored in the ROM 12 or storage 14. In this embodiment, the ROM 12 or storage 14 stores a learning processing program for performing learning processing on a neural network and an image processing program for performing image processing using a neural network. The learning processing program and the image processing program may be a single program, or may be a group of programs consisting of multiple programs or modules.

[0025] The ROM 12 stores various programs and various data. The RAM 13 temporarily stores programs or data as a working area. The storage 14 is configured with an HDD (Hard Disk Drive) or SSD (Solid State Drive) and stores various programs including the operating system and various data.

[0026] The input unit 15 includes a pointing device such as a mouse and a keyboard, and is used to perform various inputs.

[0027] The input unit 15 receives, as input, training data for training the neural network. For example, the input unit 15 receives, as input, training data including a video to be processed and a processing result for the video that has been obtained in advance.

[0028] Furthermore, the input unit 15 receives as input a moving image to be processed.

[0029] The display unit 16 is, for example, a liquid crystal display, and displays various information including the processing results. The display unit 16 may be a touch panel type and function as the input unit 15.

[0030] The communication interface 17 is an interface for communicating with other devices, and uses standards such as Ethernet (registered trademark), FDDI, and Wi-Fi (registered trademark).

[0031] Next, a description will be given of the functional configuration of the image processing device 10. Fig. 2 is a block diagram showing an example of the functional configuration of the image processing device 10.

[0032] Functionally, the image processing device 10 includes a learning unit 20 and an inference unit 22, as shown in FIG.

[0033] As shown in FIG. 3, the learning unit 20 includes an acquisition unit 30, a processing unit 38, and an update unit 40.

[0034] The acquisition unit 30 acquires the input video images and processing results of the learning data.

[0035] The processing unit 38 processes each frame of the video using a neural network including convolution processing.

[0036] The update unit 40 updates the parameters of the neural network so that the result of processing the video using the neural network matches the processing result obtained in advance.

[0037] The processes of the processing unit 38 and the updating unit 40 are repeated until a predetermined iteration end condition is met, thereby training the neural network.

[0038] As shown in FIG. 4, the inference unit 22 includes an acquisition unit 50, an overall control unit 52, a difference determination unit 54, a block setting unit 56, and a processing unit 58.

[0039] The acquisition unit 50 acquires the input moving image to be processed.

[0040] The overall control unit 52 determines whether each of the multiple frames of the moving image to be processed is a key frame. Here, it is assumed that the multiple frames are designated as key frames at a predetermined cycle. Note that a frame whose differential region ratio is equal to or greater than a threshold may be determined to be a key frame.

[0041] The difference determination unit 54 determines a difference area between a frame other than a key frame among the plurality of frames and a previous frame.

[0042] For frames other than key frames, the block setting unit 56 sets, for each of multiple layers performing convolution processing of the neural network, an update block that includes at least a portion of an update region corresponding to the difference region among multiple blocks obtained by dividing the output feature map. At this time, the block setting unit 56 sets, for each layer performing convolution processing, a difference region in accordance with parameters of the convolution processing, so as to expand the difference region from the previous layer (see FIG. 5), and sets an update block that includes at least a portion of the update region corresponding to the difference region (see FIG. 6). Figure 5 shows an example in which, compared to the difference region of the initial layer, the difference region expands with increasing layer depth, expanding the range in which normal CNN processing is performed and narrowing the range in which processing skips for reading the processing results of past frames are performed. Also, Figure 6 shows an example in which four blocks (dashed rectangles) that include at least a portion of an update region (solid rectangle) expanded around the difference region (dotted rectangle) are set as update blocks.

[0043] Furthermore, it is preferable that the block setting unit 56 sets the differential region so that it does not expand beyond a pre-specified region (see FIG. 7). It is also preferable that the block setting unit 56 sets the differential region so that it does not expand in layers after the pre-specified layer. FIG. 7 shows an example in which, compared to the differential region in the first layer, the differential region expands with increasing depth, with the pre-specified region being the upper limit, and after the layer that reaches the pre-specified region, the range in which normal CNN processing is performed does not expand.

[0044] The processing unit 58 performs a normal CNN inference process for processing key frames among the multiple frames using a neural network, and saves an output feature map for each layer.

[0045] The normal CNN inference process referred to here refers to the process of inputting an input feature map at each layer from the first to the final layer, performing convolution processing, activation function processing, downsampling processing, upsampling processing, and combining / connecting with output feature maps from other layers, and outputting an output feature map. The input feature map at the first layer is assumed to be image data consisting of three channels, such as RGB, and the output feature map at the final layer is assumed to be data in which information about the inference result is stored in each channel. For convenience, the kernel size used for convolution is assumed to be either 1x1 pixel or 3x3 pixels, but this is not limited to this.

[0046] Furthermore, the processing unit 58 performs processing using a neural network on blocks including differential regions for frames other than key frames among the multiple frames, and overwrites the saved output feature map.

[0047] The results of processing the moving image using the neural network are displayed on the display unit 16.

[0048] <Action of the image processing device according to the first embodiment> Next, the operation of the image processing device 10 according to the first embodiment will be described.

[0049] 8 is a flowchart showing the flow of the learning process by the image processing device 10. The learning process is performed by the CPU 11 reading out a learning process program from the ROM 12 or the storage 14, expanding it into the RAM 13, and executing it. In addition, learning data is input to the image processing device 10.

[0050] In step S100, the CPU 11 functions as the acquisition unit 30 to acquire the input video images and processing results of the learning data.

[0051] In step S102, the CPU 11 processes the moving images of the learning data using a neural network including convolution processing as the processing unit 38.

[0052] In step S104, the CPU 11 functions as the update unit 40 to update the parameters of the neural network so that the results of processing the video images of the learning data using the neural network match the results of processing obtained in advance.

[0053] In step S106, the CPU 11 determines whether a predetermined iteration end condition is satisfied. If the iteration end condition is not satisfied, the process returns to step S102, and the processes of the processing unit 38 and the update unit 40 are repeatedly performed. This allows the neural network to learn.

[0054] 9 is a flowchart showing the flow of image processing by the image processing device 10. Image processing is performed by the CPU 11 reading out an image processing program from the ROM 12 or storage 14, expanding it into the RAM 13, and executing it. Also, a moving image to be processed is input to the image processing device 10.

[0055] In step S107, the CPU 11 functions as the acquisition unit 50 to acquire the input moving image.

[0056] In step S109, the CPU 11 processes the moving image using the neural network trained by the above-described training process, and the display unit 16 displays the results of processing the moving image using the neural network.

[0057] The above step S109 is realized by the processing routine shown in Fig. 10. Here, each frame of the moving image is set in turn as the current frame.

[0058] First, in step S110, the CPU 11, functioning as the overall control unit 52, determines whether the current frame is a key frame. If it is determined that the current frame is a key frame, the process proceeds to step S112. On the other hand, if it is determined that the current frame is not a key frame, the process proceeds to step S114.

[0059] In step S112, the CPU 11, functioning as the processing unit 58, performs normal CNN inference processing on the current frame and stores all output feature maps of each layer in the RAM 13. The processing unit 58 outputs the inference result to the display unit 16.

[0060] In step S114, the CPU 11, functioning as the difference determination unit 54, calculates the pixel difference between the image of the current frame and the cumulative update image and determines the difference region. Here, the cumulative update image is an image in which regions determined to have differences in each frame after the key frame image are replaced with the input image of that frame. In determining the difference region, the influence of noise is removed by thresholding the pixel difference values ​​of both images and comparing them with surrounding pixels, and only regions with visually significant differences are determined to be difference regions on a pixel-by-pixel basis.

[0061] In step S116, the CPU 11, as the block setting unit 56, sets a differential region for the layer in question so as to expand the differential region outward from the previous layer in accordance with the parameters of the convolution process, sets an update region by expanding the differential region by one pixel width or several pixels width as a margin, determines whether or not each block of several pixels square contains at least a part of the update region, sets the block that contains at least a part of the update region as an update block, and stores this in the RAM 13 as update block information.

[0062] In steps S118 and S120, the CPU 11, functioning as the processing unit 58, performs processing for one layer based on the update block information read from the RAM 13. Specifically, in step S118, the CPU 11, functioning as the processing unit 58, determines whether the block in question is an update block. If the block in question is not an update block, the process proceeds to step S124 without performing any processing. As a result, for the block in question, the output feature map of the previous frame is used as the output feature map of the current frame.

[0063] On the other hand, if the block is an update block, the process proceeds to step S120. In step S120, the CPU 11, as the processing unit 58, reads the input feature map for the update block, including the surrounding pixels required for the convolution process, and performs the convolution process and activation function process on the result, as in the normal CNN inference process.

[0064] In step S122, the CPU 11, functioning as the processing unit 58, overwrites the output feature map of the previous frame in the RAM 13 at the same layer and position with the output feature map for the update block.

[0065] In step S124, the CPU 11 determines whether or not the processes of steps S118 to S122 have been completed for all blocks. If there is a block for which the processes of steps S118 to S122 have not been performed, the CPU 11 returns to step S118 and performs the processes of steps S118 to S122 for that block.

[0066] In step S126, CPU 11 determines whether or not the processing of steps S116 to S124 has been completed for all layers. If the processing of steps S116 to S124 has not been completed for all layers, the process returns to step S116 to process the next layer. On the other hand, if the processing of steps S116 to S124 has been completed for all layers, the process proceeds to step S128.

[0067] In step S128, CPU 11 determines whether or not the processing of steps S110 to S126 has been completed for all frames. If the processing of steps S110 to S126 has not been completed for all frames, the process returns to step S110, and the next frame is processed as the current frame. On the other hand, if the processing of steps S110 to S126 has been completed for all frames, the process routine ends.

[0068] The above step S116 is realized by the processing routine shown in FIG.

[0069] First, in step S130, the CPU 11 functions as the block setting unit 56 to obtain information indicating the result of the differential region determination in step S114.

[0070] In step S132, the CPU 11, functioning as the block setting unit 56, determines whether the kernel size of the previous layer is 1×1. If the kernel size of the previous layer is 1×1, the process proceeds to step S140. On the other hand, if the kernel size of the previous layer is not 1×1 but 3×3, the process proceeds to step S134.

[0071] In step S134, the CPU 11, functioning as the block setting unit 56, determines whether the layer in question is after a previously specified layer. If the layer in question is after the previously specified layer, the process proceeds to step S140. On the other hand, if the layer in question is before the previously specified layer, the process proceeds to step S136.

[0072] In step S136, the CPU 11, functioning as the block setting unit 56, determines whether or not the differential region of the current layer will exceed the pre-specified region when expanded from the previous layer in accordance with the parameters of the convolution process. If it is determined that the differential region will exceed the pre-specified region when expanded from the previous layer, the process proceeds to step S140. On the other hand, if it is determined that the differential region will not exceed the pre-specified region when expanded from the previous layer in accordance with the parameters of the convolution process, the process proceeds to step S138.

[0073] Based on the determination in step S134, the differential region is not expanded to the periphery in layers after the pre-specified layer. Furthermore, based on the determination in step S136, the differential region is not expanded beyond the pre-specified region even in layers before the specified layer. This prevents the update block from spreading over the entire feature map, reducing the amount of calculation.

[0074] This is effective when it is known in advance that local changes within an image will not affect the inference results over a wider area, for example, when it is known in advance that only relatively small objects are captured in the image, or when it can be determined in advance from information such as other inference results that there will be no change in the inference results outside the specified area.

[0075] In step S138, the CPU 11 functions as the block setting unit 56 to expand the differential region by one pixel to the periphery. This is because the influence of the differential region expands to the periphery when the kernel size of the convolution processing in the immediately preceding layer is larger than 1×1 pixels.

[0076] In step S140, the CPU 11, functioning as the block setting unit 56, determines whether or not the previous layer involves 1 / 2 downsampling. If the previous layer does not involve 1 / 2 downsampling, the process proceeds to step S144. On the other hand, if the previous layer involves 1 / 2 downsampling, the process proceeds to step S142.

[0077] In step S142, the CPU 11 functions as the block setting unit 56 and downsamples the differential region to 1 / 2 in pixel units, and if there is a differential region of one or more pixels in the 2×2 pixels, it is set as the differential region.

[0078] In step S144, the CPU 11, functioning as the block setting unit 56, determines whether or not the previous layer involves upsampling. If the previous layer does not involve upsampling, the process proceeds to step S148. On the other hand, if the previous layer involves upsampling, the process proceeds to step S146.

[0079] In step S146, the CPU 11, functioning as the block setting unit 56, upsamples the difference region in units of pixels.

[0080] In step S148, the CPU 11, as the block setting unit 56, sets an update area by expanding the differential area by one pixel width or several pixels width as a margin, determines whether each block of several pixels square includes at least a part of the update area, sets the block including at least a part of the update area as an update block, and stores this as update block information in the RAM 13. The updated differential area information is also output to the RAM 13. The differential area information may be information in pixel units, or may be a combination of block unit information and information in pixel width expanded from that.

[0081] As described above, the image processing device according to the first embodiment determines the difference region from a previous frame for frames other than key frames, sets an update block including an update region corresponding to the difference region for each of multiple layers that perform convolution processing, and overwrites the output feature map stored for the update block. Furthermore, for each layer that performs convolution processing, the difference region is set according to parameters of the convolution processing so that the difference region is expanded from the previous layer to the surrounding area, and sets an update block including the update region corresponding to the difference region. This makes it possible to reduce the amount of calculation required for processing using a neural network that includes convolution processing with a simple configuration.

[0082] In the above embodiment, the difference determination is performed on the cumulatively updated image rather than on the immediately preceding frame. This is to avoid a decrease in accuracy due to the accumulation of minute differences when multiple consecutive frames in the same area are determined to have no difference. Therefore, the difference determination may be performed on the immediately preceding frame, and key frames may be inserted more frequently instead. Furthermore, the difference determination may be performed on a reduced image of the input image to reduce the amount of calculation and the influence of noise.

[0083] [Second embodiment] Next, a second embodiment will be described. Note that the image processing device according to the second embodiment has the same configuration as that of the first embodiment, so the same reference numerals are used and the description will be omitted.

[0084] In the first embodiment, output feature maps of all layers of past frames are stored in RAM and overwritten and updated. However, in the second embodiment, the number of layers for which output feature maps are stored in RAM is limited in order to reduce memory capacity and bandwidth, which is different from the first embodiment.

[0085] <Configuration of image processing device according to second embodiment> The block setting unit 56 of the image processing device 10 according to the second embodiment sets, for frames other than key frames among the plurality of frames, an update block including an update region corresponding to a differential region among the plurality of blocks obtained by dividing the output feature map for each of predetermined storage layers among the plurality of layers performing convolution processing of a neural network. At this time, the block setting unit 56 sets a differential region for each predetermined storage layer in accordance with parameters of the convolution processing so as to expand the differential region from the previous layer toward the periphery ( FIG. 12 ), and sets an update block including the update region corresponding to the differential region ( FIG. 13 ). Furthermore, the block setting unit 56 sets a processing target block including a processing target region corresponding to the differential region for each layer performing the convolution processing ( FIG. 13 ).

[0086] Figure 12 shows an example in which the difference region expands as the layer gets deeper compared to the difference region in the first layer, expanding the range in which normal CNN processing is performed and reducing the range in which processing skips for reading the processing results of past frames are performed. It also shows an example in which an update block is set for each storage layer, where a feature map is written and overwritten. It also shows an example in which the difference region is expanded to take into account the influence of invalid data, and a processing target block, which is the part where the feature map is read, is set.

[0087] 13 shows an example in which four blocks (dashed rectangles) that include at least a portion of an update area (innermost solid rectangle) that is an expansion of the differential area (dotted rectangle) are set as update blocks, and six blocks (dashed rectangles) that include at least a portion of a processing target area (outer solid rectangle) that is an expansion of the differential area further are set as processing target blocks.

[0088] It is also preferable that the block setting unit 56 sets the differential region so that it does not expand beyond a region designated in advance.It is also preferable that the block setting unit 56 sets the differential region in a layer after the layer designated in advance so that it does not expand.

[0089] The processing unit 58 performs a normal CNN inference process for processing key frames among the multiple frames using a neural network, and stores an output feature map of each storage layer.

[0090] Furthermore, for frames other than key frames among the multiple frames, the processing unit 58 performs processing using a neural network on the block to be processed for each of the multiple layers that perform convolution processing, and overwrites the updated block of the output feature map stored in each storage layer.

[0091] <Action of the image processing device according to the second embodiment> Next, the operation of the image processing device 10 according to the second embodiment will be described. Note that the same processes as those in the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted.

[0092] The image processing device 10 performs the learning process shown in Fig. 8. The image processing device 10 also performs the image processing shown in Fig. 9. At this time, step S109 is realized by the processing routine shown in Fig. 14. Here, each frame of the moving image is set in turn as the current frame.

[0093] First, in step S110, the CPU 11, functioning as the overall control unit 52, determines whether the current frame is a key frame. If it is determined that the current frame is a key frame, the process proceeds to step S200. On the other hand, if it is determined that the current frame is not a key frame, the process proceeds to step S114.

[0094] In step S200, the CPU 11, functioning as the processing unit 58, performs normal CNN inference processing on the current frame and stores all output feature maps of each storage layer in the RAM 13. The inference results are output from the processing unit 58 to the display unit 16.

[0095] In step S114, the CPU 11 functions as the difference determination unit 54 to determine the difference region by calculating the pixel difference between the image of the current frame and the cumulative updated image.

[0096] In step S201, the CPU 11 determines whether the current layer is the initial layer or whether the previous layer is the storage layer. If the current layer is the initial layer or the previous layer is the storage layer, the process proceeds to step S202. On the other hand, if the current layer is not the initial layer and the previous layer is not the storage layer, the process proceeds to step S204.

[0097] In step S202, CPU 11, as block setting unit 56, sets a differential region for each layer up to the next storage layer by expanding the differential region outward from the previous layer in accordance with the parameters of the convolution process, and for the next storage layer, sets an update region by expanding the differential region by one pixel width or several pixels width as a margin, determines whether each block of several pixels square contains at least a portion of the update region, sets the block containing at least a portion of the update region as an update block, and stores this as update block information in RAM 13. Furthermore, for each layer up to the next storage layer, CPU 11 further expands the differential region to set a processing target region, determines whether each block contains at least a portion of the processing target region, and sets the block containing at least a portion of the processing target region as a processing target block, and stores this as processing block information in RAM 13.

[0098] For example, the number N of layers with a kernel size of 3x3 pixels up to the next storage layer is obtained, the difference region is expanded by N pixels in width and set as the difference region in the next storage layer, the difference region is expanded by one pixel or several pixels in width as a margin to set the update region, and a block including at least a part of the update region is set as the update block. Furthermore, for each layer up to the next storage layer, a processing target region is set by further expanding the difference region, and a block including at least a part of the processing target region is set as the processing target block.

[0099] This is because, except for the storage layer, there is no input feature map outside the update block, so invalid data is referenced, and the influence of the invalid data extends N pixels from the surrounding area, so a wider range needs to be processed.

[0100] Furthermore, if downsampling or upsampling is included between the layers up to the next storage layer, downsampling or upsampling of the differential region is performed together with expansion of the differential region to calculate the range of the update block or the block to be processed.

[0101] In step S204, the processing unit 58 determines whether the block is a processing target block. If the block is a processing target block, the process proceeds to step S206. On the other hand, if the block is not a processing target block, the process proceeds to step S124.

[0102] In step S206, the CPU 11, as the processing unit 58, reads the input feature map from the RAM 13, including the surrounding pixels necessary for the convolution process, and performs convolution processing of the input feature map and activation function processing on the results, as in normal CNN inference processing. No processing is performed on other blocks. Here, if the surrounding pixel data of the processing target block is not stored in memory, invalid data is read.

[0103] In step S208, the CPU 11, functioning as the processing unit 58, determines whether the layer is a storage layer. If the layer is not a storage layer, the process proceeds to step S210. On the other hand, if the layer is a storage layer, the process proceeds to step S212.

[0104] In step S210, the output feature map of the processing target block is temporarily stored in the RAM 13. This output feature map of the processing target block is stored until the processing of the next layer.

[0105] In step S212, the CPU 11, functioning as the processing unit 58, determines whether the block is an update block of the storage layer. If the block is not an update block of the storage layer, no processing is performed and the process proceeds to step S124. As a result, for the block, the output feature map of the previous frame is used as the output feature map of the current frame. On the other hand, if the block is an update block, the process proceeds to step S214.

[0106] In step S214, the CPU 11, functioning as the processing unit 58, overwrites the output feature map of the previous frame in the RAM 13 at the same position and in the same layer with the output feature map for the update block of the storage layer.

[0107] In step S124, the CPU 11 determines whether or not the processes of steps S204 to S214 have been completed for all blocks. If there is a block for which the processes of steps S204 to S214 have not been performed, the process returns to step S204, and the processes of steps S204 to S214 are performed for that block.

[0108] In step S126, CPU 11 determines whether or not the processes of steps S201 to S214 and S124 have been completed for all layers. If the processes of steps S201 to S214 and S124 have not been completed for all layers, the process returns to step S201 to process the next layer. On the other hand, if the processes of steps S201 to S214 and S124 have been completed for all layers, the process proceeds to step S128.

[0109] In step S128, CPU 11 determines whether or not the processing of steps S110 to S126 has been completed for all frames. If the processing of steps S110 to S126 has not been completed for all frames, the process returns to step S110, and the next frame is processed as the current frame. On the other hand, if the processing of steps S110 to S126 has been completed for all frames, the process routine ends.

[0110] 11 for each layer up to the next storage layer. However, in step S148, the CPU 11, as the block setting unit 56, sets an update area by expanding the differential area by one pixel width or several pixels width as a margin, determines whether or not at least a portion of the update area is included in each block of several pixels square, sets the block including at least a portion of the update area as an update block, and stores this as update block information in RAM 13. The CPU 11, as the block setting unit 56, also sets a processing target area by expanding the differential area further than the update area, determines whether or not at least a portion of the processing target area is included in each block, sets the block including at least a portion of the processing target area as a processing target block, and stores this as processing block information in RAM 13.

[0111] The other configurations and operations of the image processing device according to the second embodiment are the same as those of the first embodiment, and therefore will not be described.

[0112] As described above, the image processing device according to the second embodiment determines, for frames other than key frames, difference regions from previous frames, sets an update block including an update region corresponding to the difference region for each of predetermined storage layers, sets a processing target block including a processing target region corresponding to the difference region for each of multiple layers, performs processing on the processing target block for each of the multiple layers using a neural network, and overwrites the update block of the output feature map stored in each storage layer. Furthermore, for each storage layer, the device sets a difference region so that the difference region is expanded from the previous layer according to parameters of the convolution processing, and sets an update block including an update region corresponding to the difference region. This allows for a simple configuration to reduce the amount of computation required for processing using a neural network, including convolution processing.

[0113] The present invention is not limited to the device configuration and operation of the above-described embodiment, and various modifications and applications are possible within the scope of the gist of the present invention.

[0114] For example, although the kernel size used for convolution has been described as either 1×1 pixel or 3×3 pixels, the present invention is not limited to this. Kernel sizes other than these may also be used. For example, the kernel size used for convolution may be 5×5 pixels or 7×7 pixels. In this case, if the kernel size used in the previous layer is 5×5 pixels, the difference region is expanded by two pixels to the surrounding area, and if the kernel size used in the previous layer is 7×7 pixels, the difference region is expanded by three pixels to the surrounding area.

[0115] Furthermore, although the image processing device has been described as including a learning unit and an inference unit, the present invention is not limited to this. The device including the learning unit and the device including the inference unit may be configured as separate devices. When there are significant hardware constraints such as power consumption and size, it is preferable to configure the device including the learning unit and the device including the inference unit as separate devices.

[0116] Furthermore, the various processes executed by the CPU after reading software (programs) in the above embodiments may be executed by various processors other than the CPU. Examples of such processors include programmable logic devices (PLDs) such as field-programmable gate arrays (FPGAs), whose circuit configuration can be changed after fabrication, and dedicated electrical circuits such as application-specific integrated circuits (ASICs), which are processors with circuit configurations specifically designed to execute specific processes. Furthermore, the learning process and image processing may be executed by one of these various processors, or by a combination of two or more processors of the same or different types (e.g., multiple FPGAs, or a combination of a CPU and an FPGA). Furthermore, the hardware structure of these various processors is, more specifically, an electrical circuit that combines circuit elements such as semiconductor devices.

[0117] In addition, in each of the above embodiments, the learning processing program and the image processing program are described as being pre-stored (installed) in the storage 14, but this is not limiting. The programs may be provided in a form stored in a non-transitory storage medium such as a CD-ROM (Compact Disk Read Only Memory), a DVD-ROM (Digital Versatile Disk Read Only Memory), or a USB (Universal Serial Bus) memory. The programs may also be downloaded from an external device via a network.

[0118] The following additional notes are provided regarding the above-described embodiments.

[0119] (Additional note 1) An image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, Memory and at least one processor coupled to said memory; Including, The processor: Acquire a video image to be processed; determining a difference region between a frame other than a key frame and a previous frame among the plurality of frames; For frames other than key frames among the plurality of frames, for each of a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region is set among a plurality of blocks obtained by dividing an output feature map; For a key frame of the plurality of frames, processing the frame using the neural network and saving an output feature map for each layer; For a frame other than a key frame among the plurality of frames, processing the frame using the neural network to overwrite the output feature map stored for the update block. It is configured as follows: In setting the update block, the differential region is set so as to expand the differential region from the previous layer to the periphery according to parameters of the convolution process for each layer in which the convolution process is performed, and an update block including the update region according to the differential region is set. Image processing device.

[0120] (Additional note 2) A non-transitory storage medium storing a program executable by a computer including a neural network including convolution processing for a moving image including a plurality of frames to perform image processing, The image processing Acquire a video image to be processed; determining a difference region between a frame other than a key frame and a previous frame among the plurality of frames; For frames other than key frames among the plurality of frames, for each of a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region is set among a plurality of blocks obtained by dividing an output feature map; For a key frame of the plurality of frames, processing the frame using the neural network and saving an output feature map for each layer; For a frame other than a key frame among the plurality of frames, processing the frame using the neural network to overwrite the output feature map stored for the update block. It is configured as follows: In setting the update block, the differential region is set so as to expand the differential region from the previous layer to the periphery according to parameters of the convolution process for each layer in which the convolution process is performed, and an update block including the update region according to the differential region is set. Non-transitory storage medium.

[0121] (Additional note 3) An image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, Memory and at least one processor coupled to said memory; Including, The processor: Acquire a video image to be processed; determining a difference region between a frame other than a key frame and a previous frame among the plurality of frames; For frames other than key frames among the plurality of frames, for each of a predetermined storage layer among a plurality of layers that perform convolution processing of the neural network, set an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map, and set a processing target block including a processing target region corresponding to the difference region for each of the plurality of layers; For a key frame of the plurality of frames, processing the frame using the neural network and storing an output feature map for each of the storage layers; For frames other than key frames among the plurality of frames, for each of the plurality of layers, processing is performed using the neural network for the processing target block, and for each of the storage layers, the updated block of the stored output feature map is overwritten. It is configured as follows: In setting the update block, the difference region is set so as to expand the difference region from the previous layer in accordance with parameters of the convolution process for each storage layer, and an update block including the update region according to the difference region is set. Image processing device.

[0122] (Additional note 4) A non-transitory storage medium storing a program executable by a computer including a neural network including convolution processing for a moving image including a plurality of frames to perform image processing, The image processing Acquire a video image to be processed; determining a difference region between a frame other than a key frame and a previous frame among the plurality of frames; For frames other than key frames among the plurality of frames, for each of a predetermined storage layer among a plurality of layers that perform convolution processing of the neural network, set an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map, and set a processing target block including a processing target region corresponding to the difference region for each of the plurality of layers; For a key frame of the plurality of frames, processing the frame using the neural network and storing an output feature map for each of the storage layers; For frames other than key frames among the plurality of frames, for each of the plurality of layers, processing is performed using the neural network for the processing target block, and for each of the storage layers, the updated block of the stored output feature map is overwritten. It is configured as follows: In setting the update block, the difference region is set so as to expand the difference region from the previous layer in accordance with parameters of the convolution process for each storage layer, and an update block including the update region according to the difference region is set. Non-transitory storage medium. [Explanation of symbols]

[0123] 10 Image processing device 11 CPU 13 RAM 14. Storage 15 Input section 16 Display 20 Learning Department 22 Reasoning part 30 Acquisition Department 38 Processing section 40 Update section 50 Acquisition Department 52 Overall control unit 54 Difference judgment part 56 Block setting section 58 Processing section

Claims

1. An image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, an acquisition unit that acquires a moving image to be processed; a difference determination unit that determines a difference region between a frame other than a key frame among the plurality of frames and a previous frame; a block setting unit that sets, for each of a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region among a plurality of blocks obtained by dividing an output feature map for a frame other than a key frame among the plurality of frames; For a key frame of the plurality of frames, processing the frame using the neural network and saving an output feature map for each layer; a processing unit that performs processing using the neural network on the update block for a frame other than a key frame among the plurality of frames, and overwrites an output feature map stored for the update block; Including, The block setting unit sets the differential region for each layer in which convolution processing is performed in accordance with parameters of the convolution processing so as to expand the differential region from the previous layer to the periphery, and sets an update block including an update region corresponding to the differential region. Image processing device.

2. An image processing device including a neural network that performs convolution processing on a moving image including a plurality of frames, an acquisition unit that acquires a moving image to be processed; a difference determination unit that determines a difference region between a frame other than a key frame among the plurality of frames and a previous frame; a block setting unit that sets, for each of the plurality of frames other than key frames, an update block including an update region corresponding to the differential region among a plurality of blocks obtained by dividing an output feature map, for each of a predetermined storage layer among a plurality of layers that perform convolution processing of the neural network, and sets, for each of the plurality of layers, a processing target block including a processing target region corresponding to the differential region; For a key frame of the plurality of frames, processing the frame using the neural network and storing an output feature map for each of the storage layers; a processing unit that performs processing using the neural network on the processing target block for each of the plurality of layers for frames other than key frames among the plurality of frames, and overwrites the updated block of the output feature map stored in each of the storage layers; Including, The block setting unit sets the differential region for each storage layer in accordance with parameters of a convolution process so as to expand the differential region from a previous layer to a periphery, and sets an update block including an update region corresponding to the differential region. Image processing device.

3. The image processing device according to claim 1 , wherein the block setting unit sets the difference region so that it does not expand beyond a previously designated region.

4. The image processing device according to claim 2 , wherein the block setting unit sets the difference region so that it does not expand beyond a previously designated region.

5. The image processing device according to claim 1 , wherein the block setting unit sets the differential region in a layer subsequent to a pre-specified layer so as not to enlarge the differential region.

6. An image processing method in an image processing device including a neural network that includes convolution processing for a moving image including a plurality of frames, an acquisition unit acquires a moving image to be processed; a difference determination unit that determines a difference region between a frame other than a key frame and a previous frame among the plurality of frames; a block setting unit sets, for each of a plurality of layers that perform convolution processing of the neural network, an update block including an update region corresponding to the difference region, among a plurality of blocks obtained by dividing an output feature map, for frames other than key frames among the plurality of frames; a processing unit, for a key frame among the plurality of frames, processing the frame using the neural network and saving an output feature map of each layer; For frames other than key frames among the plurality of frames, processing is performed on the update block using the neural network, and an output feature map stored for the update block is overwritten. This includes: The block setting unit sets the differential region so that the differential region is expanded from the previous layer to the periphery according to parameters of the convolution process for each layer in which the convolution process is performed, and sets an update block including an update region according to the differential region. Image processing methods.

7. An image processing method in an image processing device including a neural network that includes convolution processing for a moving image including a plurality of frames, an acquisition unit acquires a moving image to be processed; a difference determination unit that determines a difference region between a frame other than a key frame and a previous frame among the plurality of frames; a block setting unit sets, for each of the frames other than the key frames among the plurality of frames, an update block including an update region corresponding to the difference region among the plurality of blocks obtained by dividing an output feature map, for each of a predetermined storage layer among the plurality of layers that perform convolution processing of the neural network, and sets, for each of the plurality of layers, a processing target block including a processing target region corresponding to the difference region; a processing unit, for a key frame among the plurality of frames, processing the frame using the neural network and storing an output feature map for each of the storage layers; For frames other than key frames among the plurality of frames, for each of the plurality of layers, processing is performed using the neural network for the processing target block, and for each of the storage layers, the updated block of the stored output feature map is overwritten. This includes: The block setting unit sets the differential region for each storage layer in accordance with parameters of the convolution process so as to expand the differential region from the previous layer to the periphery, and sets an update block including an update region corresponding to the differential region. Image processing methods.

8. An image processing program for causing a computer to function as the image processing device according to any one of claims 1 to 5.

Images