Image processing device, image processing method, and image processing program

Abstract

To provide an image processing device, an image processing method, and an image processing program, which can reduce the amount of computation in image processing using a neural network even when the moving range of an image object cannot be identified in advance.SOLUTION: In an image processing device that performs image processing using a CNN that performs sparse convolution processing with a feature space configured by feature arrays extracted from an input image as input, image object regions r1 and r2 containing image objects o1 and o2 respectively are extracted from an image 100, and feature arrays v1 and v2 are extracted for the image object regions r1 and r2, respectively. In the feature space, regions corresponding to the image object regions are defined as non-zero regions, feature arrays extracted for the respective image object regions are assigned to the non-zero regions and are subjected to convolution processing, and zero regions other than the non-zero regions are excluded from the convolution processing targets.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program, and in particular to a technique for reducing the amount of calculation required for convolution processing. [Background technology]

[0002] In recent years, with the improvement of computer processing power, the processing time required for multi-layered neural networks (DNNs: Deep Neural Networks) has been reduced, and DNNs have become widely used. As a result, there is a demand for DNNs to process larger amounts of data, one example of which is the field of video recognition.

[0003] Since video data is three-dimensional data with a time axis added to two-dimensional images, 3DCNN (3-Dimensional CNN), an extension of 2DCNN (2-Dimensional Convolutional Neural Network), is known as a DNN for processing video data. Because video data has a larger amount of data than still image data, 3DCNN requires a large amount of calculation, which leads to problems such as large memory usage and long calculation times.

[0004] As shown in Fig. 11, in 2DCNN, the number of weighting factors W multiplied by the input value X is equal to the square of the kernel size ("3" in Fig. 11) of the kernel used in the convolution process multiplied by the number of input channels ("2" in Fig. 11) and the number of output channels ("2" in Fig. 11). Therefore, the number of multiplications in the convolution process is equal to the number of weighting factors W multiplied by the square of the value obtained by dividing the size of each input channel (input size, "6" in Fig. 11) by the stride ("1" in Fig. 11).

[0005] Here, Stride is the interval at which the kernel is multiplied against the input. For example, if Stride is 1, the kernel is multiplied against the input, shifting it by 1. If Stride is 2, the kernel is multiplied against the input, shifting it by 2.

[0006] In summary, the number of multiplications in the convolution process of 2DCNN is (kernel size) 2 × (number of input channels) × (number of output channels) × (input channel size / Stride) 2 …(1) Similarly, the number of multiplications in the convolution process of 3DCNN is (kernel size) 3 × (number of input channels) × (number of output channels) × (input channel size / Stride) 2 × (Number of input frames / Stride) ... (2) Dividing equation (2) by equation (1) gives: (kernel size) 3 × (Number of input frames / Stride) ... (3) For example, if the kernel size is 3, the number of input frames is 16, and the stride is 1, the number of multiplications in the convolution process is 3 times that of 2DCNN. 2 ×(16 / 1) 2 =48 times.

[0007] To address this issue, lowering the resolution of the input image can reduce the amount of calculation required by 3DCNN, but this inevitably leads to a decrease in recognition accuracy when performing video image recognition.

[0008] One method to reduce the amount of computation in 3DCNN without reducing the resolution of the input image focuses on the characteristics of activation functions such as ReLU (Rectified Linear Unit).

[0009] In other words, because activation functions such as ReLU output zero when the input value is negative, neural networks using such activation functions can be said to have sparse activation, in the sense that neurons that output positive values are sparse.

[0010] By focusing on this sparsity, the amount of computation required by the neural network can be reduced by omitting convolution processing in zero regions where there are no neurons that output positive values, and performing convolution processing only in non-zero regions where neurons output positive values.

[0011] Specifically, as shown in Fig. 12, a non-zero region 1201 in the output of the n-1th layer of neural network Y is used as the input of the nth layer, and convolution processing is performed only on the non-zero region 1201 to output the calculation result. A non-zero region 1202 in the output of the nth layer becomes the input of the n+1th layer.

[0012] Therefore, the convolution process of the nth layer of the neural network Y only needs to be performed for the non-zero region 1201 of the output of the n-1th layer, and similarly, the convolution process of the n+1th layer only needs to be performed for the non-zero region 1202 of the output of the nth layer, thereby reducing the number of times the convolution process is performed.

[0013] This conventional technology can be applied not only to 3DCNN but also to a wide variety of neural networks (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0014] [Non-Patent Document 1] Mengye Ren, Andrei Pokrovsky, Bin Yang, Raquel Urtasun, "SBNet: Sparse Blocks Network for Fast Inference", Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018, pp. 8711-8720. [Non-patent document 2] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi, "You Only Look Once: Unified, Real-Time Object Detection", arXiv:1506.02640v5 [cs.CV] 9 May 2016. Summary of the Invention [Problem to be solved by the invention]

[0015] When considering the application of such conventional technology to the field of image processing, the image data input to the first layer of the neural network is often non-zero overall, so the convolution process cannot be reduced as is. For this reason, the above conventional technology uses non-zero regions that are defined in advance based on the characteristics of the image.

[0016] For example, when detecting a LiDAR (Light Detection and Ranging) object, a map mask 1301 of roads and sidewalks corresponding to the input data 13 is prepared as shown in Fig. 13. In this way, the background portion outside the map mask 1301 in the input data 13 can be excluded from the interrupt processing as a zero region, thereby reducing the amount of calculation of the neural network.

[0017] However, when detecting a vehicle, the range in which the vehicle moves is limited to the road, so areas other than the road can be excluded as zero areas. However, if the range in which an image object of interest in an image can move extends over the entire image, it is not possible to define a mask (zero area) as in the above-mentioned conventional technology.

[0018] For the same reason, even if the range in which an image object can move is not the entire image but extends widely within the image, the convolution process cannot be effectively reduced.

[0019] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, an image processing method, and an image processing program that can reduce the amount of calculation in image processing using a neural network, even when the movement range of an image object cannot be specified in advance. [Means for solving the problem]

[0020] In order to achieve the above object, an image processing device according to an embodiment of the present disclosure is an image processing device that performs image processing using CNN that receives as input a feature space configured by a feature array extracted from an input image, and Only a part of it Image Object Area as The CNN is characterized by comprising an image object region extraction unit that extracts image object regions and a feature sequence extraction unit that extracts feature sequences for each image object region, and by assigning regions in the feature space that correspond to the image object regions as non-zero regions to which the feature sequences extracted for each image object region are subjected to convolution processing, and by performing sparse convolution processing in which zero regions other than the non-zero regions are excluded from the convolution processing.

[0021] In this case, the image object area is the area of the image object. The aforementionedThe rectangular region containing the non-zero region may be associated with an image object region depending on the position and size of the rectangular region in the input image.

[0022] Alternatively, the image object may be a person, and the CNN may perform image processing to recognize the person's behavior.

[0023] Alternatively, the image object may be a person or an object, and the CNN may perform image processing to recognize the scene of the input image. 4. The image processing device according to claim 1, wherein the image processing device is a computer.

[0024] Furthermore, the predetermined image object may be an image object that is to be subjected to image processing using CNN.

[0025] The feature sequence extraction unit may also extract feature sequences using any of a convolutional neural network, a multilayer perceptron, a neural network other than a convolutional neural network or a multilayer perceptron, principal component analysis, and a predefined feature calculation method.

[0026] An image processing device according to another aspect of the present disclosure is an image processing device that performs image processing using a CNN that receives as input a feature space configured by feature sequences extracted from an input image, and includes: an image object region extraction unit that extracts an image object region that includes at least a part of a predetermined image object in the input image; and a feature sequence extraction unit that extracts a feature sequence for each image object region, wherein the CNN sets regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them targets for convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from targets for convolution processing; CNN on SBnet It is characterized by .

[0027] An image processing device according to another embodiment of the present disclosure is an image processing device that performs image processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input video, and includes: an image object region extraction unit that extracts image object regions that include at least a portion of a predetermined image object in the input video and are common to frames that include the image object; and a feature sequence extraction unit that extracts a feature sequence for each image object region, and the CNN is characterized in that it treats regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the subject of convolution processing, and performs sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

[0028] In this case, the feature sequence extracting unit may extract a feature sequence for each image object region for each frame.

[0029] In addition, the image object region extraction unit extracts a frame-specific region including at least a portion of a specified image object for each frame constituting the input video, and the image object region may be a sum region of frame-specific regions including at least a portion of an image object that is common between frames.

[0030] In addition, the image object region extraction unit extracts a rectangular frame-specific region containing at least a portion of a specified image object for each frame constituting the input video, and the image object region may be the smallest rectangular region among rectangular regions containing a union region of frame-specific regions containing at least a portion of an image object that is common between frames.

[0031] In addition, the image object region extraction unit may include a stillness determination unit that determines whether each image object is still, and extracts a frame-by-frame region that includes at least a portion of a specified image object for each frame that constitutes the input moving image, and obtains an extraction likelihood for each frame-by-frame region.Of the frame-by-frame regions related to a common image object, the frame-by-frame region with the highest extraction likelihood may be determined to be the image object region related to that image object.

[0032] In addition, the feature sequence extraction unit may extract feature sequences related to any one of a plurality of image object regions that relate to a common image object and that relate to a stationary image object, thereby extracting feature sequences related to the other image object regions.

[0033] In addition, the image object region extraction unit may obtain, for each frame-specific region, a class to which the image object related to the frame-specific region belongs, and the stillness determination unit may determine whether the image object is still or not from the class to which the image object belongs.

[0034] Furthermore, the stillness determination section may determine that an image object is still when frame-specific regions relating to the common image object are located at a common position within a frame.

[0035] Alternatively, the image object may be a person, and the CNN may perform image processing to recognize the person's behavior.

[0036] Alternatively, the image object may be a person or an object, and the CNN may perform image processing to recognize a scene in the input video.

[0037] Furthermore, the predetermined image object may be an image object that is to be subjected to image processing using CNN.

[0038] Furthermore, when a still image object identical to a still image object included in one input image is included in another input video, the CNN may set an area in the feature space corresponding to the image object area relating to the same still image object included in the other input image as a zero area and exclude it from the target of convolution processing, and may use the image processing result for the still image object included in the one input video among the image processing results by the CNN for the one input video as the image processing result for the same still image object included in the other input image.

[0039] The CNN may also be a 3D CNN.

[0040] The CNN may also be SBnet.

[0041] The feature sequence extraction unit may also extract feature sequences using any of a convolutional neural network, a multilayer perceptron, a neural network other than a CNN or a multilayer perceptron, principal component analysis, and a predefined feature calculation method.

[0042] An image processing method according to an embodiment of the present disclosure is an image processing method executed by an image processing device that executes image processing using CNN, which receives as input a feature space configured by a feature array extracted from an input image, and Only a part of it Image Object Area as The CNN includes an image object region extraction step for extracting an image object region and a feature sequence extraction step for extracting a feature sequence for each image object region, and is characterized in that the CNN performs sparse convolution processing by treating regions in the feature space that correspond to the image object regions as non-zero regions, allocating the feature sequences extracted for each image object region to the non-zero regions and making them the target of convolution processing, and by excluding zero regions other than the non-zero regions from the target of convolution processing.

[0043] An image processing method according to another embodiment of the present disclosure is an image processing method executed by an image processing device that executes video processing using a CNN that receives as input a feature space constituted by feature sequences extracted from an input video, and includes an image object region extraction step of extracting image object regions that include at least a portion of a predetermined image object in the input video and are common to frames that include the image object, and a feature sequence extraction step of extracting a feature sequence for each image object region, wherein the CNN defines regions in the feature space that correspond to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the subject of convolution processing, and executes sparse convolution processing in which zero regions other than the non-zero regions are excluded from the subject of convolution processing.

[0044] An image processing program according to an embodiment of the present disclosure is an image processing program executed by a computer that executes image processing using CNN, which receives as input a feature space configured by a feature array extracted from an input image, and Only a part of it Image Object Area as The computer is caused to execute an image object region extraction step to extract an image object region and a feature sequence extraction step to extract a feature sequence for each image object region, and the CNN is characterized in that it treats regions in the feature space corresponding to the image object regions as non-zero regions, assigns the feature sequences extracted for each image object region to the non-zero regions, and makes them the target of convolution processing, and executes sparse convolution processing in which zero regions other than the non-zero regions are excluded from the target of convolution processing.

[0045] An image processing program according to another embodiment of the present disclosure performs video processing using CNN that receives as input a feature space configured by a feature sequence extracted from an input video. RukoAn image processing program executed by a computer, the image processing program including: an image object region extraction step of extracting an image object region that includes at least a part of a predetermined image object in an input video image and is common to frames that include the image object; and a feature array extraction step of extracting a feature array for each image object region. of and a feature sequence extraction step of extracting feature sequences from the image object regions, and the CNN is characterized in that it performs sparse convolution processing by treating regions in the feature space corresponding to image object regions as non-zero regions, allocating the feature sequences extracted for each image object region to the non-zero regions and making them targets for convolution processing, and excluding zero regions other than the non-zero regions from targets for convolution processing. [Effects of the Invention]

[0046] In this way, non-zero regions in the feature space are set using an image object region that includes at least a part of a specified image object in the input image, and convolution processing is performed by doing so, while zero regions other than the non-zero regions are excluded from the convolution processing. Therefore, even if it is not possible to set non-zero regions in advance due to the characteristics of the image object, sparse convolution processing can be performed, thereby reducing the amount of calculation required for CNN. [Brief explanation of the drawings]

[0047] [Figure 1] 1 is a diagram illustrating an example of an operating environment of an image processing device 1 according to a first embodiment of the present disclosure. [Figure 2] 1A is a block diagram showing the main hardware configuration of the image processing device 1, and FIG. 1B is a block diagram showing the functional configuration of the image processing device 1. FIG. [Figure 3] 1 is a schematic diagram illustrating the process of extracting image object regions r1 and r2 from an image 100, and then extracting feature arrays v1 and v2 to generate a feature space S. FIG. [Figure 4](a) is a diagram illustrating a binary mask that sets non-zero and zero regions in feature space S, and (b) is a diagram illustrating an index list for simplifying the binary mask in (a) and inputting it into SBnet. [Figure 5] 1 is a flowchart illustrating processing for generating a feature space S to be input to a CNN by an image processing device 1 according to the first embodiment. [Figure 6] FIG. 10 is a block diagram showing a functional configuration of an image processing device 1 according to a second embodiment. [Figure 7] This is a schematic diagram that explains the processing from extracting frame-by-frame regions from a moving image 700 to extracting an image object region by determining whether the image object is a stationary object or tracking a moving image object. [Figure 8] FIG. 10 is a schematic diagram illustrating the process of extracting a feature array for each image object region for each frame and generating a feature space S for each frame. [Figure 9] 10 is a flowchart illustrating processing for generating a feature space S to be input to a 3DCNN by an image processing device 1 according to a second embodiment. [Figure 10] 10A and 10B are schematic diagrams illustrating a process for reducing the amount of calculation for obtaining spatiotemporal features of a stationary object according to a modified example of the present disclosure. [Figure 11] FIG. 1 is a diagram illustrating the amount of calculation required for a typical CNN. [Figure 12] FIG. 1 is a diagram illustrating a configuration for reducing the amount of calculation used in SBnet. [Figure 13] FIG. 10 is a diagram illustrating an example of an application in which the amount of calculation required to detect a moving object is reduced using SBnet. DETAILED DESCRIPTION OF THE INVENTION

[0048] Hereinafter, embodiments of an image processing device, an image processing method, and an image processing program according to the present disclosure will be described with reference to the drawings. [1] First embodiment The image processing device according to this embodiment processes individual frames that make up still images and moving images. (1-1) Operating environment of image processing device As shown in Fig. 1, an image processing device 1 according to this embodiment is connected to a video camera 110 via a communication network 120. The video camera 110 is capable of capturing still images as well as moving images made up of a plurality of frames. Hereinafter, still images and individual frames constituting moving images will be collectively referred to simply as "images."

[0049] The image processing device 1 acquires an image of a person s1 and an object (a "chair" in the example of FIG. 1) s2 from a video camera 110. In FIG. 1, an image 100 acquired by the image processing device 1 from the video camera 110 is displayed. The image 100 includes an image object o1 of the person s1 and an image object o2 of the object s2.

[0050] The image processing device 1 performs image processing such as identifying a person s1 or an object s2, identifying the skeleton and behavior of the person s1, and identifying the type of scene in the image. The image processing device 1 performs such image processing using a convolutional neural network (CNN).

[0051] As the name suggests, CNN performs image processing such as identifying people, objects, human skeletons and behavior, and scenes depicted in images by repeating convolution processing. The higher the image resolution, the greater the amount of calculation required for this convolution processing. (1-2) Hardware configuration of image processing device 1 The image processing device 1 is a so-called computer, and as illustrated in Figure 2(a), it comprises a CPU (Central Processing Unit) 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an HDD (Hard Disk Drive) 204, and a NIC (Network Interface Card) 205 connected via an internal bus 206.

[0052] When the image processing device 1 is reset by turning on the power or the like, the CPU 201 reads and starts up a boot program from the ROM 202, and executes the OS (Operating System) and image processing program 207, etc., read from the HDD 204, using the RAM 203 as a working memory area.

[0053] The HDD 204 can store data such as the OS, application programs such as the image processing program 207, images 100 captured using the video camera 110, and the results of image processing by the image processing program 207.

[0054] The image processing device 1 may be a single computer, or may use multiple computers for distributed processing. It goes without saying that the number of CPUs 201 is not limited to one, and a multi-processor configuration may also be used. The hardware configuration of the image processing device 1 is not limited as long as it is a computer capable of executing the image processing program 207 according to this embodiment.

[0055] The communication network 120 may be a LAN (Local Area Network), the Internet, or a combination thereof, or may be wired communication, wireless communication, or a combination thereof. It may also be serial communication such as USB (Universal Serial Bus), and the type of communication network is not limited as long as it can send and receive the image 100. (1-3) Functional configuration of image processing device 1 By having the above hardware configuration and executing an image processing program, the image processing device 1 realizes the functions shown in FIG. 2(b). (1-3-1) Image object region extraction unit 211 When the image object region extraction unit 211 receives the image 100 captured using the video camera 110, it extracts an image object region for cutting out (cropping) each image object included in the image 100.

[0056] 3, an image object (person) o1 and an image object (object) o2 are included in the image 100. The image object region extraction unit 211 extracts an image object region r1 for cropping the image object o1 and an image object region r2 for cropping the image object o2.

[0057] To extract an image object region, for example, YOLO (You Only Look Once) can be used (see Non-Patent Document 2). Using YOLO, it is possible to obtain the coordinate values (x and y coordinates of the upper left vertex on the image) and size (width w and height h of the bounding box) of a rectangular bounding box (BB) that surrounds an image object included in image 100, and therefore this bounding box can be used as the image object region of the image object.

[0058] Furthermore, by using YOLO, it is possible to obtain for each bounding box a confidence score indicating the likelihood of detecting the bounding box, and a class probability indicating which class the image object enclosed by the bounding box is classified into.

[0059] Note that an image object region may be extracted by detecting only a part of an image object, such as the face portion of an image object of a person included in the image 100. (1-3-2) Feature sequence extraction unit 212 The feature sequence extraction unit 212 extracts a feature sequence for each of the image object regions r1 and r2. This feature sequence will ultimately become input data for the CNN 214 that performs image processing.

[0060] In the prior art, feature sequences are extracted from the entire image 100, or from only a predetermined region on the image 100, and input to the CNN 214. In contrast, in the present embodiment, feature sequences are extracted from only image object regions that may differ for each image 100, and input to the CNN 214 via the feature space generation unit 213.

[0061] In the example of Figure 3, feature array v1 is extracted for image object region r1, where h1 and w1 are the height and width of image object region r1 in image 100, expressed in number of pixels, respectively. Also, c is the number of input channels. That is, CNN 214 performs c types of convolution processing using feature array v1 as input.

[0062] Similarly, a feature array v2 is extracted for the image object region r2, where h2 and w2 are the height and width of the image object region r2 in the image 100, and c is the number of input channels.

[0063] The method for extracting the feature sequence for the image object region may be the same as the method for extracting the feature sequence for the image object region in the prior art.

[0064] The feature sequence extraction unit 212 may extract feature sequences using a CNN other than the CNN 214, or may extract feature sequences using a multi-layer perceptron (MLP). A multi-layer perceptron is a forward propagation type neural network having at least three node layers.

[0065] Also, the feature sequence may be extracted using a neural network other than CNN and multilayer perceptron, principal component analysis, or a predefined feature calculation method.

[0066] Principal component analysis is a method of synthesizing a small number of uncorrelated variables called principal components that best represent the overall variation from a large number of correlated variables using multivariate analysis (a statistical data analysis method). For example, a principal component synthesized from a large number of pixel values that make up an image object region (in a color image, consisting of gradation values for each color component) can be used as a feature array. (1-3-3) Feature space generation unit 213 The feature space generation unit 213 generates a feature space S to be input to the CNN 214, using information (coordinate values and size in the YOLO example) that identifies the image object regions r1 and r2 on the image 100 extracted by the image object region extraction unit 211 and the feature array for each of the image object regions r1 and r2 extracted by the feature array extraction unit 212.

[0067] Specifically, in the feature space S input to the CNN 214, the regions corresponding to the image object regions r1 and r2 are set as non-zero regions, and the feature arrays v1 and v2 related to the image object regions r1 and r2 are assigned to the non-zero regions.

[0068] This non-zero region is the same region to which the extracted feature sequences for the regions corresponding to the image object regions r1 and r2 are assigned when the feature space S is generated by extracting feature sequences for the entire image 100.

[0069] 11 used to explain the conventional technology, the non-zero regions 1101 and 1102 are fixed in advance, whereas in this embodiment, the non-zero regions can vary depending on the image object regions r1 and r2 in the image 100. Therefore, even if the image objects o1 and o2 can be located in an unspecified range within the image 100, the size of the non-zero regions can be minimized within the range in which the CNN 214 can perform image processing.

[0070] The feature space generation unit 213 generates information for identifying non-zero regions in the feature space S. In the example of Fig. 3, h and w in the feature space S are the height and width, respectively, of the image 100. Also, c is the number of input channels.

[0071] Since the feature arrays v1 and v2 share the same number of input channels c as the feature space S, by specifying the heights h1 and h2 and widths w1 and w2, the range of the non-zero region in the feature space S can be identified and the feature arrays v1 and v2 can be assigned within the feature space S.

[0072] The non-zero regions may be identified using a binary mask that assigns a "1" to pixels in the non-zero regions of the image 100 and a "0" to pixels in the zero regions.

[0073] When the CNN 214 uses SBnet (see Non-Patent Document 1), the feature space generation unit 213 converts the binary mask into an index list to identify non-zero regions. For example, in the example of FIG. 4(a), "1" is assigned to pixels in the non-zero regions (shaded areas) of the binary mask 401, and "0" is assigned to pixels in the zero regions (non-shaded areas).

[0074] In contrast to this, the pixel in Figure 4(a) whose x coordinate is "3" and whose y coordinate is "0" is a non-zero area to which "1" is assigned, and in the example of Figure 4(b), index list 402 has a column where the x coordinate column is "3" and the y coordinate column is "0." In other words, index list 402 contains the coordinate values of pixels that belong to the non-zero area.

[0075] The feature space generation unit 213 may identify non-zero regions and zero regions in the feature space S in this manner.

[0076] A feature array is assigned to each x- and y-coordinate in the non-zero region of the feature space S. The number of elements in the feature array is equal to the number of input channels c. (1-3-4)CNN214 The CNN 214 performs sparse convolution processing, which convolves only the non-zero regions of the feature space S and does not convolve the zero regions. Therefore, the amount of calculation can be reduced by the amount of convolution processing for the zero regions.

[0077] Furthermore, when the image object that is the subject of the convolution process of the CNN 214 can move over a wide range on the image, the conventional technology is forced to widen the non-zero region, which is less effective in reducing the amount of calculation required for the convolution process.

[0078] On the other hand, in this embodiment, the non-zero area can be narrowed down to the range occupied by the image object on the image, so even if the range over which the image object can move on the image is wide, the amount of calculation required for the convolution process can be reduced effectively. (1-4) Processing of the image processing device 1 The image processing device 1 executes the image processing program 207 with the aim of reducing the amount of calculation required for image processing by the CNN 214. That is, as shown in Fig. 5, first, when an image 100 is acquired by receiving it from the video camera 110 or reading it from the HDD 204 (S501), the image object area extraction unit 211 extracts an image object area for each image object (S502).

[0079] Next, the feature sequence extraction unit 212 extracts a feature sequence for each image object region in accordance with the processing content of the CNN 214 (S503).

[0080] The feature space generation unit 213 sets non-zero regions and zero regions in the feature space S according to the image object region (S504), and assigns the non-zero region feature array (S505). Then, image processing of the CNN 124 is performed using the assigned feature array as an input to the feature space S.

[0081] After the CNN 214 executes image processing, for example, the processing results are stored in the HDD 204, and the processing ends. [2] Second embodiment Next, a second embodiment of the present disclosure will be described. The image processing device 1 according to this embodiment is a device that performs moving image processing using 3DCNN, and has a configuration that is generally common to the image forming processing device 1 according to the first embodiment, but differs in the way in which the image object region is set. The following description will mainly focus on the differences.

[0082] In this specification, common reference numerals are used to designate components and the like that are common between the embodiments and the modified examples. (2-1) Functional configuration of image processing device 1 As shown in Figure 6, the image processing device 1 has the functions of a frame-by-frame region extraction unit 601, a stationary object determination unit 602, a tracking unit 603, an image object region extraction unit 604, a feature array extraction unit 605, a feature space generation unit 606, and a 3DCNN 607. (2-1-1) Frame-by-frame region extraction unit 601 The frame region extraction unit 601 extracts a frame region as a region for cropping an image object for each frame that constitutes a moving image.

[0083] In the example of FIG. 7, frames t1 to tn constituting a moving image 700 include image objects (people) o1(t1) to o1(tn) and image objects (materials) o2(t1) to o2(tn), respectively.

[0084] The frame-specific region extraction unit 601 extracts frame-specific regions r1(t1) to r1(tn) for cropping image objects (people) o1(t1) to o1(tn) and frame-specific regions r2(t1) to r2(tn) for cropping image objects (objects) o2(t1) to o2(tn) from frames t1 to tn.

[0085] Similar to the image object region extraction unit 211 according to the first embodiment, the frame region extraction unit 601 may also extract frame regions r1(t1) to r1(tn) and r2(t1) to r2(tn) using YOLO.

[0086] Note that frame-by-frame regions r1(t1) to r1(tn) may be extracted by targeting only a portion of an image object, such as the face portion of an image object of a person included in each of frames t1 to tn. (2-1-2) Stationary object determination section 602 The stationary object determination unit 602 determines whether an image object in a moving image is a stationary object or a moving object.

[0087] As described above, when detecting a bounding box surrounding an image object using YOLO, it is also possible to obtain a class probability indicating which class the image object is classified into.

[0088] If it is determined using this class probability that the image object belongs to the class of stationary objects such as chairs, the stationary object determination unit 602 determines that the image object is a stationary object.

[0089] On the other hand, if it is determined from the class probability that the image object belongs to the moving object class, the stationary object determination unit 602 determines that the image object is not a stationary object. Which classes belong to stationary objects and which classes belong to moving objects are set in advance.

[0090] In addition, for frame-specific areas relating to image objects o1(t1) to o1(tn) that are common between frames, such as frame area r1(t1) to r1(tn), or frame-specific areas relating to image objects o2(t1) to o2(tn) that are common between frames, such as frame area r2(t1) to r2(tn), if the frame-specific areas relating to image objects that are common between frames are in the same position, the image objects may be determined to be stationary.

[0091] Furthermore, if the environment in which the video camera 110 is photographed is constant, it is also possible to determine whether an image object is a stationary object by calculating the difference between frames.

[0092] Whether or not frame-specific regions are in the same position can be determined based on criteria such as whether specific vertices of the frame-specific regions are within a specified range of each other, or whether the proportion of the area of the overlapping frame-specific regions to the area of the frame-specific regions is equal to or greater than a specified proportion.

[0093] In the example of Figure 7, all of the frame-specific regions r2(t1) to r2(tn) are classified into the "chair" class, and the "chair" class belongs to stationary objects, so the image objects o1(t1) to o2(tn) are determined to be stationary objects. (2-1-3) Tracking unit 603 When an image object in a moving image is a moving object, the tracking unit 603 tracks the image object to identify the correspondence between frame-by-frame regions. An image object determined not to be a stationary object by the stationary object determination unit 602 is a moving object, and therefore becomes a target for tracking by the tracking unit 603.

[0094] As described above, when detecting a bounding box surrounding an image object using YOLO, it is also possible to obtain a class probability indicating which class the image object is classified into.

[0095] If it is possible to use this class probability to uniquely determine a combination of bounding boxes (frame-specific regions) that have a common class between frames, the image object enclosed by the bounding box may be tracked.

[0096] Other conventional techniques for adding image objects contained in video may be used to track the image objects and thereby identify the frame-by-frame region correspondences between frames.

[0097] In the example of Figure 7, frame regions r2(t1) to r2(tn) are all classified into the "person" class, which belongs to moving objects, and there is only one frame region classified into the "person" class in each frame. Therefore, frame regions r2(t1) to r2(tn) are identified as corresponding frame regions between frames. (2-1-4) Image object region extraction unit 604 The image object region extraction unit 604 extracts an image object region including a frame-specific region related to a common image object as an image object region common between frames, thereby extracting an image object region for cropping the image object, and in particular, utilizing the image object region as a non-zero region.

[0098] In the example of FIG. 7, the smallest rectangular area among the rectangular areas including the frame-specific areas r1(t1) to r1(tn) related to the image objects o1(t1) to o1(tn) is extracted as the image object area r1.

[0099] Furthermore, as described above, YOLO can obtain a confidence score for each bounding box that indicates the likelihood of detecting that bounding box.

[0100] In the example of Fig. 7, this reliability score is used to select the frame region r2(tk) using the bounding box with the highest reliability among the frame regions r2(t1) to r2(tn) related to a common stationary object (chair), and the frame region r2(tk) is determined to be the image object region r2. Needless to say, t1≦tk≦tn. (2-1-5) Feature sequence extraction unit 605 The feature sequence extraction unit 605 extracts a feature sequence for each image object region for each frame, similar to the feature sequence extraction unit 212 according to the first embodiment.

[0101] In the example of Figure 8, feature array v1 is extracted for image object region r1, where h1 and w1 are the height and width, in number of pixels, of image object region r1 in each frame of video 700. Also, c is the number of input channels. That is, the 3DCNN 214 performs c types of convolution processing using feature array v1 as input.

[0102] Image objects o1(t1) to o1(tn) are moving objects (people) and may be in different positions and sizes in each frame. For this reason, the image object region r1 has a larger size (height h1 and width w1) than the individual frame regions r1(t1) to r1(tn).

[0103] Similarly, feature array v2 is extracted for image object region r2, where h2 and w2 in feature array v2 are the height and width of image object region r2 in each frame of video 700, and c is the number of input channels.

[0104] In addition, for image object areas related to stationary image objects, a feature array may be extracted for the image object area included in one of multiple frames, and when extracting feature arrays for image object areas included in other frames, the previously extracted feature array may be reused. (2-1-6) Feature space generation unit 606 Similar to the feature space generation unit 213 according to the first embodiment, the feature space generation unit 213 generates a feature space S for the entire video 700 to be input to the 3DCNN 607, using information identifying image object regions r1 and r2 on the image 100 extracted by the image object region extraction unit 604 for each frame constituting the video 700, and the feature array for each image object region r1 and r2 extracted by the feature array extraction unit 605.

[0105] 8, a feature space S is generated for each frame, similar to the feature space S according to the first embodiment. The feature spaces S generated for each of the frames r1 to tn are input to the 3DCNN 607.

[0106] If an image object appears and disappears partway through the video 700, the feature array assigned to the non-zero region of a frame that does not contain the image object may be a zero array, which is an array in which all array components are zero.

[0107] Alternatively, for frames that do not include the image object, the region in the feature space S corresponding to the image object region may be set to a zero region. (2-1-7)3DCNN607 Similar to the CNN 214 according to the first embodiment, the 3DCNN 607 performs sparse convolution, which performs convolution only on non-zero regions of the feature space S for the entire video 700, and does not perform convolution on zero regions. Therefore, the amount of calculation can be reduced by the amount of convolution on the zero regions.

[0108] Furthermore, in the prior art, the entire range in which an image object can move within the video 700 must be considered a non-zero region, whereas in the present embodiment, the non-zero region can be narrowed down to the range in which the image object actually moved within the video 700, which is highly effective in reducing the amount of calculation required for the convolution process. (2-2) Processing of the image processing device 1 9, the image processing device 1 acquires the moving image 700 by receiving it from the video camera 110 or reading it from the HDD 204 (S901). As described above, since it is necessary to extract the image object area by extracting the frame area, the image processing device 1 acquires the entire moving image 700 in a lump, rather than acquiring the frames that make up the moving image 700 one by one, in parallel with the shooting by the video camera 110.

[0109] The image processing device 1 detects a frame-by-frame region for each frame constituting the moving image 700 using the frame-by-frame region extraction unit 601 (S902), and then determines whether the image object related to the frame-by-frame region is a still object using the stationary object determination unit 602 (S903).

[0110] Furthermore, for frame-by-frame areas relating to image objects that are determined to be moving rather than stationary, the tracking unit 603 performs tracking processing of the moving object (S904).

[0111] An image object region for cropping the image object is extracted from the frame-specific region in which the image object is common (S905). In this case, as described above, the method for extracting image objects differs between frame-specific regions related to stationary objects and frame-specific regions related to moving objects. Furthermore, a feature sequence is extracted for each frame and for each image object region (S906).

[0112] Thereafter, non-zero regions and zero regions are set in the feature space using the image object region for each image object (S907), and a feature array is assigned to the non-zero region (S908).

[0113] The 3DCNN 607 performs image processing using the feature space S in which feature arrays are assigned to non-zero regions as input (S909). The image processing results by the 3DCNN 607 may be stored in the HDD 204, for example. [3] Variation The present disclosure has been described above based on the embodiments, but it goes without saying that the present disclosure is not limited to the above-described embodiments, and the following modified examples can be implemented. (3-1) In the above second embodiment, the case where image processing is performed once using the 3DCNN 607 has been described as an example. However, it goes without saying that the present disclosure is not limited to this. Image processing may be performed multiple times using the 3DCNN 607. In such a case, the following may be performed.

[0114] If the video camera 110 is fixed, a stationary object remains stationary even in the captured image. By focusing on this point, the amount of calculation required for the 3DCNN 607 can be further reduced.

[0115] That is, as shown in FIG. 10, first, image processing is performed by the 3DCNN 607 on a stationary object between time T and T+ΔT to calculate spatiotemporal features.

[0116] After that, if a stationary object that remains stationary between time T' and T'+△T is the same as the stationary object for which the spatiotemporal features were previously calculated, the feature array is not calculated for each image object region of the stationary object. Therefore, the non-zero region related to the stationary object is not set in the feature space S, nor is a feature array assigned to the non-zero region. Instead, image processing is first performed by the 3DCNN 607, and the calculated spatiotemporal features are reused.

[0117] Whether or not the object is the same as the still object whose spatiotemporal features were calculated earlier may be determined, for example, by calculating the difference between an image including the earlier still object and an image including the later still object, or by other methods.

[0118] In this way, the amount of calculation required for image processing by the 3DCNN 607 can be reduced, and the amount of calculation required to generate the feature space S can also be reduced. (3-2) In the above embodiment, an example has been described in which an image object region is extracted using YOLO. However, it goes without saying that the present disclosure is not limited to this example, and the following method may be used instead.

[0119] For example, an image object contained in image 100 may be extracted, and then the rectangular area having the vertex with the smallest x-coordinate and y-coordinate in the image object and the vertex with the largest x-coordinate and y-coordinate in the image object at diagonal positions may be defined as the image object area.

[0120] Furthermore, the image object region may be extracted using a method other than the above.

[0121] Regardless of the method used to extract an image object region, by applying the present disclosure, the amount of computation required by CNN can be reduced, as in the above embodiment.

[0122] Furthermore, in order to reduce the amount of computation required for CNN, it is desirable to extract the image object region so that it is as small as possible. Therefore, if only the image object is included in the image object region, the amount of computation required for CNN can be reduced to the greatest extent. (3-3) In the above embodiment, the case where a moving image is processed using 3DCNN has been described as an example, but it goes without saying that the present disclosure is not limited to this, and instead of 3DCNN, a moving image may be processed using a CNN other than 3DCNN or a neural network other than CNN. Even when a moving image is processed using a neural network other than 3DCNN, the same effect as above can be obtained by applying the present disclosure. (3-4) The present disclosure may be an image processing method realized by processing by the above-described image processing device 1. Furthermore, the present disclosure may be an image processing program realized by executing these methods by a processor such as a CPU, or may be a digital signal consisting of this image processing program. The present disclosure may also be the image processing program or the digital signal recorded on a computer-readable recording medium. Examples of computer-readable recording media include flexible disks, hard disks, CD-ROMs, MOs, DVDs, DVD-ROMs, DVD-RAMs, Blu-ray (registered trademark) Discs, and semiconductor memories. The present disclosure may also be the digital signal recorded on these recording media. Furthermore, the present disclosure may also be such that the image processing program or the digital signal is transmitted via a telecommunications line, a wireless or wired communication line, a network such as the Internet, data broadcasting, or the like. The present disclosure may also provide a computer system having a microprocessor and a memory, wherein the memory stores the image processing program, and the microprocessor operates in accordance with the image processing program. The image processing program or the digital signal may also be implemented by another independent computer system by recording it on the recording medium and transferring it, or by transferring the program or the digital signal via the network or the like. (3-5) The above-described embodiments and modifications may be combined with each other. Furthermore, all the numbers used above are merely examples to specifically explain the present disclosure, and the present disclosure is not limited to the numbers used as examples. The division of functional blocks in the block diagram is an example, and multiple functional blocks may be realized as a single functional block, one functional block may be divided into multiple blocks, or some functions may be moved to another functional block.Furthermore, the functions of multiple functional blocks having similar functions may be processed in parallel or time-shared by a single piece of hardware or software. The order in which the steps are performed is merely an example for specifically explaining the present disclosure, and other orders may be used. Some of the steps may be performed simultaneously (in parallel) with other steps. Furthermore, various modifications of the present embodiment that are within the scope of what would occur to a person skilled in the art are also included in the present disclosure, as long as they do not deviate from the gist of the present disclosure. [Industrial Applicability]

[0123] The image processing device, image processing method, and image processing program according to the present disclosure are useful as techniques for reducing the amount of calculation required for convolution processing. [Explanation of symbols]

[0124] 1. Image processing device 100……………………………Image (still image) 110……………………………Video camera 120……………………………Communications network 207...................................................Image processing program 211, 604...Image object region extraction unit 212, 605…………………Feature sequence extraction section 213, 606...Feature space generation section 214………………………………CNN(Convolutional Neural Network) 401……………………………Binary Mask 402……………………………Index List 601: Frame-by-frame region extraction unit 602………………………………Stationary object determination section 603……………………………Tracking section 607………………………………3DCNN(3-dimensional CNN) 700……………………………Video 1201, 1202...Non-zero area 1301…………………………Map Mask o1, o2...Image objects o1(t1)~o1(tn)...Image objects o2(t1)~o2(tn)...Image objects r1, r2...Image object area r1(t1)~r1(tn)...Area by frame r2(t1)~r2(tn)...Area by frame v1, v2………………………Feature array S…………………………………Feature space

Claims

1. An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image, an image object region extraction unit that extracts only a portion of a predetermined image object in an input image as an image object region; a feature sequence extraction unit that extracts a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process.

1. An image processing device comprising:

2. The image object region is a rectangular region including the part of the image object, The non-zero regions are mapped to image object regions according to the position and size of the rectangular regions in the input image.

2. The image processing device according to claim 1, wherein:

3. The image object is a person, CNN performs image processing to recognize human behavior.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. The image object is a person or object, CNN performs image processing to recognize the scene of the input image.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. The predetermined image object is an image object that is a target of image processing using a CNN.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. The feature sequence extraction unit extracts the feature sequence by any one of a convolutional neural network, a multilayer perceptron, a neural network other than a convolutional neural network or a multilayer perceptron, principal component analysis, and a predefined feature calculation method.

6. The image processing device according to claim 1,

7. An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image, an image object region extraction unit that extracts an image object region including at least a part of a predetermined image object from an input image; a feature sequence extraction unit that extracts a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution processing is performed, excluding them from the convolution processing target. The CNN is SBnet 1. An image processing device comprising:

8. An image processing device that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video, an image object region extraction unit that extracts an image object region that includes at least a part of a predetermined image object in an input video sequence and that is common to frames that include the image object; a feature sequence extraction unit that extracts a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process.

1. An image processing device comprising:

9. The feature sequence extraction unit extracts a feature sequence for each image object region for each frame.

9. The image processing device according to claim 8,

10. The image object region extraction unit extracting a frame-by-frame region including at least a portion of a predetermined image object for each frame constituting the input video; The image object region is a sum of frame regions that include at least a part of an image object that is common between frames.

10. The image processing device according to claim 8, wherein the image processing device is a computer.

11. The image object region extraction unit extracting a rectangular frame-by-frame region including at least a portion of a predetermined image object for each frame constituting the input video; The image object area is the smallest rectangular area among rectangular areas including the sum area of frame-specific areas that includes at least a part of an image object that is common between frames.

10. The image processing device according to claim 8, wherein the image processing device is a computer.

12. The image object region extraction unit a stillness determination unit that determines whether each image object is still; extracting a frame-by-frame region including at least a part of a predetermined image object for each frame constituting the input video sequence, and obtaining an extraction likelihood for each frame-by-frame region; Among the frame regions relating to a common image object, the frame region with the highest extraction likelihood is determined as the image object region relating to the image object.

10. The image processing device according to claim 8, wherein the image processing device is a computer.

13. The feature sequence extraction unit extracts feature sequences related to any one of a plurality of image object regions that are related to a common image object and are related to a stationary image object, thereby extracting feature sequences related to the other image object regions.

13. The image processing device according to claim 12.

14. the image object region extraction unit acquires, for each frame region, a class to which the image object related to the frame region belongs; The stillness determination unit determines whether the image object is still or not based on the class to which the image object belongs.

14. The image processing device according to claim 12 or 13.

15. The stillness determination unit determines that an image object is still when frame-specific regions relating to the common image object are located at a common position within a frame.

14. The image processing device according to claim 12 or 13.

16. The image object is a person, CNN performs image processing to recognize human behavior.

16. The image processing device according to claim 8,

17. The image object is a person or object, CNN performs image processing to recognize scenes from input video images.

17. The image processing device according to claim 8,

18. The predetermined image object is an image object that is a target of image processing using a CNN.

18. The image processing device according to claim 8, wherein the image processing device is a processor.

19. When a still image object included in one input image is included in another input video, The CNN excludes, from the target of the convolution process, a region in the feature space corresponding to an image object region relating to the same still object image object included in the other input image as a zero region; Among the image processing results by the CNN for the one input video, the image processing result for a still image object included in the one input video is set as the image processing result for the same still image object included in the other input image.

16. The image processing device according to claim 12,

20. CNN is 3DCNN 20. The image processing device according to claim 8, wherein the image processing device is a computer.

21. CNN is SBnet 21. The image processing device according to claim 8, wherein:

22. The feature sequence extraction unit extracts the feature sequence by any one of a convolutional neural network, a multilayer perceptron, a neural network other than a CNN or a multilayer perceptron, a principal component analysis, and a predefined feature calculation method.

22. The image processing device according to claim 8, wherein the image processing device is a computer.

23. An image processing method executed by an image processing device that executes image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image, an image object region extraction step of extracting only a part of a predetermined image object from the input image as an image object region; a feature sequence extraction step of extracting a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process. An image processing method comprising:

24. An image processing method executed by an image processing device that executes video processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video, the method comprising: an image object region extraction step of extracting an image object region that includes at least a part of a predetermined image object in the input video sequence and that is common to frames that include the image object; a feature sequence extraction step of extracting a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process. An image processing method comprising:

25. An image processing program executed by a computer that performs image processing using a CNN that receives as input a feature space configured by a feature array extracted from an input image, an image object region extraction step of extracting only a part of a predetermined image object from the input image as an image object region; a feature sequence extraction step of extracting a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process.

1. An image processing program comprising:

26. An image processing program executed by a computer that performs video processing using a CNN that receives as input a feature space configured by a feature array extracted from an input video, an image object region extraction step of extracting an image object region that includes at least a part of a predetermined image object in the input video sequence and that is common to frames that include the image object; a feature sequence extraction step of extracting a feature sequence for each image object region; CNN uses the following feature space: A region corresponding to an image object region is set as a non-zero region, and the feature array extracted for each image object region is assigned to the non-zero region, and the region is subjected to convolution processing; and For zero regions other than non-zero regions, sparse convolution is performed, excluding them from the convolution process.

1. An image processing program comprising:

Images