Image processing device and image processing method

Abstract

To provide a technology for enabling correlation operation capable of efficiently responding to a change in a photographing orientation.SOLUTION: An image processing method includes: applying, to a first photographed image, filter coefficients which are set in a first array based on first orientation information indicating a photographing orientation of the first photographed image to generate a first feature map; acquiring a template feature corresponding to an object on the basis of the first feature map; registering the template feature in an array based on the first orientation information; applying, to a second photographed image, filter coefficients which are set in a second array based on second orientation information indicating a photographing orientation of the second photographed image to generate a second feature map; performing correlation operation between the second feature map and a template feature which is set in an array based on the second orientation information from the registered template feature; and detecting the object from the second photographed image on the basis of a result of the correlation operation.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to image processing technology. [Background technology]

[0002] Hierarchical computation methods (pattern recognition methods based on deep learning technology) such as convolutional neural networks (hereafter abbreviated as CNN) have been attracting attention as pattern recognition methods that are robust against changes in the recognition target. For example, Non-Patent Document 1 discloses various application and implementation examples.

[0003] As an application example of CNN, an object tracking processing method that uses the cross-correlation between feature amounts calculated by CNN has been proposed (Non-Patent Document 2, etc.). On the other hand, dedicated processing devices have been proposed for high-speed processing of CNN, which requires high computational costs (Patent Document 1, etc.). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-74967 [Non-patent literature]

[0005] [Non-Patent Document 1] Yann LeCun, Koray Kavukvuoglu and Clement Farabet: Convolutional Networks and Applications in Vision, Proc. International Symposium on Circuits and Systems (ISCAS'10), IEEE, 2010, [Non-patent document 2] Luca Bertinetto, Jack Valmadre, Joao F. Henriques, Andrea Vedaldi, Philip HS Torr:Fully-Convolutional Siamese Networks for Object Tracking, ECCV 2016 Workshops Summary of the Invention [Problem to be solved by the invention]

[0006] In the tracking processing method disclosed in Non-Patent Document 2, CNN features of an object are given instead of CNN coefficients, and convolution processing is performed to calculate highly accurate cross-correlation values ​​between CNN features. By utilizing local cross-correlation values ​​between different frames in an image, this method can be applied to applications such as tracking a specific object in a video image.

[0007] On the other hand, in tracking processing, the orientation of the object within the angle of view may change significantly depending on the orientation of the image capture device (upright shooting, vertical shooting, reverse shooting). In such cases, by rotating and processing the input image according to the orientation of the image capture device, it is possible to continue tracking processing regardless of the orientation of the image capture device.

[0008] However, when the device executes the rotation process of the input image, the processing time increases, a large buffer memory is required for the processing, etc., which increases the processing cost in inexpensive systems. This invention provides a technology that enables correlation calculations that can efficiently accommodate changes in the shooting position. [Means for solving the problem]

[0009] According to one aspect of the present invention, a first image is captured based on first posture information indicating a posture of a first captured image. coefficient With arrays Set the filter coefficients and use the first coefficient array a first generation means for applying the set filter coefficient to the first captured image to generate a first feature map; A template feature corresponding to the object is obtained based on the first feature map, and the template feature is converted into a template based on the first pose information. 1. Feature A registration method for registering in an array, A second image based on second posture information indicating the posture of the second captured image. coefficient With arrays Set the filter coefficients and use the second coefficient array second generation means for applying the set filter coefficient to the second captured image to generate a second feature map; The registered template features are calculated based on the second pose information. Second feature With arrays Set the second feature array configured template features; the second feature map; A calculation means for performing a correlation calculation of a detection means for detecting the object from the second captured image based on the result of the correlation calculation; The present invention is characterized by comprising: [Effects of the Invention]

[0010] According to the configuration of the present invention, correlation calculation that can efficiently accommodate variations in the shooting posture is possible. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram showing an example of the hardware configuration of a correlation calculation unit 201. [Figure 2] FIG. 1 is a block diagram showing an example of the hardware configuration of an image processing apparatus. [Figure 3] FIG. 2 is a block diagram showing the operation of a processing unit 102. [Figure 4] FIG. 2 is a diagram showing the operation of CNN processed by the calculation processing unit 102 and the conversion processing unit 105. [Figure 5] 10 is a diagram illustrating the retrieval of template features from a feature map. [Figure 6] 10A and 10B are diagrams showing conventional template feature extraction and correlation calculation when the shooting posture changes. [Figure 7] 10A and 10B are diagrams showing extraction of template features and correlation calculation according to the first embodiment when the shooting posture changes. [Figure 8]FIG. 2 is a block diagram showing an example of the arrangement of a rotation processing unit 107. [Figure 9] FIG. 2 is a diagram showing an example of the configuration of a memory area in a buffer 103. [Figure 10] 10 is a flowchart showing the operation of the image processing device. [Figure 11] A diagram that schematically explains an example of a feature plane of a CNN when processing using a line buffer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] [First embodiment] An image processing device according to this embodiment applies filter coefficients set in a first array based on first orientation information indicating the orientation of a first captured image to the first captured image to generate a first feature map, acquires template features corresponding to an object based on the first feature map, and registers the template features in an array based on the first orientation information. The image processing device then applies filter coefficients set in a second array based on second orientation information indicating the orientation of a second captured image to the second captured image to generate a second feature map, performs a correlation calculation between the second feature map and the registered template features set in the array based on the second orientation information, and detects the object from the second captured image based on the result of the correlation calculation. An example of such an image processing device is described below.

[0014] First, an example of the hardware configuration of an image processing device according to this embodiment will be described using the block diagram in Fig. 2. Devices applicable to the image processing device according to this embodiment include an imaging device capable of capturing still images and moving images, a smartphone, and a tablet terminal device or personal computer equipped with the imaging device.

[0015] The image input unit 202 is an imaging unit that includes an optical system, a photoelectric conversion device such as a CCD (Charge-Coupled Device) or a CMOS (Complimentary Metal Oxide Semiconductor) sensor, a driver circuit that controls the operation of the optical system and the photoelectric conversion device, an A / D converter, an image processing circuit, etc. External light passes through the optical system and enters the photoelectric conversion device, which outputs an analog image signal in response to the incident light. The analog image signal is converted into a digital image signal by the A / D converter. The digital image signal is then converted into a captured image through processes such as demosaicing and color processing in the image processing circuit. In other words, the image input unit 202 acquires a captured image as an input image.

[0016] The acquisition unit 210 has a posture sensor that measures its own posture as the posture of the image processing device, and outputs posture information indicating the posture measured by the posture sensor. As an example in this embodiment, the acquisition unit 210 acquires and outputs posture information indicating which of four directions perpendicular to the optical axis of the image capture unit the posture measured by the posture sensor is in. That is, the acquisition unit 210 acquires information regarding the orientation of the image capture unit relative to the object (upright, held vertically, held backwards). That is, the orientation of the image capture unit corresponds to the direction in which the user holds the image processing device, and corresponds to upright shooting, vertical shooting, or backward shooting.

[0017] The correlation calculation unit 201 performs various processes such as a registration process for acquiring and registering template features based on an input image acquired by the image input unit 202, and a correlation calculation process based on the input image acquired by the image input unit 202 and the registered template features.

[0018] A CPU (Central Processing Unit) 203 executes various processes using computer programs and data stored in a ROM (Read Only Memory) 204 and a RAM (Random Access Memory) 205. As a result, the CPU 203 controls the operation of the entire image processing device, and also executes or controls various processes that will be described as being performed by the image processing device.

[0019] The ROM 204 stores setting data for the image processing device, computer programs and data related to the startup of the image processing device, computer programs and data related to the basic operation of the image processing device, etc. The ROM 204 also stores computer programs and data for causing the CPU 203 to execute or control various processes that will be described as being performed by the image processing device.

[0020] The RAM 205 has an area for storing an input image input from the image input unit 202, an area for storing posture information acquired by the acquisition unit 210, and an area for storing data output from the correlation calculation unit 201. The RAM 205 also has an area for storing computer programs and data loaded from the ROM 204, and a work area used by the CPU 203 when executing various processes. In this way, the RAM 205 can provide various areas as needed. The RAM 205 is configured, for example, by a large-capacity DRAM (Dynamic Access Memory) or the like.

[0021] A DMAC (Direct Memory Access Controller) 206 controls data transfer between the image input unit 202 or RAM 205 and the correlation calculation unit 201 .

[0022] The user interface unit 208 has a user interface such as a button, a switch, a touch panel, etc., which the user can operate to input various instructions (for example, instructions on the tracking target) to the CPU 203. The user interface unit 208 also has a display screen (such as a liquid crystal screen or a touch panel screen) for displaying the processing results (for example, the results of the tracking process) by this device.

[0023] The image input unit 202 , acquisition unit 210 , correlation calculation unit 201 , CPU 203 , ROM 204 , RAM 205 , DMAC 206 , and user interface unit 208 are all connected to a system bus 207 .

[0024] The correlation calculation unit 201 performs the above correlation calculation process in accordance with instructions from the CPU 203, and generates a detection map indicating the likelihood of the position of an object (hereinafter, for the purpose of explanation, assumed to be a tracking target object) in the input image. The detection map is stored in the RAM 205 by the CPU 203. The CPU 203 provides various applications using the results of the tracking process of the tracking target object based on the detection map stored in the RAM 205. For example, the tracking process results are fed back to the image input unit 202 and used for controlling the focus of the optical system for tracking the tracking target object. Note that the method of using the detection map generated based on the correlation calculation by the correlation calculation unit 201 is not limited to a specific method of use. Furthermore, the methods of using the various maps mentioned in the following explanation are not limited to the specific methods of use described below.

[0025] Next, an example of the hardware configuration of the correlation calculation unit 201 will be described using the block diagram in Fig. 1. The I / F unit 101 is an interface that can be accessed by the CPU 203 and the DMAC 206 via the system bus 207, and the correlation calculation unit 201 transmits and receives data to and from the outside via the I / F unit 101.

[0026] The buffer 103 stores a two-dimensional array (coefficient pattern) of weighting coefficients (CNN coefficients) for each layer (each hierarchy) of the CNN, and is a buffer (memory device) that can supply the coefficient pattern with low delay.

[0027] Furthermore, buffer 103 stores, as template features, a set of CNN features within a local region in the "two-dimensional array (feature map) of CNN features in the final layer of CNN" obtained by conversion processing unit 105. CPU 203 reads, via I / F unit 101, the "feature map of the final layer of CNN" obtained by conversion processing unit 105 and stored in buffer 104, and extracts, as template features, a set of CNN features within a local region in the read feature map. CPU 203 then rotates the extracted template features in accordance with the orientation information acquired by acquisition unit 210, and stores the rotated template features in buffer 103 via I / F unit 101.

[0028] The buffer 104 is a buffer (memory device) that can store, with low latency, the feature map obtained by the conversion processing unit 105. The buffers 103 and 104 can be implemented, for example, by high-speed memories or registers. Although the buffers 103 and 104 are shown as separate buffers in FIG. 1, they may each be separate memory areas within a single buffer.

[0029] When the rotation processing unit 107 acquires a coefficient pattern and orientation information from the CPU 203 via the I / F unit 101, it rotates the acquired coefficient pattern in accordance with the acquired orientation information and then supplies it to the arithmetic processing unit 102. For example, when the orientation information indicates "portrait holding" (i.e., when the input image acquired by the image input unit 202 is portrait orientation), the rotation processing unit 107 rotates the coefficient pattern by 90 degrees clockwise and then supplies it to the arithmetic processing unit 102.

[0030] On the other hand, when the rotation processing unit 107 acquires the template features and the orientation information from the CPU 203 via the I / F unit 101, the rotation processing unit 107 rotates the acquired template features in accordance with the acquired orientation information and then supplies them to the calculation processing unit 102. For example, when the orientation information indicates "portrait holding" (that is, when the input image acquired by the image input unit 202 is portrait), the rotation processing unit 107 rotates the template features by 90 degrees clockwise and then supplies them to the calculation processing unit 102.

[0031] The calculation processing unit 102 performs a convolution operation, and the conversion processing unit 105 performs a nonlinear conversion on the result of the convolution operation by the calculation processing unit 102. Note that the nonlinear conversion in the conversion processing unit 105 uses well-known activation processing such as ReLU (Rectified Linear Unit) or a sigmoid function. When ReLU is used, it can be realized by threshold processing, and when a sigmoid function is used, values ​​are converted using a lookup table or the like. The control unit 106 controls various operations in the correlation calculation unit 201.

[0032] Next, the operation of the CNN (processing for generating a feature map) performed by the calculation processing unit 102 and the transformation processing unit 105 will be described with reference to Fig. 4(a). In the CNN, a convolution operation 403 is performed between an input image 401 and a coefficient pattern 402, and a nonlinear transformation 404 is performed on the result of the convolution operation 403 to generate a feature map 405.

[0033] Here, if the kernel size (filter coefficient matrix) of the convolution operation is columnSize × rowSize, and the number of feature maps in the layer before the layer to be processed in the CNN is L, one feature map is calculated based on the convolution operation shown below.

[0034]

number

[0035] input(x,y): Reference pixel value at 2D coordinates (x, y) output(x,y): Calculation result in 2D coordinates (x,y) weight(column, row): CNN coefficient at coordinate (x+column, y+row) L: number of feature maps in the previous layer columnSize, rowSize: horizontal and vertical size of the 2D convolution kernel Generally, in CNN computation, multiple convolution kernels are scanned pixel by pixel in the input image, and a product-sum operation is repeated according to the above formula, and the final product-sum operation result is subjected to nonlinear transformation (activation processing) to calculate a feature map. That is, pixel data for one feature map is generated by multiple spatial filter operations and a nonlinear operation on the sum of those operations. In this embodiment, the CNN coefficients correspond to the spatial filter coefficients. In practice, multiple feature maps are generated for each layer.

[0036] The calculation processing unit 102 has a multiplier and an accumulator, and executes the convolution calculation shown in the above formula using the multiplier and the accumulator. The transformation processing unit 105 then performs nonlinear transformation on the result of the convolution calculation to generate a feature map. In a typical CNN, the above process is repeated as many times as the number of feature maps to be generated.

[0037] Next, the acquisition of template features from feature maps will be described with reference to FIG. 5. FIG. 5 illustrates a case where three feature maps 501 are obtained from the final layer of the CNN. In this case, the CPU 203 extracts a feature group (CNN feature group) within a region (spatial small region) having a size of 3 pixels x 3 pixels from each of the three feature maps 501 as a template feature 502. In this case, the data size of the template feature 502 is 9. The position of the "spatial small region" is the position of a pre-specified target. For example, in the case of object tracking processing, the template feature is a feature amount of the tracked target. The position of the tracked target can be determined by utilizing the correlation between such template features and feature maps (correlation map). In other words, a position in the input image showing a high correlation can be determined as the position of the tracked target in the input image.

[0038] Next, the operation of the CNN (various arithmetic processes including correlation calculations using template features) processed by the arithmetic processing unit 102 and the conversion processing unit 105 will be described with reference to FIG. 4(b). In the CNN, a convolution operation 408 is performed between an input image 406 and a coefficient pattern 407, and a nonlinear transformation 409 is performed on the result of the convolution operation 408 to generate a feature map 410. Here, it is assumed that the feature map 410 consists of three feature maps, and the registered template features 411 are the three template features 502 in FIG. 5. Next, the CNN performs a convolution operation 412 between the feature map 410 (three feature maps) and the template features 411 (three template features 502), thereby calculating the correlation between the feature map 410 and the template features 411. By repeatedly performing the convolution operation 412 within the feature map, three types of correlation maps are calculated from the three feature maps as correlation maps 413, in the case of FIG. 5. The correlation calculation here operates in the same way as a so-called depth-wise CNN calculation process in which the input feature map and the output map are linked one-to-one (L = 1 in the above equation). The process is the Siam correlation calculation method described in Non-Patent Document 2, etc. Next, in the CNN, a convolution operation 415 is performed between the correlation map 413 and the coefficient pattern 414, and a non-linear transformation 416 is performed on the result of the convolution operation 415 to generate a feature map (detection map) 417. The feature map 417 is a single feature map. By performing CNN processing (convolution operation and non-linear transformation) on the correlation map 413, it is possible to clarify the correlation and obtain a detection map that enables more stable object detection. The value of each element (element value) in the detection map represents the likelihood (probability) that the element is an element constituting the object corresponding to the template feature. Therefore, the position where the element value peaks in the detection map can be determined to be the position of the object corresponding to the template feature.

[0039] 4(b) for each frame of a video or for each of multiple still images captured periodically or irregularly, a map can be generated that enables the detection of the position of an object corresponding to the template feature for each image. In other words, a map can be generated that enables tracking of a specific object across multiple images.

[0040] Next, the operation of the image processing apparatus according to this embodiment will be described with reference to the flowchart of Fig. 10. In step S1001, the CPU 203 performs various initialization processes required for the operation of the correlation calculation unit 201.

[0041] In step S1002, the CPU 203 reads out various operating parameters necessary for the operation of the correlation calculation unit 201 from the ROM 204 and stores them in the RAM 205. Note that the source of the operating parameters is not limited to the ROM 204.

[0042] In step S1003, CPU 203 determines whether to generate template features and store them in buffer 103. For example, when a new template feature is to be registered in buffer 103, CPU 203 determines that "template features are to be generated and stored in buffer 103." Also, for example, when a template feature stored in buffer 103 is to be updated with a new template feature, CPU 203 determines that "template features are to be generated and stored in buffer 103."

[0043] If it is determined that "template features are generated and stored in buffer 103" as a result of this determination, the process proceeds to step S1004. On the other hand, if it is not determined that "template features are generated and stored in buffer 103", the process proceeds to step S1008. In step S1004, CPU 203 acquires orientation information indicating the orientation of the imaging unit acquired by acquisition unit 210.

[0044] In step S1005, the CPU 203 acquires the coefficient pattern stored in the buffer 103 via the I / F unit 101. Then, the CPU 203 controls the DMAC 206 to output the acquired coefficient pattern, the orientation information acquired in step S1004, and the input image acquired by the image input unit 202 (the captured image captured in the capturing orientation indicated by the orientation information) to the correlation calculation unit 201. The input image input to the correlation calculation unit 201 via the I / F unit 101 is stored in the buffer 104. The rotation processing unit 107 rotates the coefficient pattern acquired from the CPU 203 via the I / F unit 101 in accordance with the orientation information acquired from the CPU 203 via the I / F unit 101. For example, if the orientation information indicates that the image capture unit is held "vertically" (that is, if the input image acquired by the image input unit 202 is captured vertically), the coefficient pattern is rotated 90 degrees clockwise. The arithmetic processing unit 102 performs a convolution operation on the input image stored in the buffer 104 and the coefficient pattern rotated by the rotation processing unit 107. The transformation processing unit 105 performs nonlinear transformation on the result of the convolution operation by the arithmetic processing unit 102 to generate a feature map (first generation). The generated feature map is stored in the buffer 104. Thereafter, the arithmetic processing unit 102 performs a convolution operation on the feature map stored in the buffer 104 (the feature map corresponding to the previous layer) and the coefficient pattern rotated by the rotation processing unit 107, and the transformation processing unit 105 performs a process of nonlinearly transforming the result of the convolution operation by the arithmetic processing unit 102 to generate a feature map and storing the generated feature map in the buffer 104, for each layer toward the final layer of the CNN. In this way, the feature maps for each layer of the CNN are stored in the buffer 104. The CPU 203 then acquires the feature map for the final layer of the CNN stored in the buffer 104 via the I / F unit 101. Then, a group of CNN features within a local region corresponding to the target object in the acquired feature map is acquired as template features.

[0045] In step S1006, CPU 203 rotates the template features acquired in step S1005 in accordance with the orientation information acquired in step S1004 (performs a rotation of the template features that is the reverse of the rotation of the coefficient pattern by rotation processing unit 107). For example, if the orientation information indicates "portrait holding" (i.e., the input image acquired by image input unit 202 is portrait orientation), CPU 203 rotates the template features acquired in step S1005 by 90 degrees counterclockwise. In step S1007, CPU 203 stores the template features that were reverse-rotated in step S1006 in buffer 103 via I / F unit 101.

[0046] In step S1008, the CPU 203 acquires the orientation information acquired by the acquisition unit 210. Here, the orientation information acquired in step S1008 is information indicating the orientation of the imaging unit measured by the orientation sensor at a measurement timing different from the measurement timing corresponding to the orientation information acquired in step S1004.

[0047] In step S1009, the CPU 203 acquires the coefficient pattern and template feature stored in the buffer 103 via the I / F unit 101. Then, the CPU 203 controls the DMAC 206 to output the acquired coefficient pattern and template feature, the orientation information acquired in step S1008, and the input image acquired by the image input unit 202 (the captured image captured at the capturing orientation indicated by the orientation information) to the correlation calculation unit 201. The input image input to the correlation calculation unit 201 via the I / F unit 101 is stored in the buffer 104. The rotation processing unit 107 rotates the coefficient pattern acquired from the CPU 203 via the I / F unit 101 in accordance with the orientation information acquired from the CPU 203 via the I / F unit 101. The calculation processing unit 102 performs a convolution operation on the input image stored in the buffer 104 and the coefficient pattern rotated by the rotation processing unit 107. The transformation processing unit 105 performs nonlinear transformation on the result of the convolution operation performed by the arithmetic processing unit 102 to generate a feature map (second generation) and stores the generated feature map in the buffer 104. Thereafter, the arithmetic processing unit 102 performs a convolution operation on the feature map stored in the buffer 104 (the feature map corresponding to the previous layer) and the coefficient pattern rotated by the rotation processing unit 107, and the transformation processing unit 105 performs a nonlinear transformation on the result of the convolution operation performed by the arithmetic processing unit 102 to generate a feature map and store the generated feature map in the buffer 104, for each layer toward the final layer of the CNN, thereby storing the feature maps in each layer of the CNN in the buffer 104. The rotation processing unit 107 also rotates the template feature acquired from the CPU 203 via the I / F unit 101 in accordance with the orientation information acquired from the CPU 203 via the I / F unit 101. For example, if the orientation information indicates "portrait holding" (i.e., if the input image acquired by the image input unit 202 is portrait orientation), the rotation processing unit 107 rotates the template feature by 90 degrees clockwise. Then, the calculation processing unit 102 performs a convolution operation between the rotated template feature and the "feature map in the final layer of CNN" stored in the buffer 104, thereby obtaining a correlation map indicating the correlation between the feature map and the template feature.Next, the correlation map is subjected to the same processing as the above-mentioned "hierarchical convolution operation and nonlinear transformation of the input image by the calculation processing unit 102 and the transformation processing unit 105," to obtain a map obtained from the final layer of the CNN as a detection map.

[0048] In step S1010 , the CPU 203 acquires the detection map calculated by the correlation calculation unit 201 from the correlation calculation unit 201 via the I / F unit 101 , and stores the acquired detection map in the RAM 205 .

[0049] In step S1011, CPU 203 determines whether a processing termination condition has been met. For example, if the user operates user interface unit 208 to input a processing termination instruction, it is determined that the processing termination condition has been met. Also, for example, if the elapsed time since the start of processing according to the flowchart of Fig. 10 reaches a specified time, or if the number of repetitions of steps S1003 to S1010 reaches a specified number, it is determined that the processing termination condition has been met. In this way, the processing termination condition is not limited to a specific condition.

[0050] If the end condition of the process is satisfied as a result of such determination, the process according to the flowchart of Fig. 10 ends. On the other hand, if the end condition of the process is not satisfied, the process proceeds to step S1003.

[0051] Next, the features of this embodiment will be described. First, conventional extraction of template features and correlation calculation when the shooting posture changes will be described with reference to FIG. 6. An input image 601 is a photographed image of an object 602 taken vertically. Below, a case will be described in which template features are generated from such an input image 601 and registered. Note that even in the case of vertical shooting, the input image output by the image input unit 202 is assumed to be a horizontally long raster image (the object is rotated and photographed within the input image).

[0052] In this case, first, the input image 601 is rotated according to the shooting orientation (vertical shooting in this case), and then CNN processing is performed on the rotated input image 603 using the coefficient pattern 604 to acquire a feature map. Because the coefficient pattern 604 is obtained by learning from an upright object, the input image 601 must be rotated in order to extract a feature map similar to that obtained when the shooting orientation is upright. Then, from the acquired feature map, features of the region at the position of the object 606 are extracted as template feature 605 (although the position is intentionally shifted in FIG. 6 for the sake of explanation, the actual position of the template feature 605 is the position of the object 606). In this case, the template feature 605 is registered for subsequent correlation calculation.

[0053] After the template feature 605 is registered, when a new vertically shot input image 607 is input, the input image 607 is rotated according to the shooting posture of the input image 607, and a feature map is obtained by performing CNN processing using the coefficient pattern on the rotated input image 608. Then, a correlation map 609 is generated from the obtained feature map and the previously registered template feature 605, and a detection map 610 is generated from the correlation map 609.

[0054] Image rotation is not necessary when the input image 612 is taken in an upright position relative to the object 606. In this case, a feature map is obtained by performing CNN processing on the input image 612 using a coefficient pattern, a correlation map 614 is generated from the feature map and the previously registered template feature 605, and a detection map 615 is generated from the correlation map 614.

[0055] Conventionally, input images are rotated according to the shooting position in this way before processing. However, because input images generally have a large number of pixels, the processing time required for input image rotation increases, and the buffer memory required for input image rotation processing also increases, resulting in increased processing costs. In general, vertical / horizontal image conversion requires frame memory, which can be a major issue with inexpensive imaging devices, for example.

[0056] In contrast, in this embodiment, template feature extraction and correlation calculation when the shooting posture changes will be described with reference to FIG. 7 . In the case of an input image 701 obtained by shooting an object vertically, instead of rotating the input image 701, the coefficient pattern 702 is rotated 90 degrees clockwise, and a feature map is generated by performing CNN processing using the rotated coefficient map 702 and the input image 701. A group of CNN features within the region of the object in the feature map is acquired as template feature 704a. This template feature 704a is then rotated according to the shooting posture at the time of registration and registered. Specifically, template feature 704a is rotated 90 degrees counterclockwise (90 degrees in the opposite direction to the rotation direction of coefficient map 702) to generate template feature 704b. In this case, as described below, during correlation calculation, it is possible to always perform correlation calculation using the registered template 704b regardless of the shooting posture.

[0057] Assume that a new input image 705 is obtained in which the shooting orientation at the time of correlation calculation was portrait. In this case, coefficient map 706 is rotated 90 degrees clockwise, and feature map 709 is obtained by performing CNN processing using the rotated coefficient map 706 and input image 705. Then, correlation calculation is performed between the acquired feature map 709 and template feature 708, which is the previously registered template feature 704b rotated 90 degrees clockwise, to generate correlation map 711. Then, CNN processing is performed on correlation map 711 (using coefficient map 710 rotated 90 degrees clockwise) to generate a detection map.

[0058] On the other hand, suppose that an input image 712 is obtained in an upright shooting position during the correlation calculation. In this case, a feature map 716 is obtained by performing CNN processing using an unrotated coefficient pattern 713 and the input image 712. Then, a correlation calculation is performed between the obtained feature map 716 and the previously registered template feature 704b to generate a correlation map 718. Then, a CNN processing is performed on the correlation map 718 (using an unrotated coefficient pattern 717) to generate a detection map.

[0059] Also, assume that input image 719 has been obtained, which was taken in a vertical orientation, opposite to that of input image 701, during the correlation calculation. In this case, feature map 723 is obtained by performing CNN processing using coefficient pattern 720 rotated 90 degrees counterclockwise and input image 719. Then, correlation calculation is performed between the obtained feature map 723 and template feature 722, which is the previously registered template feature 704b rotated 90 degrees counterclockwise, to generate correlation map 725. Then, CNN processing (using coefficient pattern 724 rotated 90 degrees counterclockwise) is performed on correlation map 725 to generate a detection map.

[0060] In this way, by rotating and registering the template features according to the shooting posture at the time of registration of the template features, it is possible to use a hardware-based coefficient rotation mechanism to calculate an appropriate correlation map regardless of the shooting posture.

[0061] Next, we will explain the rotation of coefficient patterns and template features performed by the rotation processing unit 107. The rotation processing performed by the rotation processing unit 107 is a rotation processing of two-dimensional arrays such as coefficient patterns and template features, and is performed by changing the order in which elements are read from the two-dimensional arrays.

[0062] As shown in FIG. 3, a 3×3 coefficient pattern acquired from the CPU 203 via the I / F unit 101 is stored in a buffer 303 , and a 3×3 unit feature map read from the buffer 104 is stored in a buffer 304 .

[0063] The buffer 303 for storing coefficient patterns has nine registers (C 0,0 , C 0,1 , C 0,2 , C 1,0 , C 1,1 , C 1,2 , C 2,0 , C 2,1 , C 2,2 Each of the nine registers stores the CNN coefficient at the corresponding position in the 3x3 coefficient pattern. 0,0The leftmost CNN coefficient in the top row of the coefficient pattern is stored in C. 0,1 The center CNN coefficient in the top row of the coefficient pattern is stored in C. 0,2 The rightmost CNN coefficient in the top row of the coefficient pattern is stored in register C. 1,0 The leftmost CNN coefficient in the center row of the coefficient pattern is stored in C. 1,1 The central CNN coefficient in the central row of the coefficient pattern is stored in C 1,2 The rightmost CNN coefficient in the center row of the coefficient pattern is stored in register C. 2,0 The leftmost CNN coefficient in the bottom row of the coefficient pattern is stored in C. 2,1 The center CNN coefficient in the bottom row of the coefficient pattern is stored in C. 2,2 stores the CNN coefficient at the right end of the bottom row in the coefficient pattern.

[0064] The buffer 304 for storing the feature map has nine registers (F 0,0 , F 0,1 , F 0,2 , F 1,0 , F 1,1 , F 1,2 , F 2,0 , F 2,1 , F 2,2 ) Each of the nine registers stores the CNN feature at the corresponding position in the 3x3 feature map. That is, register F 0,0 The leftmost CNN feature in the top row of the feature map is stored in F. 0,1 The CNN feature in the center of the top row in the feature map is stored in F. 0,2 The rightmost CNN feature in the top row of the feature map is stored in register F. 1,0 The CNN feature at the left end of the center row in the feature map is stored in F. 1,1 The central CNN feature of the central row in the feature map is stored in F. 1,2 The CNN feature at the right end of the center row in the feature map is stored in register F. 2,0 The leftmost CNN feature in the bottom row of the feature map is stored in F. 2,1The center CNN feature in the bottom row of the feature map is stored in F. 2,2 stores the CNN feature at the right end of the bottom row in the feature map.

[0065] The arithmetic processing unit 102 (multiplier 301) performs a product-sum operation on the CNN coefficient data string output from the rotation processing unit 107 based on the coefficient pattern stored in the buffer 303 and the feature map stored in the buffer 304. The arithmetic processing unit 102 (accumulator 302) performs cumulative addition of the results of the product-sum operation, thereby realizing the convolution operation according to the above formula.

[0066] As will be described later, the rotation processing unit 107 reads out each CNN coefficient of the coefficient pattern stored in the buffer 303 in an order determined according to the posture information, and outputs a one-dimensional data string (CNN coefficient data string) in which the read CNN coefficients are arranged in the read order. In other words, the nth (1≦n≦9) CNN coefficient from the beginning in the CNN coefficient data string is the nth CNN coefficient read out from the coefficient pattern stored in the buffer 303.

[0067] The multiplier 301 also multiplies the nine registers in the buffer 304 in the order of raster data (F 0,0 , F 0,1 , F 0,2 , F 1,0 , F 1,1 , F 1,2 , F 2,0 , F 2,1 , F 2,2 The CNN features registered in the referenced register are read out.

[0068] The multiplier 301 then calculates the multiplication result of the nth CNN coefficient (1≦n≦9) in the CNN coefficient data string by the nth CNN feature read from the buffer 304, and accumulates the nine multiplication results to complete one spatial filter calculation. In practice, the cumulative sum of data corresponding to multiple coefficients and multiple feature planes is calculated according to the CNN connection relationships. In other words, the spatial filter calculation for multiple feature planes in the previous layer is executed on the feature plane. Therefore, the number of spatial filters is the number of feature planes in all layers multiplied by the number of feature planes to be processed.

[0069] In such a case, a description will be given of a case where the coefficient pattern is rotated by the rotation processing unit 107. As described above, since the coefficient pattern and the template feature are both two-dimensional arrays, the following description can also be applied to the rotation of the template feature.

[0070] An example of the configuration of the rotation processing unit 107 will be described with reference to the block diagram in FIG. 8. The coefficient selection unit 802 obtains the readout order registered in the coefficient selection table 803 in association with the orientation information obtained from the CPU 203 via the I / F unit 101. The coefficient selection table 803 stores, for example, four types of readout orders according to the shooting orientation. For example, if the orientation information indicates "portrait holding", the coefficient pattern 801 is rotated 90 degrees clockwise. Therefore, the coefficient selection table 803 stores the orientation information indicating "portrait holding" and the readout order "C 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 " and are registered in correspondence with each other.

[0071] The coefficient selection unit 802 then refers to the nine registers of the buffer 303 in the order of reading. For example, if the orientation information indicates that the camera is held vertically, the coefficient pattern 801 is rotated 90 degrees clockwise. 2,0 , C1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 The coefficient selection unit 802 then reads out the CNN coefficients stored in the register in the order they were referenced, and outputs a one-dimensional data string (CNN coefficient data string) in which the read-out CNN coefficients are arranged in the order they were read out. 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 " in the order of ", the CNN coefficients stored in the referenced register are read, and the 3x3 pattern obtained by arranging the read CNN coefficients in the raster data order in the read order is a coefficient pattern obtained by rotating the coefficient pattern 801 held in the buffer 303 by 90 degrees clockwise. 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 ", and the CNN coefficients stored in the referenced registers are read out. The one-dimensional data string (CNN coefficient data string) obtained by arranging the read CNN coefficients in the read order is a coefficient pattern obtained by rotating the coefficient pattern 801 held in the buffer 303 by 90 degrees clockwise. In other words, the only difference is the number of dimensions of the data array.

[0072] Then, the coefficient selection table 803 outputs the CNN coefficient data string obtained in this manner to the multiplier 301. Note that the generation of the CNN coefficient data string may be performed by the multiplier 301.

[0073] When buffer 303 is configured with a register, coefficient selection unit 802 can be configured with a selector that selects its output in sequence. In this way, rotation processing unit 107 can be configured using coefficient selection table 803 with a relatively small amount of data and a coefficient selection selector, and the increase in cost required for rotation processing unit 107 is slight. Furthermore, when supporting multiple types of kernel sizes, it is only necessary to change the information stored in coefficient selection table 803 and the configuration of coefficient selection unit 802 depending on the type of kernel.

[0074] Next, an example of the configuration of memory areas in buffer 103 will be described with reference to Fig. 9. Fig. 9 shows an example of the configuration of memory areas in buffer 103 in the example of Fig. 4, where memory area 901 stores coefficient pattern 402 / coefficient pattern 407, and memory area 902 stores coefficient pattern 414. Furthermore, memory area 903 stores template feature 411.

[0075] The coefficient patterns stored in memory area 901 and memory area 902 are transferred to buffer 303 in specified units (3x3 units in the example of FIG. 8) under the control of control unit 106. Also, the template features stored in memory area 903 are transferred to buffer 303 in specified units (3x3 units in the example of FIG. 8) under the control of control unit 106.

[0076] This configuration makes it possible to rotate coefficient patterns and template features. For example, when template feature 704b, which is obtained by rotating template feature 704a in FIG. 7, is stored in buffer 103, elements 6, 3, 0, 7, 4, 1, 8, 5, and 2, which are arranged in raster data order in template feature 704a, are read in the order of elements 0, 1, 2, 3, 4, 5, 6, 7, and 8 to form 3x3 template feature 704b in which elements 0, 1, 2, 3, 4, 5, 6, 7, and 8 are arranged in raster data order, and template feature 704b is stored in buffer 103. When performing correlation calculations on consecutive images, even if the shooting posture changes, processing corresponding to the shooting posture can be performed simply by setting the posture information.

[0077] As described above, according to this embodiment, template features are rotated to an upright orientation and stored according to the shooting posture when the template features are generated, so that correlation calculations can be efficiently performed according to the shooting posture during correlation calculations. In other words, regardless of the shooting posture, rotation of the captured image is not required during correlation calculations, and template features can be processed using a coefficient map rotation mechanism. In other words, CNN processing and correlation calculations can be performed using common hardware in response to posture changes.

[0078] As a result, when the image processing device according to this embodiment is applied to the tracking processing of an object, the object can be tracked smoothly without any special processing even if the shooting posture changes (for example, when taking a picture while changing the way the image processing device is held).

[0079] In this embodiment, the image processing device has been described as having the image input unit 202, but this is not limiting, and the image input unit 202 may be an external device. For example, the image processing device may be configured to perform the above-described various processes based on an input image received from the external image input unit 202 via a wired or wireless network.

[0080] Similarly, the image processing device has been described as having the acquisition unit 210 that acquires posture information, but this is not limiting, and the acquisition unit 210 may be an external device. For example, the image processing device may perform the various processes described above using posture information received from the external acquisition unit 210 via a wired or wireless network.

[0081] In addition, in this embodiment, a posture sensor is used to measure the shooting posture, but the method for acquiring the shooting posture is not limited to a specific method. For example, the shooting posture may be estimated from multiple captured images, the shooting posture may be measured using another type of sensor, or the shooting posture may be acquired by combining several methods. Furthermore, the user may input posture information by operating the user interface unit 208.

[0082] Furthermore, in this embodiment, a CNN is used for the hierarchical spatial filter calculation, but the hierarchical spatial filter calculation is not limited to a specific method, and may be performed using, for example, other types of hierarchical neural networks.

[0083] Furthermore, the various processes described as being performed by the correlation calculation unit 201 may be executed by a processor such as the CPU 203, a graphics processing unit (GPU), or a digital signal processing unit (DSP).

[0084] [Second embodiment] In this embodiment, differences from the first embodiment will be described, and unless otherwise specified below, the present embodiment will be considered to be the same as the first embodiment. In the first embodiment, feature maps for each layer are stored in the buffer 104. In this case, however, the capacity required for the buffer 104 increases. The increase in capacity becomes a problem, particularly when the number of feature maps for each layer is large. Therefore, a configuration in which processing is performed across multiple layers for each small region, rather than processing for each layer, may be adopted. In this case, processing for each layer is performed in specified units (e.g., line units). As a result, for example, consider storing the feature map 410 and correlation map 413, which are intermediate results of the hierarchical processing illustrated in FIG. 4. In this case, when storing data in the buffer 104, a memory area corresponding to (the number of lines required for spatial filter calculation × the number of maps) can be secured in the buffer 104 for processing. In other words, the buffer 104 is used as a line-unit circular buffer to process a hierarchical network.

[0085] FIG. 11 is a diagram illustrating an example of a CNN feature plane when processing using a line buffer. 1101 denotes a circular line buffer for the input image 406, 1102 denotes a circular line buffer for the feature map 410, and 1103 denotes a circular line buffer for the correlation map 413. Note that FIG. 11 shows an example in which the spatial filter size is 3 × 3. The memory for holding the detection map 417 of the final layer is configured as a frame buffer that holds all results. For example, for the feature map 410, after reference data that can be filtered is accumulated in the line buffer 1101 of the input image 406, a convolution operation for one line is performed. The feature map and correlation map are calculated by cycling through the circular line buffers 1101 to 1103 line by line. The calculation processing unit 102 processes the network across layers while switching the feature plane to be processed line by line.

[0086] The control for processing in the line buffer is managed by, for example, the control unit 106, which sequentially performs convolution operations 408, 412, and 415 for each line of the input image in step S1009. Such processing can be realized, for example, by the configuration disclosed in Japanese Patent No. 5184824.

[0087] During convolution calculation, the rotation direction of the CNN coefficients is specified to the rotation processing unit 107 according to the orientation information. In this embodiment, template features are always registered in an upright state regardless of the imaging orientation at the time of registration, so during correlation calculation, the same rotation can be specified for all layers according to the imaging orientation. In other words, even when correlation calculation is performed across multiple layers, efficient processing can be achieved without performing special processing between layers in response to variations in imaging orientation.

[0088] [Third embodiment] In the first and second embodiments, the rotation of the two-dimensional array is performed by changing the read order of the elements of the two-dimensional array, but the rotation of the two-dimensional array may be performed by other methods, such as by using a hardware rotation mechanism.

[0089] Alternatively, two-dimensional arrays rotated at multiple angles may be created in advance and stored in RAM 205, and one of the multiple created two-dimensional arrays may be selected and used according to the orientation information. For example, a two-dimensional array rotated 90 degrees clockwise, a two-dimensional array rotated 90 degrees counterclockwise, and a two-dimensional array rotated 180 degrees clockwise / counterclockwise may be created in advance. Then, for example, if the orientation information indicates clockwise portrait shooting, the two-dimensional array rotated 90 degrees clockwise is selected, and if the orientation information indicates counterclockwise portrait shooting, the two-dimensional array rotated 90 degrees counterclockwise is selected.

[0090] In the first and second embodiments, the template feature is a map of CNN features extracted from a local region in the feature map, but the present invention is not limited to this and may be any map of CNN features obtained based on the feature map. For example, the template feature may be a map obtained by processing CNN features extracted from a partial region in the feature map.

[0091] In the first and second embodiments, the template features are registered by converting them into upright template features according to the shooting posture, but they may be converted into a predetermined specific orientation regardless of the shooting posture and registered. In this case, however, it is necessary to control the rotation of the template features when calculating the correlation map 413, for example, so that it differs from that of other convolution operations.

[0092] Furthermore, in the first and second embodiments, the case where the image when the template features are acquired and the image that is the target of the correlation calculation are different has been described, but the image when the template features are acquired and the image that is the target of the correlation calculation may be the same.

[0093] 1, the rotation processing unit 107, the calculation processing unit 102, the conversion processing unit 105, and one or more of the functional units included therein may be implemented as hardware or software. In the latter case, the software is executed by the control unit 106 or the CPU 203, thereby realizing the functions of the corresponding functional units.

[0094] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0095] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0096] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0097] The disclosure of this specification includes the following image processing device, image processing method, and computer program.

[0098] (Item 1) a first generation means for applying filter coefficients set in a first array based on first orientation information indicating an orientation of a first captured image to the first captured image to generate a first feature map; a registration means for acquiring template features corresponding to the object based on the first feature map and registering the template features in an arrangement based on the first orientation information; a second generation means for applying filter coefficients set in a second array based on second orientation information indicating an orientation of a second captured image to the second captured image to generate a second feature map; a calculation means for performing a correlation calculation between the second feature map and the registered template features set in an arrangement based on the second posture information; a detection means for detecting the object from the second captured image based on the result of the correlation calculation; An image processing device comprising:

[0099] (Item 2) the first generation means rotates and sets the filter coefficients in accordance with the first attitude information; 2. The image processing device according to item 1, wherein the registration means registers the acquired template feature by rotating it in a direction opposite to the rotation of the filter coefficients by the first generation means.

[0100] (Item 3) the second generation means rotates and sets the filter coefficients in accordance with the second attitude information; 3. The image processing device according to item 2, wherein the calculation means performs a correlation calculation between the second feature map and a template feature obtained by rotating the registered template feature in accordance with the rotation of the filter coefficients by the second generation means.

[0101] (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the registration means acquires features within the region of the object in the first feature map as template features.

[0102] (Item 5) the first generation means generates the first feature map based on a convolution operation between the filter coefficients set in the first array and the first captured image; 5. The image processing device according to any one of items 1 to 4, wherein the second generation means generates the second feature map based on a convolution operation between the filter coefficients set in the second array and the second captured image.

[0103] (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the calculation means performs the correlation calculation by a convolution calculation of the second feature map and a template feature set in an array based on the second posture information.

[0104] (Item 7) 7. The image processing device according to item 5 or 6, wherein the convolution operation is performed using a hierarchical neural network.

[0105] (Item 8) 8. The image processing device according to item 7, wherein the convolution operation is performed in a specified unit for each layer of the hierarchical neural network.

[0106] (Item 9) The image processing device described in any one of items 1 to 8, characterized in that the detection means generates a detection map indicating the likelihood of the position of the object in the second captured image based on the result of the correlation calculation.

[0107] (Item 10) moreover, 10. The image processing device according to item 9, further comprising a means for controlling photography in accordance with the detection map.

[0108] (Item 11) moreover, 11. The image processing device according to any one of items 1 to 10, further comprising a means for acquiring images captured as the first captured image and the second captured image.

[0109] (Item 12) moreover, 12. The image processing device according to any one of items 1 to 11, further comprising: means for acquiring the first attitude information and the second attitude information.

[0110] (Item 13) An image processing method performed by an image processing device, a first generation step in which a first generation means of the image processing device applies filter coefficients set in a first array based on first posture information indicating a posture of a first captured image to the first captured image to generate a first feature map; a registration step in which a registration means of the image processing device acquires template features corresponding to the object based on the first feature map and registers the template features in an arrangement based on the first orientation information; a second generation step in which second generation means of the image processing device applies filter coefficients set in a second array based on second posture information indicating a posture of a second captured image to the second captured image to generate a second feature map; a calculation step in which a calculation means of the image processing device performs a correlation calculation between the second feature map and the registered template features set in an arrangement based on the second posture information; a detection step in which a detection means of the image processing device detects the object from the second captured image based on a result of the correlation calculation; An image processing method comprising:

[0111] (Item 14) A computer program for causing a computer to function as each of the means of the image processing device according to any one of items 1 to 12.

[0112] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0113] 101: I / F unit 102: Arithmetic processing unit 103: Buffer 104: Buffer 105: Conversion processing unit 106: Control unit 107: Rotation processing unit

Claims

1. a first generation means for setting filter coefficients using a first coefficient array based on first posture information indicating a posture of a first captured image, and applying the filter coefficients set in the first coefficient array to the first captured image to generate a first feature map; a registration means for acquiring a template feature corresponding to the object based on the first feature map, and registering the template feature in a first feature array based on the first orientation information; a second generation means for setting filter coefficients using a second coefficient array based on second posture information indicating a posture of a second captured image, and applying the filter coefficients set in the second coefficient array to the second captured image to generate a second feature map; a computing means for setting the registered template features in a second feature array based on the second posture information, and performing a correlation calculation between the template features set in the second feature array and the second feature map; a detection means for detecting the object from the second captured image based on the result of the correlation calculation; An image processing device comprising:

2. the first generation means rotates the filter coefficients in the first coefficient array in accordance with the first orientation information to set the filter coefficients in the first coefficient array; 2. The image processing device according to claim 1, wherein the registration means rotates the acquired template feature in an inverse manner to the rotation of the filter coefficients by the first generation means, and registers the acquired template feature in the first feature array.

3. the second generating means rotates the filter coefficients in the second coefficient array in accordance with the second orientation information to set the filter coefficients in the second coefficient array; 3. The image processing device according to claim 2, wherein the calculation means rotates the registered template feature in the second feature array in accordance with the rotation of the filter coefficients by the second generation means, and sets the registered template feature in the second feature array.

4. The image processing apparatus according to claim 1 , wherein the registration means acquires features within the region of the object in the first feature map as template features.

5. the first generation means generates the first feature map based on a convolution operation of the filter coefficients set in the first coefficient array and the first captured image; The image processing device according to claim 1 , wherein the second generating means generates the second feature map based on a convolution operation between the filter coefficients set in the second coefficient array and the second captured image.

6. 2. The image processing apparatus according to claim 1, wherein the calculation means performs the correlation calculation by a convolution operation of the second feature map and the template feature set in the second feature array.

7. 6. The image processing apparatus according to claim 5, wherein the convolution operation is performed using a hierarchical neural network.

8. 8. The image processing apparatus according to claim 7, wherein the convolution operation is performed in a predetermined unit for each layer of the hierarchical neural network.

9. 2. The image processing apparatus according to claim 1, wherein the detecting means generates a detection map indicating likelihood of the position of the object in the second captured image based on the result of the correlation calculation.

10. moreover, 10. The image processing apparatus according to claim 9, further comprising a control unit that controls photography in accordance with the detection map.

11. The image processing device described in Claim 10, characterized in that the control means performs control to track and photograph the object.

12. moreover, 2. The image processing apparatus according to claim 1, further comprising: means for acquiring images taken as the first photographed image and the second photographed image.

13. moreover, 2. The image processing apparatus according to claim 1, further comprising: means for acquiring the first orientation information and the second orientation information.

14. An image processing method performed by an image processing device, a first generation step in which a first generation means of the image processing device sets filter coefficients in a first coefficient array based on first posture information indicating a posture of a first captured image, and applies the filter coefficients set in the first coefficient array to the first captured image to generate a first feature map; a registration step in which a registration means of the image processing device acquires a template feature corresponding to the object based on the first feature map and registers the template feature in a first feature array based on the first orientation information; a second generation step in which second generation means of the image processing device sets filter coefficients in a second coefficient array based on second posture information indicating a posture of a second captured image, and applies the filter coefficients set in the second coefficient array to the second captured image to generate a second feature map; a calculation step in which a calculation means of the image processing device sets the registered template features in a second feature array based on the second posture information, and performs a correlation calculation between the template features set in the second feature array and the second feature map; a detection step in which a detection means of the image processing device detects the object from the second captured image based on a result of the correlation calculation; An image processing method comprising:

15. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images