Image processing device, image processing method

Abstract

This invention provides an image processing device, method, and program that enable correlation calculations capable of efficiently responding to variations in shooting posture. [Solution] The method involves applying filter coefficients set in a first array based on first pose information indicating the shooting posture of the first captured image to the first captured image to generate a first feature map, obtaining template features corresponding to the object based on the first feature map, and registering the template features in an array based on the first pose information. A second feature map is generated by applying filter coefficients set in a second array based on second pose information indicating the shooting posture of the second captured image to the second captured image, performing a correlation calculation between the second feature map and the template features set in an array based on the second pose information, and detecting the object from the second captured image based on the result of the correlation calculation.

Description

Technical Field

[0001] The present invention relates to image processing technology.

Background Art

[0002] Hierarchical operation methods (pattern recognition methods based on deep learning technology) represented by convolutional neural networks (hereinafter abbreviated as CNN) have attracted attention as pattern recognition methods that are robust against variations in recognition targets. For example, various application examples and implementation examples are disclosed in Non-Patent Document 1.

[0003] As an application example of CNN, an object tracking processing method that uses the cross-correlation between feature amounts calculated by CNN has been proposed (Non-Patent Document 2, etc.). On the other hand, dedicated processing devices for processing CNN, which require high computational costs, at high speed have been proposed (Patent Document 1, etc.).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Non-Patent Documents

[0005]

Non-Patent Document 1

Non-Patent Document 2

[0006] The tracking method disclosed in Non-Patent Document 2 calculates highly accurate cross-correlation values ​​between CNN features by performing a convolution operation using CNN features of the target object instead of CNN coefficients. By utilizing local cross-correlation values ​​between different frames in an image, it can be applied to applications such as tracking a specific object in a moving image.

[0007] On the other hand, during tracking, the orientation of the object within the field of view can change significantly depending on the orientation of the camera (upright, vertical, or reverse). In such cases, the tracking process can be continued regardless of the orientation of the camera by rotating the input image according to the orientation of the camera.

[0008] However, if the device performs the rotation processing of the input image, the processing time increases and a large buffer memory is required for processing, which increases processing costs in low-cost systems. This invention provides a technology that enables correlation calculation that can efficiently respond to changes in the shooting posture. [Means for solving the problem]

[0009] One aspect of the present invention is characterized by comprising: a first generation means for generating a first feature map by applying filter coefficients set in a first array based on first posture information indicating the shooting posture of a first captured image to the first captured image; a registration means for acquiring template features corresponding to an object based on the first feature map and registering the template features in an array based on the first posture information; a second generation means for generating a second feature map by applying filter coefficients set in a second array based on second posture information indicating the shooting posture of a second captured image to the second captured image; a calculation means for performing a correlation calculation between the second feature map and the registered template features set in an array based on the second posture information; and a detection means for detecting the object from the second captured image based on the result of the correlation calculation. [Effects of the Invention]

[0010] According to the configuration of the present invention, correlation calculations that can efficiently respond to changes in the shooting posture are made possible. [Brief explanation of the drawing]

[0011] [Figure 1] A block diagram showing an example of the hardware configuration of the correlation calculation unit 201. [Figure 2] A block diagram showing an example of the hardware configuration of an image processing device. [Figure 3] A block diagram showing the operation of the arithmetic processing unit 102. [Figure 4] A diagram illustrating the operation of the CNN processed by the arithmetic processing unit 102 and the transformation processing unit 105. [Figure 5] A diagram illustrating the retrieval of template features from a feature map. [Figure 6] This figure illustrates the extraction and correlation calculation of conventional template features when the shooting posture changes. [Figure 7] A figure showing the extraction and correlation calculation of template features according to the first embodiment when the shooting posture changes. [Figure 8] A block diagram showing an example configuration of the rotation processing unit 107. [Figure 9]A diagram showing an example of the memory area configuration in buffer 103. [Figure 10] A flowchart illustrating the operation of an image processing device. [Figure 11] A schematic diagram illustrating an example of a CNN feature face when processing using a line buffer. [Modes for carrying out the invention]

[0012] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0013] [First Embodiment] The image processing apparatus according to this embodiment generates a first feature map by applying filter coefficients set in a first array based on first pose information indicating the shooting pose of the first captured image to the first captured image, obtains template features corresponding to the object based on the first feature map, and registers the template features in an array based on the first pose information. The image processing apparatus then generates a second feature map by applying filter coefficients set in a second array based on second pose information indicating the shooting pose of the second captured image to the second captured image, performs a correlation calculation between the second feature map and the template features set in an array based on the second pose information, and detects the object from the second captured image based on the result of the correlation calculation. An example of such an image processing apparatus will be described below.

[0014] First, an example of the hardware configuration of the image processing device according to this embodiment will be explained using the block diagram in Figure 2. The image processing device according to this embodiment can be an imaging device capable of capturing still images or moving images, a smartphone, a tablet terminal device or personal computer equipped with the imaging device, or other such devices.

[0015] The image input unit 202 is an imaging unit that includes an optical system, a photoelectric conversion device such as a CCD (Charge-Coupled Device) or CMOS (Complimentary Metal Oxide Semiconductor) sensor, a driver circuit for controlling the operation of the optical system and the photoelectric conversion device, an A / D converter, an image processing circuit, etc. External light passes through the optical system and enters the photoelectric conversion device, which outputs an analog image signal according to the incoming light, and the analog image signal is converted into a digital image signal by the A / D converter. The digital image signal is converted into a captured image through processing such as demosaicing and color processing in the image processing circuit. In other words, the image input unit 202 acquires a captured image as an input image.

[0016] The acquisition unit 210 has a posture sensor that measures its own posture as the posture of the image processing device, and outputs posture information indicating the posture measured by the posture sensor. In this embodiment, as an example, the acquisition unit 210 acquires and outputs posture information indicating which of the four directions perpendicular to the optical axis of the imaging unit the posture measured by the posture sensor is in. In other words, the acquisition unit 210 acquires information regarding the orientation of the imaging unit relative to the object (upright, vertical, or inverted). That is, the orientation of the imaging unit corresponds to the user's holding direction of the image processing device and corresponds to upright shooting, vertical shooting, or inverted shooting.

[0017] The correlation calculation unit 201 performs various processes, such as a registration process to acquire and register template features based on the input image acquired by the image input unit 202, and a correlation calculation process based on the input image acquired by the image input unit 202 and the registered template features.

[0018] The CPU (Central Processing Unit) 203 executes various processes using computer programs and data stored in the ROM (Read Only Memory) 204 and RAM (Random Access Memory) 205. In this way, the CPU 203 controls the operation of the entire image processing unit and executes or controls the various processes described as being performed by the image processing unit.

[0019] ROM204 stores configuration data for the image processing unit, computer programs and data related to the startup of the image processing unit, and computer programs and data related to the basic operation of the image processing unit. ROM204 also stores computer programs and data that cause the CPU203 to execute or control various processes described as being performed by the image processing unit.

[0020] The RAM 205 has an area for storing input images received from the image input unit 202, an area for storing posture information acquired by the acquisition unit 210, and an area for storing data output from the correlation calculation unit 201. Furthermore, the RAM 205 has an area for storing computer programs and data loaded from the ROM 204, and a work area used by the CPU 203 when executing various processes. In this way, the RAM 205 can provide various areas as appropriate. The RAM 205 is composed of, for example, a large-capacity DRAM (Dynamic Access Memory).

[0021] The DMAC (Direct Memory Access Controller) 206 controls data transfer between the image input unit 202 or RAM 205 and the correlation calculation unit 201.

[0022] The user interface unit 208 has a user interface such as buttons, switches, and a touch panel, and the user can input various instructions (for example, instructions for the target to track) to the CPU 203 by operating it. The user interface unit 208 also has a display screen (such as an LCD screen or a touch panel screen) for displaying the processing results of this device (for example, the results of the tracking process).

[0023] The image input unit 202, acquisition unit 210, correlation calculation unit 201, CPU 203, ROM 204, RAM 205, DMAC 206, and user interface unit 208 are all connected to the system bus 207.

[0024] The correlation calculation unit 201 performs the above correlation calculation process according to instructions from the CPU 203 and generates a detection map that shows the likelihood of the position of an object (here, for explanatory purposes, referred to as the tracking target object) in the input image. The detection map is stored in the RAM 205 by the CPU 203. The CPU 203 provides various applications using the tracking processing results of the tracking target object based on the detection map stored in the RAM 205. For example, the tracking processing results are fed back to the image input unit 202 and used to control the focus of the optical system for tracking the tracking target object. Note that the method of using the detection map generated based on the correlation calculation by the correlation calculation unit 201 is not limited to a specific method. Also, the methods of using the various maps that appear in the following description are not limited to the specific methods described below.

[0025] Next, an example of the hardware configuration of the correlation calculation unit 201 will be explained using the block diagram in Figure 1. The I / F unit 101 is an interface that can be accessed by the CPU 203 and DMAC 206 via the system bus 207, and the correlation calculation unit 201 sends and receives data to and from the outside via the I / F unit 101.

[0026] Buffer 103 stores a two-dimensional array (coefficient pattern) of the weight coefficients (CNN coefficients) for each layer (each hierarchical level) of the CNN, and buffer 103 is a buffer (memory device) capable of supplying these coefficient patterns with low latency.

[0027] Furthermore, buffer 103 stores CNN feature groups within local regions of the "two-dimensional array of CNN features of the final layer of the CNN (feature map)" obtained by the conversion processing unit 105 as template features. CPU 203 reads the "feature map of the final layer of the CNN" obtained by the conversion processing unit 105 and stored in buffer 104 via I / F unit 101, and extracts CNN feature groups within local regions of the read feature map as template features. Then, CPU 203 rotates the extracted template features according to the pose information acquired by the acquisition unit 210, and stores the rotated template features in buffer 103 via I / F unit 101.

[0028] Buffer 104 is a buffer (memory device) capable of storing the feature map obtained by the conversion processing unit 105 with low latency. Buffers 103 and 104 can be implemented using, for example, high-speed memory or registers. In Figure 1, buffers 103 and 104 are shown as separate buffers, but they may also be represented as separate memory areas within a single buffer.

[0029] When the rotation processing unit 107 obtains a coefficient pattern and orientation information from the CPU 203 via the I / F unit 101, it rotates the obtained coefficient pattern according to the obtained orientation information before supplying it to the arithmetic processing unit 102. For example, if the orientation information indicates that the image is "portrait orientation" (i.e., the input image obtained by the image input unit 202 is portrait orientation), the rotation processing unit 107 rotates the coefficient pattern 90 degrees clockwise before supplying it to the arithmetic processing unit 102.

[0030] On the other hand, when the rotation processing unit 107 obtains template features and orientation information from the CPU 203 via the I / F unit 101, it rotates the obtained template features according to the obtained orientation information before supplying them to the arithmetic processing unit 102. For example, if the orientation information indicates that the image is "portrait orientation" (i.e., the input image obtained by the image input unit 202 is portrait orientation), the rotation processing unit 107 rotates the template features 90 degrees clockwise before supplying them to the arithmetic processing unit 102.

[0031] The arithmetic processing unit 102 performs a convolution operation, and the transformation processing unit 105 performs a nonlinear transformation on the result of the convolution operation performed by the arithmetic processing unit 102. The nonlinear transformation in the transformation processing unit 105 utilizes well-known activation processes such as ReLU (Rectified Linear Unit) and the sigmoid function. When using ReLU, it can be implemented using thresholding; when using the sigmoid function, values ​​are transformed using a lookup table or the like. The control unit 106 performs various operational controls in the correlation calculation unit 201.

[0032] Next, the operation of the CNN (processing for generating a feature map) performed by the arithmetic processing unit 102 and the transformation processing unit 105 will be explained using Figure 4(a). In the CNN, a convolution operation 403 is performed on the input image 401 and the coefficient pattern 402, and a nonlinear transformation 404 is performed on the result of the convolution operation 403 to generate a feature map 405.

[0033] Here, if the kernel (filter coefficient matrix) size of the convolution operation is columnSize × rowSize, and the number of feature maps in the layer preceding the layer being processed in the CNN is L, then one feature map is calculated based on the convolution operation shown below.

[0034]

number

[0035] input(x,y): Reference pixel value in 2D coordinates (x,y) output(x,y): Calculation result in 2D coordinates (x, y) weight(column, row): CNN coefficients at coordinates (x+column, y+row) L: Number of feature maps in the previous layer columnSize, rowSize: Horizontal and vertical dimensions of the 2D convolution kernel. In general, the computation process in a CNN involves repeatedly performing sum-of-products operations while scanning multiple convolution kernels pixel by pixel in the input image according to the above formula, and then calculating a feature map by applying a nonlinear transformation (activation process) to the final sum-of-products result. That is, pixel data for one feature map is generated by multiple spatial filter operations and a nonlinear operation on their sum. In this embodiment, the CNN coefficients correspond to the spatial filter coefficients. In reality, multiple feature maps are generated for each layer.

[0036] The arithmetic processing unit 102 has a multiplier and an adder, and performs the convolution operation shown in the above formula using the multiplier and adder. The transformation processing unit 105 then performs a nonlinear transformation on the result of the convolution operation to generate a feature map. In a typical CNN, the above process is repeated for the number of feature maps to be generated.

[0037] Next, we will explain how to obtain template features from feature maps using Figure 5. Figure 5 shows a case where three feature maps 501 are obtained from the final layer of the CNN. In this case, the CPU 203 extracts a set of features (CNN features) within a 3x3 pixel area (spatial sub-region) from each of the three feature maps 501 as template features 502. In this case, the data size of the template features 502 is 9. The location of the "spatial sub-region" is the location of a pre-specified object. For example, in object tracking processing, the template features become the feature quantities of the object being tracked. By utilizing the correlation (correlation map) between such template features and feature maps, the location of the object being tracked can be determined. That is, a location showing a high correlation in the input image can be determined to be the location of the object being tracked in the input image.

[0038] Next, the operation of the CNN (various operations including correlation operations using template features) processed by the arithmetic processing unit 102 and the transformation processing unit 105 will be explained using Figure 4(b). In the CNN, a convolution operation 408 is performed on the input image 406 and the coefficient pattern 407, and a nonlinear transformation 409 is performed on the result of the convolution operation 408 to generate a feature map 410. Here, the feature map 410 is assumed to be three feature maps, and the registered template features 411 are assumed to be the three template features 502 in Figure 5. Next, the CNN calculates the correlation between the feature map 410 and the template features 411 by performing a convolution operation 412 on the feature map 410 (three feature maps) and the template features 411 (three template features 502). By repeatedly performing the convolution operation 412 within the feature map, in the case of Figure 5, three types of correlation maps are calculated as correlation map 413 from the three feature maps. The correlation operation here is the same as a so-called depth-wise CNN operation, where the connection between the input feature map and the output map is one-to-one (L=1 in the above equation). The content of this operation is the Siam correlation operation method described in Non-Patent Document 2, etc. Next, the CNN performs a convolution operation 415 between the correlation map 413 and the coefficient pattern 414, and a nonlinear transformation 416 is performed on the result of the convolution operation 415 to generate a feature map (detection map) 417. The feature map 417 is a single feature map. By performing CNN processing (convolution operation and nonlinear transformation) on the correlation map 413, a detection map can be obtained that clarifies the correlation and enables more stable detection of the target object. The value of each element in the detection map (element value) represents the likelihood (probability) that the element is an element that constitutes the target object corresponding to the template feature. Therefore, the position where the element value in the detection map peaks can be determined to be the position of the target object corresponding to the template feature.

[0039] By performing the operation shown in Figure 4(b), for example, on each frame included in a video or on each of multiple still images captured periodically or irregularly, it is possible to generate a map that allows the position of an object corresponding to a template feature to be detected for each image. In other words, it becomes possible to generate a map that enables tracking of a specific object across multiple images.

[0040] Next, the operation of the image processing apparatus according to this embodiment will be described according to the flowchart in Figure 10. In step S1001, the CPU 203 performs various initialization processes necessary for the operation of the correlation calculation unit 201.

[0041] In step S1002, the CPU 203 reads various operating parameters necessary for the operation of the correlation calculation unit 201 from the ROM 204 and stores them in the RAM 205. Note that the source of the operating parameters is not limited to the ROM 204.

[0042] In step S1003, the CPU 203 determines whether or not to generate a template feature and store it in buffer 103. For example, if a new template feature is to be registered in buffer 103, it determines to "generate a template feature and store it in buffer 103". Also, for example, if a template feature stored in buffer 103 is to be updated with a new template feature, it also determines to "generate a template feature and store it in buffer 103".

[0043] If, as a result of this judgment, it is decided to "generate template features and store them in buffer 103", the process proceeds to step S1004. On the other hand, if it is not decided to "generate template features and store them in buffer 103", the process proceeds to step S1008. In step S1004, the CPU 203 acquires attitude information indicating the orientation of the imaging unit acquired by the acquisition unit 210.

[0044] In step S1005, the CPU 203 acquires the coefficient pattern stored in the buffer 103 via the I / F unit 101. The CPU 203 then controls the DMAC 206 to output the acquired coefficient pattern, the posture information acquired in step S1004, and the input image acquired by the image input unit 202 (an image taken in the shooting posture indicated by the posture information) to the correlation calculation unit 201. The input image input to the correlation calculation unit 201 via the I / F unit 101 is stored in the buffer 104. The rotation processing unit 107 rotates the coefficient pattern acquired from the CPU 203 via the I / F unit 101 according to the posture information acquired from the CPU 203 via the I / F unit 101. For example, if the posture information indicates that the shooting unit is held "vertically" (i.e., the input image acquired by the image input unit 202 is taken vertically), the coefficient pattern is rotated 90 degrees clockwise. The arithmetic processing unit 102 performs a convolution operation on the input image stored in the buffer 104 and the coefficient pattern rotated by the rotation processing unit 107. The transformation processing unit 105 performs a nonlinear transformation on the result of the convolution operation by the arithmetic processing unit 102 to generate a feature map (first generation). The generated feature map is stored in the buffer 104. Subsequently, the "convolution operation of the feature map stored in the buffer 104 (the feature map corresponding to the previous layer) and the coefficient pattern rotated by the rotation processing unit 107" by the arithmetic processing unit 102 and the "process of generating a feature map by performing a nonlinear transformation on the result of the convolution operation by the arithmetic processing unit 102 and storing the generated feature map in the buffer 104" by the transformation processing unit 105 is performed for each layer toward the final layer of the CNN, thereby storing the feature map of each layer of the CNN in the buffer 104. The CPU 203 then acquires the "feature map of the final layer of the CNN" stored in the buffer 104 via the I / F unit 101. Then, in the acquired feature map, the CNN features within the local region corresponding to the object are obtained as template features.

[0045] In step S1006, the CPU 203 rotates the template features acquired in step S1005 according to the orientation information acquired in step S1004 (performing a reverse rotation on the template features in contrast to the rotation of the coefficient pattern by the rotation processing unit 107). For example, if the orientation information indicates that the image is "portrait orientation" (i.e., the input image acquired by the image input unit 202 is portrait orientation), the template features acquired in step S1005 are rotated 90 degrees counterclockwise. In step S1007, the CPU 203 stores the template features that were reversed in step S1006 in the buffer 103 via the I / F unit 101.

[0046] In step S1008, the CPU 203 acquires the posture information acquired by the acquisition unit 210. Here, the posture information acquired in step S1008 is information indicating the posture of the imaging unit measured by the posture sensor at a measurement timing different from the measurement timing corresponding to the posture information acquired in step S1004.

[0047] In step S1009, the CPU 203 acquires the coefficient pattern and template features stored in the buffer 103 via the I / F unit 101. The CPU 203 then controls the DMAC 206 to output the acquired coefficient pattern and template features, the posture information acquired in step S1008, and the input image acquired by the image input unit 202 (an image taken in the shooting posture indicated by the posture information) to the correlation calculation unit 201. The input image input to the correlation calculation unit 201 via the I / F unit 101 is stored in the buffer 104. The rotation processing unit 107 rotates the coefficient pattern acquired from the CPU 203 via the I / F unit 101 according to the posture information acquired from the CPU 203 via the I / F unit 101. The calculation processing unit 102 performs a convolution operation on the input image stored in the buffer 104 and the coefficient pattern rotated by the rotation processing unit 107. The transformation processing unit 105 performs a nonlinear transformation on the result of the convolution operation performed by the arithmetic processing unit 102 to generate a feature map (second generation) and stores the generated feature map in the buffer 104. Subsequently, the arithmetic processing unit 102 performs a convolution operation on the feature map stored in the buffer 104 (the feature map corresponding to the previous layer) and the coefficient pattern rotated by the rotation processing unit 107, and the transformation processing unit 105 performs a nonlinear transformation on the result of the convolution operation performed by the arithmetic processing unit 102 to generate a feature map and stores the generated feature map in the buffer 104. This process is performed for each layer towards the final layer of the CNN, thereby storing the feature map for each layer of the CNN in the buffer 104. In addition, the rotation processing unit 107 rotates the template features obtained from the CPU 203 via the I / F unit 101 according to the pose information obtained from the CPU 203 via the I / F unit 101. For example, if the orientation information indicates that the image is "portrait orientation" (i.e., the input image acquired by the image input unit 202 is portrait orientation), the rotation processing unit 107 rotates the template feature 90 degrees clockwise. Then, the calculation processing unit 102 performs a convolution operation between the rotated template feature and the "feature map in the final layer of the CNN" stored in the buffer 104 to obtain a correlation map showing the correlation between the feature map and the template feature.Next, the same processing as described above, "hierarchical convolution operation and nonlinear transformation on the input image by the arithmetic processing unit 102 and the transformation processing unit 105," is performed on the correlation map to obtain the map from the final layer of the CNN as the detection map.

[0048] In step S1010, the CPU 203 obtains the detection map obtained by the correlation calculation unit 201 from the correlation calculation unit 201 via the I / F unit 101, and stores the obtained detection map in the RAM 205.

[0049] In step S1011, the CPU 203 determines whether the processing termination conditions have been met. For example, if the user operates the user interface unit 208 and inputs a command to terminate the processing, the CPU 203 determines that the processing termination conditions have been met. Also, for example, if the elapsed time since the start of processing according to the flowchart in Figure 10 reaches a specified time, or if the number of repetitions of the processing in steps S1003 to S1010 reaches a specified number, the CPU 203 determines that the processing termination conditions have been met. Thus, the processing termination conditions are not limited to specific conditions.

[0050] If the termination conditions are met as a result of this judgment, the process will terminate according to the flowchart in Figure 10. On the other hand, if the termination conditions are not met, the process will proceed to step S1003.

[0051] Next, the features of this embodiment will be described. First, the conventional extraction and correlation calculation of template features when the shooting posture changes will be explained using Figure 6. The input image 601 is an image taken of the object 602 in a vertical orientation. Below, we will describe the case in which template features are generated and registered from such an input image 601. Note that even in the case of vertical shooting, the input image output by the image input unit 202 is assumed to be a horizontal raster image (the object is rotated and captured within the input image).

[0052] In this case, first, the input image 601 is rotated according to the shooting orientation (in this case, vertical shooting), and a CNN process using the coefficient pattern 604 is performed on the rotated input image 603 to obtain a feature map. Since the coefficient pattern 604 is obtained by learning on an upright object, it is necessary to rotate the input image 601 in order to extract a feature map similar to the case when the shooting orientation is upright. Then, the features of the region at the position of the object 606 are extracted from the obtained feature map as a template feature 605 (in Figure 6, it is intentionally shifted for explanatory purposes, but the actual position of the template feature 605 is the position of the object 606). In this case, the template feature 605 is registered for the subsequent correlation calculation.

[0053] After registering template features 605, when a new vertically-oriented input image 607 is input, the input image 607 is rotated according to the shooting orientation of the input image 607, and a CNN processing using a coefficient pattern is performed on the rotated input image 608 to obtain a feature map. Then, a correlation map 609 is generated from the obtained feature map and the previously registered template features 605, and a detection map 610 is generated from the correlation map 609.

[0054] If the input image 612 is upright relative to the object 606, image rotation is unnecessary. In this case, a CNN process is performed on the input image 612 using a coefficient pattern to obtain a feature map, a correlation map 614 is generated from the feature map and the previously registered template feature 605, and a detection map 615 is generated from the correlation map 614.

[0055] Traditionally, the input image was rotated according to the shooting orientation for processing. However, because input images generally have a large number of pixels, the processing time required for rotating the input image increases, and the buffer memory used for rotating the input image also increases, leading to increased processing costs. Generally, vertical-to-horizontal image conversion requires frame memory, which poses a significant challenge, for example, in inexpensive imaging equipment.

[0056] In contrast, in this embodiment, the extraction of template features and correlation calculation when the shooting posture changes will be explained using Figure 7. In the case of an input image 701 in which the object is photographed vertically, instead of rotating the input image 701, the coefficient pattern 702 is rotated 90 degrees clockwise, and a feature map is generated by performing CNN processing using the rotated coefficient map 702 and the input image 701. Then, the CNN feature group within the region of the object in the feature map is obtained as template feature 704a. Then, this template feature 704a is rotated according to the shooting posture at the time of registration and registered. Specifically, template feature 704b is registered by rotating template feature 704a 90 degrees counterclockwise (rotated 90 degrees in the opposite direction of rotation to the rotation direction of the coefficient map 702). In this case, as will be explained below, when performing correlation calculation, it becomes possible to always perform correlation calculation using the registered template 704b regardless of the shooting posture.

[0057] Suppose a new input image 705 is obtained in which the shooting orientation during correlation calculation is vertical. In this case, the coefficient map 706 is rotated 90 degrees clockwise, and a feature map 709 is obtained by performing CNN processing using the rotated coefficient map 706 and the input image 705. Then, a correlation calculation is performed between the obtained feature map 709 and the previously registered template feature 704b rotated 90 degrees clockwise to obtain template feature 708, which generates a correlation map 711. Finally, CNN processing is performed on the correlation map 711 (using the coefficient map 710 rotated 90 degrees clockwise) to generate a detection map.

[0058] On the other hand, suppose an input image 712 is obtained in which the shooting posture during correlation calculation is upright. In this case, a feature map 716 is obtained by performing CNN processing using the input image 712 and the coefficient pattern 713 that does not rotate. Then, a correlation calculation is performed between the obtained feature map 716 and the previously registered template feature 704b to generate a correlation map 718. Then, CNN processing is performed on the correlation map 718 (using the coefficient pattern 717 that does not rotate) to generate a detection map.

[0059] Also, assume that an input image 719 in portrait orientation, where the shooting pose during the correlation operation is in the opposite direction to the input image 701, is obtained. In this case, a feature map 723 is acquired by performing CNN processing using the coefficient pattern 720 rotated 90 degrees counterclockwise and the input image 719. Then, a correlation map 725 is generated by performing a correlation operation between the acquired feature map 723 and a template feature 722 obtained by rotating the previously registered template feature 704b 90 degrees counterclockwise. Then, CNN processing (using the coefficient pattern 724 rotated 90 degrees counterclockwise) is performed on the correlation map 725 to generate a detection map.

[0060] In this way, by rotating and registering the template feature according to the shooting pose at the time of registering the template feature, it is possible to calculate an appropriate correlation map regardless of the shooting pose by utilizing the mechanism of coefficient rotation by hardware.

[0061] Next, the rotation of the coefficient pattern and the template feature performed by the rotation processing unit 107 will be described. The rotation processing performed by the rotation processing unit 107 is a rotation processing of a two-dimensional array such as a coefficient pattern or a template feature, which is performed by changing the order of reading elements from the two-dimensional array.

[0062] As shown in FIG. 3, the 3x3 coefficient pattern acquired from the CPU 203 via the I / F unit 101 is stored in the buffer 303, and the feature map in 3x3 units read from the buffer 104 is stored in the buffer 304.

[0063] The buffer 303 for storing the coefficient pattern has nine registers (C 0,0 、C 0,1 、C 0,2 、C 1,0 、C 1,1 、C 1,2 、C 2,0 、C 2,1 、C 2,2 ). In each of the nine registers, the CNN coefficient at the corresponding position in the 3x3 coefficient pattern is stored. That is, register C 0,0The leftmost CNN coefficient in the top row of the coefficient pattern is stored in C. 0,1 The middle CNN coefficient of the top row in the coefficient pattern is stored in C. 0,2 The register C stores the CNN coefficient at the rightmost end of the top row in the coefficient pattern. 1,0 The leftmost CNN coefficient in the center row of the coefficient pattern is stored in C. 1,1 The middle CNN coefficient of the middle row in the coefficient pattern is stored in C. 1,2 The register C stores the CNN coefficients at the rightmost end of the center row in the coefficient pattern. 2,0 The leftmost CNN coefficient in the bottom row of the coefficient pattern is stored in C. 2,1 The middle CNN coefficient of the bottom row in the coefficient pattern is stored in C. 2,2 This variable stores the CNN coefficient at the rightmost end of the bottom row in the coefficient pattern.

[0064] The buffer 304 that stores the feature map has 9 registers (F 0,0 F 0,1 F 0,2 F 1,0 F 1,1 F 1,2 F 2,0 F 2,1 F 2,2 It has ). Each of the 9 registers stores the CNN feature at the corresponding position in the 3x3 feature map. That is, register F 0,0 The leftmost CNN feature in the top row of the feature map is stored here. 0,1 The CNN feature in the top row of the feature map is stored in F. 0,2 This register stores the CNN feature at the rightmost end of the top row in the feature map. 1,0 The CNN feature at the leftmost end of the center row in the feature map is stored here. 1,1 This stores the central CNN feature of the central row in the feature map. 1,2 The CNN feature at the far right of the center row in the feature map is stored in register F. 2,0 This stores the CNN feature at the leftmost end of the bottom row in the feature map. 2,1This stores the CNN feature in the middle of the bottom row of the feature map. 2,2 This contains the CNN feature at the far right of the bottom row in the feature map.

[0065] The arithmetic processing unit 102 (multiplier 301) performs a sum-of-products operation on the CNN coefficient data sequence output from the rotation processing unit 107 and the feature map stored in buffer 304, based on the coefficient pattern stored in buffer 303. The arithmetic processing unit 102 (accumulator adder 302) then performs a cumulative addition of the results of the sum-of-products operation to realize a convolution operation according to the above formula.

[0066] As described later, the rotation processing unit 107 reads each CNN coefficient of the coefficient pattern stored in the buffer 303 in an order determined according to the pose information, and outputs a one-dimensional data sequence (CNN coefficient data sequence) in which the read CNN coefficients are arranged in the order they were read. In other words, the nth (1≦n≦9)th CNN coefficient from the beginning in the CNN coefficient data sequence is the nth CNN coefficient read from the coefficient pattern stored in the buffer 303.

[0067] The multiplier 301 also processes the nine registers in buffer 304 in raster data order (F 0,0 F 0,1 F 0,2 F 1,0 F 1,1 F 1,2 F 2,0 F 2,1 F 2,2 The references are made in the following order, and the CNN features registered in the referenced register are read out.

[0068] The multiplier 301 then calculates the result of multiplying the nth (1≦n≦9)th CNN coefficient in the CNN coefficient data sequence by the nth CNN feature read from buffer 304, and completes one spatial filter operation by cumulatively adding the nine calculated multiplication results. In practice, depending on the CNN's connection relationships, it calculates the cumulative sum of data corresponding to multiple coefficients and multiple feature faces. That is, it performs the spatial filter operation on the feature face that was performed on multiple feature faces of the previous layer. Therefore, the number of spatial filters is equal to the total number of feature faces in all layers multiplied by the number of feature faces being processed.

[0069] In this case, the rotation of the coefficient pattern by the rotation processing unit 107 will be described. As mentioned above, since both the coefficient pattern and the template features are the same two-dimensional array, the following explanation can also be applied to the rotation of the template features.

[0070] An example of the configuration of the rotation processing unit 107 will be explained using the block diagram in Figure 8. The coefficient selection unit 802 obtains the reading order registered in the coefficient selection table 803 in association with the posture information obtained from the CPU 203 via the I / F unit 101. The coefficient selection table 803 stores, for example, four types of reading orders depending on the shooting posture. For example, if the posture information indicates that it is "held vertically", the coefficient pattern 801 is rotated 90 degrees clockwise. The coefficient selection table 803 stores posture information indicating "held vertically" and the reading order "C 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 " and are registered in correspondence.

[0071] The coefficient selection unit 802 then references the nine registers in buffer 303 in the order they were read. For example, if the orientation information indicates that the camera is "held vertically", the coefficient pattern 801 is rotated 90 degrees clockwise. 2,0 , C1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 The registers are referenced in the order of "C". The coefficient selection unit 802 then reads the CNN coefficients stored in the registers in the order they were referenced, and outputs a one-dimensional data sequence (CNN coefficient data sequence) in which the read CNN coefficients are arranged in the order they were read. Note that the registers are referred to as "C 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 The registers are referenced in the order of "C", the CNN coefficients stored in the referenced registers are read, and the 3x3 pattern obtained by arranging the read CNN coefficients in the order they were read and in raster data order is a coefficient pattern obtained by rotating the coefficient pattern 801 held by buffer 303 90 degrees clockwise. However, the registers are referenced in the order of "C 2,0 , C 1,0 , C 0,0 , C 2,1 , C 1,1 , C 0,1 , C 2,2 , C 1,2 , C 0,2 The CNN coefficients are read from the referenced registers in the order of ',' and the one-dimensional data sequence (CNN coefficient data sequence) obtained by arranging the read CNN coefficients in the order they were read is the same as the coefficient pattern 801 held by buffer 303, rotated 90 degrees clockwise. In other words, only the number of dimensions of the data array is different.

[0072] The coefficient selection table 803 then outputs the CNN coefficient data sequence obtained in this way to the multiplier 301. Alternatively, the multiplier 301 may generate the CNN coefficient data sequence.

[0073] Furthermore, if the buffer 303 is configured with registers, the coefficient selection unit 802 can be configured with a selector that sequentially selects its output. In this way, the rotation processing unit 107 can be configured using a coefficient selection table 803 with a relatively small amount of data and a coefficient selection selector, and the increase in the cost required for the rotation processing unit 107 is minimal. Also, when supporting multiple types of kernel sizes, it is only necessary to change the information stored in the coefficient selection table 803 and the configuration of the coefficient selection unit 802 according to the type of kernel.

[0074] Next, an example of the memory area configuration in buffer 103 will be explained using Figure 9. Figure 9 shows an example of the memory area configuration of buffer 103 in the example in Figure 4, where coefficient patterns 402 and 407 are stored in memory area 901, and coefficient pattern 414 is stored in memory area 902. In addition, template feature 411 is stored in memory area 903.

[0075] The coefficient patterns stored in memory area 901 and memory area 902 are transferred to buffer 303 in a predetermined unit (3x3 units in the example of Figure 8) based on control by the control unit 106. Similarly, the template features stored in memory area 903 are transferred to buffer 303 in a predetermined unit (3x3 units in the example of Figure 8) based on control by the control unit 106.

[0076] This configuration enables the rotation of coefficient patterns and template features. For example, in Figure 7, when storing template feature 704b, which is obtained by rotating template feature 704a, in buffer 103, elements 6, 3, 0, 7, 4, 1, 8, 5, 2 in template feature 704a, which are arranged in raster data order, are read out in the order of elements 0, 1, 2, 3, 4, 5, 6, 7, 8 to form a 3x3 template feature 704b in which elements 0, 1, 2, 3, 4, 5, 6, 7, 8 are arranged in raster data order, and this template feature 704b is stored in buffer 103. When performing correlation calculations on consecutive images, even if there is a change in the shooting posture, processing corresponding to the shooting posture can be performed simply by setting posture information.

[0077] Thus, according to this embodiment, the template features are rotated in the upright direction and stored according to the shooting orientation when the template features are generated, so the correlation calculation can be efficiently performed according to the shooting orientation. In other words, during the correlation calculation, regardless of the shooting orientation, the rotation of the captured image is not required, and the template features can be processed using the rotation mechanism of the coefficient map. That is, CNN processing and correlation calculation can be processed in response to orientation changes using common hardware.

[0078] As a result, when the image processing device according to this embodiment is applied to object tracking, the object can be smoothly tracked without any special processing, even if the shooting posture changes (for example, when shooting while changing how the image processing device is held).

[0079] In this embodiment, the image processing device has been described as having an image input unit 202, but it is not limited to this, and the image input unit 202 may be an external device. For example, the image processing device may perform the above-mentioned various processes based on an input image received from an external image input unit 202 via a wired or wireless network.

[0080] Similarly, although the image processing device has been described as having an acquisition unit 210 for acquiring posture information, it is not limited to this, and the acquisition unit 210 may be an external device. For example, the image processing device may perform the above-mentioned various processes using posture information received from an external acquisition unit 210 via a wired or wireless network.

[0081] Furthermore, although a posture sensor was used to measure the shooting posture in this embodiment, the method for acquiring the shooting posture is not limited to a specific method. For example, the shooting posture may be estimated from multiple captured images, the shooting posture may be measured using other types of sensors, or several methods may be combined to acquire the shooting posture. In addition, the user may input posture information by operating the user interface unit 208.

[0082] Furthermore, although a CNN was used for hierarchical spatial filtering in this embodiment, hierarchical spatial filtering is not limited to a specific method and may be performed using, for example, other types of hierarchical neural networks.

[0083] Furthermore, the various processes described as being performed by the correlation calculation unit 201 may also be executed by processors such as the CPU 203, GPU (Graphics Processing Unit), and DSP (Digital Signal Processing Unit).

[0084] [Second Embodiment] This embodiment will explain the differences from the first embodiment, and unless otherwise specified below, it will be assumed to be the same as the first embodiment. In the first embodiment, the feature maps in each layer were stored in buffer 104, but in that case, the capacity required for buffer 104 increases. The increase in capacity becomes a problem in particular when the number of feature maps in each layer is large. Therefore, instead of processing for each layer, it is also possible to use a configuration in which processing is carried out in small areas across multiple layers. In this case, the processing of each layer is carried out in a predetermined unit (for example, line by line). This allows, for example, the storage of the feature map 410 and correlation map 413, which are intermediate results of the hierarchical processing exemplified in Figure 4. Then, (when stored in buffer 104), it is possible to process by allocating a memory area in buffer 104 corresponding to (number of lines required for spatial filtering × number of maps). In other words, buffer 104 is used as a circular buffer on a line basis to process the hierarchical network.

[0085] Figure 11 schematically illustrates an example of a CNN feature surface when processing using line buffers. 1101 represents a cyclic line buffer for the input image 406, 1102 represents a cyclic line buffer for the feature map 410, and 1103 represents a cyclic line buffer for the correlation map 413. Note that Figure 11 shows an example where the spatial filter size is 3x3. The memory for holding the detection map 417 of the final layer is composed of a frame buffer that holds all results. For example, for the feature map 410, after filterable reference data is accumulated in the line buffer 1101 of the input image 406, a convolution operation for one line is performed. The feature map and correlation map are calculated by cyclically processing line buffers 1101 to 1103 line by line. The arithmetic processing unit 102 processes the network across layers by switching the feature surface to be processed line by line.

[0086] The control for processing with the line buffer is managed, for example, by the control unit 106, which sequentially performs convolution operations 408, 412, and 415 for each line of the input image in step S1009. Such processing can be realized, for example, by the configuration disclosed in Japanese Patent No. 5184824.

[0087] During the convolution operation, the rotation direction of the CNN coefficients is specified to the rotation processing unit 107 according to the pose information. In this embodiment, template features are always registered in an upright state regardless of the shooting pose at the time of registration, so the same rotation may be specified for all layers according to the shooting pose during the correlation operation. In other words, even when processing correlation operations across multiple layers, processing can be performed efficiently without having to perform special processing between layers to respond to changes in the shooting pose.

[0088] [Third Embodiment] In the first and second embodiments, the rotation of the two-dimensional array was performed by changing the reading order of the elements of the two-dimensional array, but the rotation of the two-dimensional array may be performed by other methods. For example, the two-dimensional array may be rotated by a hardware rotation mechanism.

[0089] Alternatively, two-dimensional arrays rotated at multiple angles may be created in advance and stored in RAM205, and one of these pre-created two-dimensional arrays may be selected and used according to the orientation information. For example, a two-dimensional array rotated 90 degrees clockwise, a two-dimensional array rotated 90 degrees counterclockwise, and a two-dimensional array rotated 180 degrees clockwise / counterclockwise may be created in advance. Then, for example, if the orientation information indicates a vertical shot in a clockwise direction, the two-dimensional array rotated 90 degrees clockwise is selected, and if the orientation information indicates a vertical shot in a counterclockwise direction, the two-dimensional array rotated 90 degrees counterclockwise is selected.

[0090] Furthermore, in the first and second embodiments, the template feature was assumed to be a map of CNN features extracted from local regions in the feature map, but it is not limited to this; any map of CNN features obtained based on the feature map is acceptable. For example, the template feature may be a map obtained by processing CNN features extracted from a portion of the feature map.

[0091] Furthermore, in the first and second embodiments, the registration of template features was described as being converted to an upright template feature according to the shooting orientation before registration. However, it is also possible to convert and register the template feature in a predetermined specific orientation regardless of the shooting orientation. In this case, however, it is necessary to control the rotation of the template feature when calculating the correlation map 413, for example, so that it differs from other convolution operations.

[0092] Furthermore, although the first and second embodiments described cases where the image used to acquire the template features and the image used for correlation calculation are different, the image used to acquire the template features and the image used for correlation calculation may be the same.

[0093] Furthermore, the rotation processing unit 107, arithmetic processing unit 102, conversion processing unit 105 shown in Figure 1, and one or more functional units contained therein, may be implemented in hardware or in software. In the latter case, this software is executed by the control unit 106 and the CPU 203, thereby realizing the functions of the corresponding functional units.

[0094] Furthermore, the numerical values, processing timing, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0095] Furthermore, some or all of the embodiments described above may be used in appropriate combinations. Alternatively, some or all of the embodiments described above may be used selectively.

[0096] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0097] The disclosures herein include the following image processing apparatus, image processing methods, and computer programs.

[0098] (Item 1) A first generation means generates a first feature map by applying filter coefficients set in a first array based on first posture information indicating the shooting posture of the first captured image to the first captured image, A registration means that acquires template features corresponding to an object based on the first feature map and registers the template features in an array based on the first posture information, A second generation means generates a second feature map by applying filter coefficients set in a second array based on second posture information indicating the shooting posture of the second captured image to the second captured image, A calculation means for performing a correlation calculation between the second feature map and a template feature in which the registered template features are set in an array based on the second pose information, A detection means for detecting the object from the second captured image based on the results of the correlation calculation. An image processing apparatus characterized by comprising:

[0099] (Item 2) The first generation means rotates and sets the filter coefficient according to the first attitude information, The image processing apparatus according to item 1, characterized in that the registration means registers the acquired template features by rotating them in the opposite direction to the rotation of the filter coefficients by the first generation means.

[0100] (Item 3) The second generation means rotates and sets the filter coefficient according to the second attitude information, The image processing apparatus according to item 2, characterized in that the calculation means performs a correlation calculation between the second feature map and a template feature obtained by rotating the registered template feature in accordance with the rotation of the filter coefficient by the second generation means.

[0101] (Item 4) The image processing apparatus according to any one of items 1 to 3, characterized in that the registration means acquires features within the region of the object in the first feature map as template features.

[0102] (Item 5) The first generation means generates the first feature map based on a convolution operation between the filter coefficients set in the first array and the first captured image. The image processing apparatus according to any one of items 1 to 4, characterized in that the second generation means generates the second feature map based on a convolution operation between the filter coefficients set in the second array and the second captured image.

[0103] (Item 6) The image processing apparatus according to any one of items 1 to 5, characterized in that the calculation means performs the correlation calculation by convolution of the second feature map and template features set in an array based on the second pose information.

[0104] (Item 7) The image processing apparatus according to item 5 or 6, characterized in that the convolution operation is performed using a hierarchical neural network.

[0105] (Item 8) The image processing apparatus according to item 7, characterized in that the convolution operation is performed in a predetermined unit for each layer of the hierarchical neural network.

[0106] (Item 9) The image processing apparatus according to any one of items 1 to 8, characterized in that the detection means generates a detection map showing the likelihood of the position of the object in the second captured image based on the result of the correlation calculation.

[0107] (Item 10) moreover, The image processing apparatus according to item 9, characterized by comprising means for performing control related to shooting according to the detection map.

[0108] (Item 11) moreover, An image processing apparatus according to any one of items 1 to 10, characterized by comprising means for acquiring the first captured image and the image captured as the second captured image.

[0109] (Item 12) moreover, An image processing apparatus according to any one of items 1 to 11, characterized by comprising means for acquiring the first posture information and the second posture information.

[0110] (Item 13) An image processing method performed by an image processing device, The first generation means of the image processing apparatus includes a first generation step of generating a first feature map by applying filter coefficients set in a first array based on first posture information indicating the shooting posture of the first captured image to the first captured image, The registration means of the image processing device includes a registration step of acquiring template features corresponding to an object based on the first feature map and registering the template features in an array based on the first pose information, The second generation means of the image processing apparatus includes a second generation step of generating a second feature map by applying filter coefficients set in a second array based on second posture information indicating the shooting posture of the second captured image to the second captured image, The calculation means of the image processing device performs a calculation step of correlation calculation between the second feature map and the registered template features set in an array based on the second pose information, The detection means of the image processing apparatus performs a detection step of detecting the object from the second captured image based on the result of the correlation calculation. An image processing method characterized by comprising:

[0111] (Item 14) A computer program for causing a computer to function as one of the means of an image processing device described in any one of items 1 through 12.

[0112] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0113] 101: I / F section 102: Arithmetic processing section 103: Buffer 104: Buffer 105: Conversion processing section 106: Control section 107: Rotation processing section

Claims

1. A first generation means generates a first feature map by applying filter coefficients set in a first array based on first posture information indicating the shooting posture of the first captured image to the first captured image, A registration means that acquires template features corresponding to an object based on the first feature map and registers the template features in an array based on the first posture information, A second generation means generates a second feature map by applying filter coefficients set in a second array based on second posture information indicating the shooting posture of the second captured image to the second captured image, A calculation means for performing a correlation calculation between the second feature map and a template feature in which the registered template features are set in an array based on the second posture information, A detection means for detecting the object from the second captured image based on the results of the correlation calculation. An image processing apparatus characterized by comprising:

2. The first generation means rotates and sets the filter coefficient according to the first attitude information, The image processing apparatus according to claim 1, characterized in that the registration means registers the acquired template features by rotating them in the opposite direction to the rotation of the filter coefficients by the first generation means.

3. The second generation means rotates and sets the filter coefficient according to the second attitude information, The image processing apparatus according to claim 2, characterized in that the calculation means performs a correlation calculation between the second feature map and a template feature obtained by rotating the registered template feature in accordance with the rotation of the filter coefficient by the second generation means.

4. The image processing apparatus according to claim 1, characterized in that the registration means acquires features within the region of the object in the first feature map as template features.

5. The first generation means generates the first feature map based on a convolution operation between the filter coefficients set in the first array and the first captured image. The image processing apparatus according to claim 1, characterized in that the second generation means generates the second feature map based on a convolution operation between the filter coefficients set in the second array and the second captured image.

6. The image processing apparatus according to claim 1, wherein the calculation means performs the correlation calculation by convolution of the second feature map and template features set in an array based on the second pose information.

7. The image processing apparatus according to claim 5, characterized in that the convolution operation is performed using a hierarchical neural network.

8. The image processing apparatus according to claim 7, characterized in that the convolution operation is performed in a predetermined unit for each layer of the hierarchical neural network.

9. The image processing apparatus according to claim 1, characterized in that the detection means generates a detection map showing the likelihood of the position of the object in the second captured image based on the result of the correlation calculation.

10. moreover, The image processing apparatus according to claim 9, further comprising means for performing control related to shooting according to the detection map.

11. moreover, The image processing apparatus according to claim 1, further comprising means for acquiring the first captured image and the second captured image.

12. moreover, The image processing apparatus according to claim 1, further comprising means for acquiring the first posture information and the second posture information.

13. An image processing method performed by an image processing device, The first generation means of the image processing apparatus includes a first generation step of generating a first feature map by applying filter coefficients set in a first array based on first posture information indicating the shooting posture of the first captured image to the first captured image, The registration means of the image processing device includes a registration step of acquiring template features corresponding to an object based on the first feature map and registering the template features in an array based on the first posture information, The second generation means of the image processing apparatus includes a second generation step of generating a second feature map by applying filter coefficients set in a second array based on second posture information indicating the shooting posture of the second captured image to the second captured image, The calculation means of the image processing apparatus performs a calculation step of performing a correlation calculation between the second feature map and the template features which are set in an array based on the second pose information, The detection means of the image processing apparatus includes a detection step of detecting the object from the second captured image based on the result of the correlation calculation, An image processing method characterized by comprising:

14. A computer program for causing a computer to function as each of the means of the image processing apparatus described in claim 1.

Images