Image processing apparatus, image processing method, and program product

Abstract

The image processing apparatus includes a tracking object setting section that sets a tracking object in a first frame of a dynamic image; a first feature tracking section that tracks the tracking object in a second frame based on a first feature of the tracking object set in the first frame; a second feature tracking section that tracks the tracking object in the second frame based on a second feature of the tracking object set in the first frame; a tracking management section that mixes a tracking result based on the first feature tracking section and a tracking result based on the second feature tracking section at a prescribed mixing rate; and an output section that outputs a detection position of the tracking object in the second frame based on a mixed result mixed by the tracking management section.

Description

Technical Field

[0001] This invention relates to image processing apparatus, image processing method, and program products. Background Technology

[0002] In motion analysis of camera-captured images, high-precision tracking of the camera subject is required. Patent Document 1 discloses a technique for detecting the tracked object in a second frame based on the reliability derived from two feature quantities of the tracked object in the first frame.

[0003] Existing technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2012-203613 Summary of the Invention

[0006] The problem that the invention aims to solve

[0007] The methods for observing objects such as humans captured by a fisheye camera vary depending on their relative position to the camera. Therefore, changes in characteristic quantities can sometimes differ even for the same subject depending on its relative position to the camera.

[0008] One aspect of this invention is to provide a technique for improving the tracking accuracy of a target object by using images captured by a fisheye camera.

[0009] Methods for solving problems

[0010] To achieve the above objectives, the present invention adopts the following configuration.

[0011] The first aspect of this disclosure relates to an image processing apparatus comprising: a tracking object setting unit that sets a tracking object in a first frame of a moving image; a first feature tracking unit that tracks the tracking object in a second frame based on a first feature of the tracking object set in the first frame; a second feature tracking unit that tracks the tracking object in the second frame based on a second feature of the tracking object set in the first frame; a tracking management unit that mixes the tracking results based on the first feature tracking unit and the tracking results based on the second feature tracking unit at a predetermined mixing rate; and an output unit that outputs the detection position of the tracking object in the second frame based on the mixing result mixed by the tracking management unit.

[0012] The image processing apparatus tracks an object by mixing the tracking results of a first feature and a second feature of the object at a predetermined mixing rate based on the object's position within the camera's field of view. By tracking based on multiple features, the image processing apparatus can improve the tracking accuracy of the object.

[0013] The tracking result based on the first feature tracking unit can also be a likelihood map representing the likelihood of the location of the tracked object in the second frame, calculated based on the first feature. Similarly, the tracking result based on the second feature tracking unit can be a likelihood map representing the likelihood of the location of the tracked object in the second frame, calculated based on the second feature. The output unit outputs the position with the highest likelihood in the likelihood map of the mixed result as the detection position of the tracked object in the second frame. The image processing device can easily obtain the detection position of the tracked object based on the hierarchical distinction map of the mixed result.

[0014] The specified mixing rate can also be set based on the position of the tracked object within the camera's field of view in the first frame. Since the first and second features of the tracked object are sometimes not consistently obtainable depending on its position within the camera's field of view, the image processing device can improve the tracking accuracy of the tracked object by increasing the mixing rate with respect to the features that can be consistently obtained.

[0015] The specified mixing ratio can also be set based on the orientation of the tracked object relative to the shooting plane of the first frame. By increasing the mixing ratio of features that can be stably obtained with respect to the first and second features of the tracked object, the image processing device can improve the tracking accuracy of the tracked object.

[0016] The image processing apparatus may also include a detection unit that detects objects to be tracked from the first frame. The image processing apparatus can set the objects detected by the detection unit as tracking objects. In addition, if tracking of a tracking object fails, the image processing apparatus can re-detect the object and set it as a tracking object.

[0017] The tracking object setting unit can also obtain the distance from the center of the camera range in the first frame to the tracking object, and the tracking management unit sets a predetermined blending rate based on the distance. The image processing device can set an appropriate blending rate based on the distance from the center of the camera range to the tracking object.

[0018] The first feature can be a color feature, and the second feature can be a shape feature. The image processing device can improve the tracking accuracy of the object by mixing the tracking results based on color features that can be stably obtained near the periphery of the camera range and shape features that can be stably obtained near the center of the camera range.

[0019] The tracking management unit can also set a predetermined mixing rate so that the greater the distance from the center of the camera range in the first frame to the tracked object, the greater the mixing rate of the color features. Since the color features are acquired more stably with a greater distance from the center of the camera range, the image processing device can improve the tracking accuracy of the tracked object by increasing the mixing rate of the color features.

[0020] The first feature tracking unit can also calculate the color distance between the color of the tracked object and the color in the first frame in the second frame, and generate a likelihood map representing the likelihood of the tracked object's location in the second frame based on the color distance. The second feature tracking unit calculates the difference between the second frame and the shape of the image that shifted the position of the tracked object in the first frame, and generates a likelihood map representing the likelihood of the tracked object's location in the second frame based on the difference. By generating the likelihood maps of the first feature and the second feature, the image processing apparatus can easily mix the tracking results.

[0021] The second feature tracking unit can also use a Kernelized Correlation Filter (KCF) to generate a hierarchical classification map representing the likelihood of the location of the tracked object in the second frame. By using the KCF, the image processing device is able to produce a hierarchical classification map of shape features with high accuracy.

[0022] The output unit centers on the position of the tracked object in the second frame and outputs a detection box of the same size as the box surrounding the tracked object in the first frame. The image processing device can easily output the detection box without obtaining the size of the tracked object detected in the second frame.

[0023] The second aspect of the present invention relates to an image processing method that causes a computer to perform: a tracking object setting step, setting a tracking object in a first frame of a moving image; a first feature tracking step, tracking the tracking object in a second frame based on a first feature of the tracking object set in the first frame; a second feature tracking step, tracking the tracking object in the second frame based on a second feature of the tracking object set in the first frame; a tracking management step, mixing the tracking results in the first feature tracking step and the tracking results in the second feature tracking step at a predetermined mixing rate; and an output step, outputting the detection position of the tracking object in the second frame based on the mixing result mixed in the tracking management step.

[0024] This invention can also be understood as a program for implementing the method via a computer, and a recording medium that non-temporarily records the program. Furthermore, the above-described means and processes can be combined with each other as much as possible to constitute this invention.

[0025] Invention Effects

[0026] According to the present invention, the tracking accuracy of a target object can be improved by utilizing images captured by a fisheye camera. Attached Figure Description

[0027] Figure 1 This figure illustrates an application example of the image processing apparatus involved in the implementation method.

[0028] Figure 2 This is a diagram illustrating the hardware structure of an image processing device.

[0029] Figure 3 This is a diagram illustrating the functional structure of an image processing device.

[0030] Figure 4 This is a flowchart illustrating the tracing process.

[0031] Figure 5 This diagram illustrates the settings for the object being tracked.

[0032] Figure 6 This is a diagram illustrating color feature-based tracking.

[0033] Figure 7 This is a diagram illustrating a specific example of color feature-based tracking.

[0034] Figure 8 This diagram illustrates the generation of a heat map that distinguishes color features.

[0035] Figure 9 This is a diagram illustrating shape-feature-based tracking.

[0036] Figure 10 This is a diagram representing a specific example of shape feature-based tracking.

[0037] Figure 11 This diagram illustrates the generation of a hierarchical differentiation map of shape features.

[0038] Figure 12 This is a graph illustrating the calculations for the mixing ratio.

[0039] Figure 13 This is a diagram illustrating the mixing of features. Detailed Implementation

[0040] Hereinafter, one aspect of the embodiments of the present invention will be described based on the accompanying drawings.

[0041] <Example>

[0042] Figure 1 This diagram illustrates an application example of the image processing apparatus according to the embodiment. In cases where the image processing apparatus tracks a human body or similar object from a captured image, it can track the object based on, for example, color or shape features.

[0043] However, the method of observing the tracked object captured by a fisheye camera varies depending on its relative position to the camera. For example, when shooting with a fisheye camera mounted on the ceiling, the tracked object located in the inner periphery of the center of the field of view is captured when viewed from the ceiling side. Alternatively, the tracked object located in the outer periphery of the field of view is captured when viewed from the side (lateral direction).

[0044] For example, when the human body being tracked is located in the outer periphery of the camera's field of view, the area where the color of the clothing can be seen is larger compared to when it is located in the inner periphery, thus the color characteristics are stable. On the other hand, when the human body is located in the inner periphery of the camera's field of view, the area where the color of the clothing can be seen is narrower compared to when it is located in the outer periphery, thus the color characteristics become unstable.

[0045] Furthermore, when the human body being tracked is located in the outer periphery of the camera's field of view, the area of ​​the hands and feet that can be seen is larger compared to when it is located in the inner periphery, thus making the shape features unstable. On the other hand, when the human body is located in the inner periphery of the camera's field of view, the area of ​​the hands and feet that can be seen is smaller compared to when it is located in the outer periphery, thus making the shape features stable.

[0046] Image processing devices can improve tracking accuracy by using stable features for tracking. That is, in the outer perimeter of the camera range, tracking algorithms related to color features are used for tracking, while in the inner perimeter of the camera range, tracking algorithms related to shape features are used for tracking, thereby improving the following ability of the object.

[0047] Therefore, the image processing device modifies the tracking algorithm based on the relative position of the object being tracked relative to the camera. Specifically, it tracks the object by combining (mixing) tracking results based on multiple features according to a predetermined ratio, based on the relative position of the object being tracked relative to the camera.

[0048] The image processing apparatus achieves high-precision tracking of an object by combining a tracking algorithm based on multiple features in a proportion corresponding to the relative position of the object to be tracked relative to the camera. The image processing apparatus of this invention can be applied, for example, to image sensors used for motion analysis.

[0049] <Implementation Method>

[0050] (Hardware configuration)

[0051] Reference Figure 2 An example of the hardware structure of the image processing device 1 will be described. Figure 2This diagram illustrates the hardware configuration of an image processing apparatus 1. The image processing apparatus 1 includes a processor 101, a main storage device 102, an auxiliary storage device 103, a communication interface (I / F) 104, and an output device 105. The processor 101 reads a program stored in the auxiliary storage device 103 into the main storage device 102 and executes it, thereby realizing the function of... Figure 3 The functions of each functional structure are described below. Communication interface 104 is an interface for wired or wireless communication. Output device 105 is, for example, a device for outputting data to a display or the like.

[0052] The image processing device 1 can be a general-purpose computer such as a personal computer, server computer, tablet terminal, or smartphone, or it can be an embedded computer such as a motherboard computer. The image processing device 1 can be implemented through distributed computing across multiple computer devices, or it can implement parts of each functional unit through a cloud server. Furthermore, parts of each functional unit of the image processing device 1 can also be implemented using dedicated hardware devices such as FPGAs or ASICs.

[0053] Image processing device 1 is connected to camera 2 via wired (USB cable, LAN cable, etc.) or wireless (WiFi, etc.) connection to receive image data captured by camera 2. Camera 2 is a camera device having an optical system including lenses and an image sensor (CCD, CMOS, etc.).

[0054] Alternatively, the image processing device 1 can be integrated with the camera 2. Furthermore, some of the processing performed by the image processing device 1, such as object detection and tracking of the captured image, can also be performed by the camera 2. Additionally, the object tracking results based on the image processing device 1 can be sent to an external device for notification to the user.

[0055] (Functional Composition)

[0056] Figure 3 This is a diagram illustrating the functional structure of the image processing apparatus 1. The image processing apparatus 1 includes an image acquisition unit 11, a processing unit 12, and an output unit 13. The processing unit 12 includes a detection unit 121, a tracking object setting unit 122, a first feature tracking unit 123, a second feature tracking unit 124, and a tracking management unit 125.

[0057] The image acquisition unit 11 sends the dynamic image data acquired from the camera 2 to the processing unit 12. The detection unit 121 of the processing unit 12 detects human bodies or other objects to be tracked based on the frame images received from the image acquisition unit 11. The detection unit 121 can detect objects, for example, using background subtraction or inter-frame subtraction.

[0058] The tracking object setting unit 122 sets the objects detected by the detection unit 121 as tracking objects. The tracking object setting unit 122 learns the characteristics of the tracking objects and tracks the tracking objects in subsequent frames based on the learned content. The tracking object setting unit 122 learns features such as the color and shape characteristics of the tracking objects.

[0059] The tracking object setting unit 122 obtains the distance between the tracking object and the center position of the frame image. This distance can be calculated as the distance between the center coordinates of the tracking object and the center coordinates of the frame image. The obtained distance is used to set a specified blending rate when blending color and shape features.

[0060] The first feature tracking unit 123 tracks the tracking object from the frame following the frame where the tracking object was set (also called the first frame) (also called the second frame), based on a first feature, such as a color feature, learned by the tracking object setting unit 122. The first feature tracking unit 123 generates a color feature hierarchy map (likelihood map) as the tracking result by calculating the color distance between the tracking object and the object in the first frame in the second frame. Color distance is an indicator representing the similarity between two colors, such as the distance in a color space like RGB.

[0061] The second feature tracking unit 124 tracks the tracking object based on a second feature, such as a shape feature, learned by the tracking object setting unit 122 from the second frame after the first frame in which the tracking object was set. For example, the second feature tracking unit 124 offsets and overlaps the tracking object set in the first frame relative to the second frame, calculates the difference with respect to shape, and thereby generates a hierarchical differentiation map (likelihood map) of shape features as the tracking result.

[0062] The tracking management unit 125 sets a predetermined mixing rate based on the distance between the tracking object and the center position of the frame image obtained by the tracking object setting unit 122, which mixes the tracking results regarding the first feature and the tracking results regarding the second feature. The predetermined mixing rate is set, for example, based on the distance from the center position of the frame image to the tracking object.

[0063] For example, it is set that the farther the distance from the center of the frame image to the tracked object, the greater the mixing ratio of the color feature (first feature) and the smaller the mixing ratio of the shape feature (second feature). Conversely, it is set that the closer the distance from the center of the frame image to the tracked object, the smaller the mixing ratio of the color feature and the greater the mixing ratio of the shape feature.

[0064] The tracking management unit 125 mixes the tracking results for the first feature and the tracking results for the second feature at a set mixing rate. The tracking results for the first and second features are expressed as differences between the tracking results for each feature and the tracking object. The tracking management unit 125 mixes the tracking results for the first and second features at a mixing rate set according to the location of the tracking object. Thus, the tracking management unit 125 can obtain tracking results that appropriately reflect the tracking results for each feature based on the location of the tracking object.

[0065] The output unit 13 outputs the detection result of the tracked object in the second frame based on the coordinates of the position with the least difference in the tracking result mixed by the tracking management unit 125. If there are multiple coordinates with the least difference, the output unit 13 may also output the center coordinates of the region including these coordinates as the detection result of the tracked object.

[0066] (Follow-up processing)

[0067] Reference Figure 4 Explain the overall process of tracking and processing. Figure 4 This is a flowchart illustrating the tracking process. The tracking process begins, for example, by the user instructing the image processing device 1 to perform tracking. Figure 4 In the example shown, loop processing L1 is performed on each frame of the dynamic image.

[0068] In S101, the processing unit 12 acquires a frame image from the image acquisition unit 11. In S102, the processing unit 12 determines whether the tracking flag is ON. The tracking flag is set to ON if a tracking target has been set, and set to OFF if no tracking target has been set. The setting value of the tracking flag can be recorded in the main storage device 102 or the auxiliary storage device 103. When the tracking flag is ON (S102: Yes), the process proceeds to S107. When the tracking flag is OFF (S102: No), the process proceeds to S103.

[0069] In S103, the detection unit 121 detects a human body from the frame image. For example, the detection unit 121 can detect a human body by using a background difference method to extract regions that have changed between the frame image and a pre-prepared background image, or an inter-frame difference method to extract regions that have changed between frames.

[0070] In addition, Figure 4 In the process described, the tracking object is assumed to be a human body, but it is not limited to this; any animal body that becomes the tracking object can be used. The animal body being tracked is preferably one whose extracted features, such as color and shape features, vary in stability depending on its relative position to the camera.

[0071] In S104, the detection unit 121 determines whether a human body was detected in S103. If a human body was detected (S104: Yes), the process proceeds to S105. If no human body was detected (S104: No), the process returns to S101, and tracking processing for subsequent dynamic image frames begins.

[0072] In S105, the tracking object setting unit 122 sets the human body detected in S104 as the tracking object. If multiple human bodies are detected in S104, the tracking object setting unit 122 can also set multiple human bodies as tracking objects and track them separately. The tracking object setting unit 122 acquires (learns) the color and shape features of the tracking object.

[0073] Here, refer to Figure 5 This section explains the setup of the tracking object. The first frame is the starting frame for detecting the human body and setting the tracking object 501. The second frame is the frame following the starting frame. Tracking is based on the characteristics of the tracking object 501 set in the first frame to determine the location of the tracking object 501 in the second frame.

[0074] exist Figure 4 In S106, the tracking object setting unit 122 sets the tracking flag to ON. The value of the tracking flag is recorded in the main storage device 102, etc., and is retained even if the processing proceeds to the next frame without being initialized. The processing returns to S101, and tracking processing for the next motion picture frame begins. In S102, the tracking flag is set to ON, therefore, the processing proceeds to S107.

[0075] In S107, the first feature tracking unit 123 obtains a tracking result based on the color features of the frame image. Here, refer to... Figures 6 to 8 This section explains tracking based on the color features of the tracked object. Figure 6 This diagram illustrates color-based tracking. The first frame includes the tracking object 601 defined in S105. In tracking example A, the object 602 has a similar color to the tracking object 601. In tracking example B, the object 603 has a different color from the tracking object 601.

[0076] In color-based tracking, objects whose colors are closer to the tracked object are considered the tracking result. Figure 6 In the example, if there are both object 602 and object 603 in the frame image, the tracking result determines that the object 602 has a closer color.

[0077] Figure 7This diagram illustrates a specific example of color feature-based tracking. The tracking object setting unit 122 sets the tracking object 701 in the first frame. The first feature tracking unit 123 learns (trains) a region including the tracking object 701 as a learning region, and tracks a region with a similar color to the tracking object 701 in the second frame. Since the color of object 702 is similar to that of object 703 compared to the tracking object 701, it is determined to be a tracking result.

[0078] Figure 8 This diagram illustrates the generation of a hierarchical distinction map of color features. The tracking object setting unit 122 sets the tracking object 801 in the first frame. The first feature tracking unit 123 learns the tracking object 801. The first feature tracking unit 123 tracks the second frame and calculates the difference between the learned color of the tracking object 801 and the color of the second frame. The color difference can be calculated, for example, by comparing the average color of the pixels contained in the tracking object 801 with the color of each pixel in the second frame.

[0079] The first feature tracking unit 123 generates a color feature hierarchy map based on the color distances calculated from each pixel of the second frame. Figure 8 The layer distinction map 802 shown is a schematic representation of the layer distinction map generated by the first feature tracking unit 123. In the color feature layer distinction map 802, the smaller the color distance, the higher the likelihood. In color feature-based tracking, the center coordinate 803 of the region in the layer distinction map 802 where the tracking position of the tracked object 801 in the second frame can be determined as having the highest likelihood is selected. The center coordinate 803 is also called the peak position 803 of the layer distinction map 802.

[0080] In human detection, clothing color varies significantly between individuals compared to shape features. Therefore, color features can be tracked using simpler algorithms than shape features. Furthermore, Figure 8 An example of creating a hierarchy map 802 based on color distance is shown, but it is not limited to this. The first feature tracking unit 123 may also create the hierarchy map 802 based on the similarity of the histogram of the color of the tracked object 801.

[0081] exist Figure 4 In step S108, the second feature tracking unit 124 obtains tracking results based on shape features of the frame image. Here, refer to... Figures 9 to 11 This section explains tracking based on the shape features of the object being tracked. Figure 9 This diagram illustrates shape-based tracking. The first frame contains the tracking object 901 set in S105. In tracking example A, the shape of object 902 is similar to that of tracking object 901. In tracking example B, the shape of object 903 is different from that of tracking object 901.

[0082] In shape-based tracking, objects whose shapes are closer to the tracked object are considered the tracking result. Figure 9 In the example, if there are both object 902 and object 903 in the frame image, the tracking result determines that the object 902 has a closer shape.

[0083] Figure 10 This diagram illustrates a specific example of shape-feature-based tracking. The tracking object setting unit 122 sets the tracking object 1001 in the first frame. The second feature tracking unit 124 uses the region including the tracking object 1001 as a learning region and tracks a region with a similar shape to the tracking object in the second frame. Since object 1002 has a similar shape to the tracking object 1001 compared to object 1003, it is determined to be a tracking result.

[0084] Figure 11 This diagram illustrates the generation of a hierarchical distinction map of shape features. The tracking object setting unit 122 sets the tracking object 1101 in the first frame 1110. Frames 1111 and 1112 are obtained by moving the tracking object 1101 by 1 pixel and 2 pixels in the positive X-axis direction in the first frame 1110, respectively.

[0085] The second feature tracking unit 124 overlaps the first frame 1110, frame 1111, and frame 1112 with the second frame 1120. Frames 1130, 1131, and 1132 are frames that overlap the first frame 1110, frame 1111, and frame 1112 with the second frame 1120 respectively.

[0086] In frame 1130, the second feature tracking unit 124 assigns a likelihood of similar shape to the position (e.g., center coordinates) of the unmoved tracking object 1101 based on the difference between the unmoved tracking object 1101 and the second frame 1120. For example, the second feature tracking unit 124 inputs the region of the second frame at the same position as the tracking object 1101 into a recognizer that learns the tracking object 1101 through machine learning, thereby obtaining a likelihood of similar shape.

[0087] Similarly, in frame 1131, the second feature tracking unit 124 assigns a similar likelihood of shape to the position of the tracked object 1101 after it has moved one pixel along the positive X-axis, based on the difference between the tracked object 1101 that has moved one pixel along the positive X-axis and the second frame 1120. Furthermore, in frame 1132, the second feature tracking unit 124 assigns a similar likelihood of shape to the position of the tracked object 1101 after it has moved two pixels along the positive X-axis, based on the difference between the tracked object 1101 that has moved two pixels along the positive X-axis and the second frame 1120.

[0088] Thus, by moving the tracked object 1101 within the second frame to obtain the difference, the second feature tracking unit 124 can generate a hierarchical differentiation map of shape features. For example... Figure 11 The hierarchical differentiation diagram 1140 shown is a schematic representation of the hierarchical differentiation diagram generated by the second feature tracking unit 124. In the hierarchical differentiation diagram 1140 of shape features, it is set that the smaller the difference (the greater the likelihood of similar shapes), the higher the likelihood.

[0089] In addition, Figure 11 Although examples are shown of moving the tracked object 1101 by 1 pixel and 2 pixels in the positive X-axis direction to obtain the difference, the second feature tracking unit 124 can obtain the difference by moving the tracked object 1101 to each pixel within the second frame. By obtaining the difference for each pixel in the entire second frame, the second feature tracking unit 124 can generate a hierarchical differentiation map 1140.

[0090] Furthermore, when the movement range of the tracked object 1101 is limited, the second feature tracking unit 124 can, for example, move the tracked object 1101 within the range of -10 to +10 along the X-axis and within the range of -10 to +10 along the Y-axis to obtain the difference. The second feature tracking unit 124 can set the maximum value of the difference in the region (pixel) where the tracked object 1101 is not moved, thereby generating a layer distinction map 1140.

[0091] In shape-based tracking, the tracking position of the object 1101 in the second frame can be determined as the center coordinate 1141 of the region with the highest likelihood in the hierarchy map 1140. The center coordinate 1141 is also called the peak position 1141 of the hierarchy map 1140.

[0092] exist Figure 11 In the example, a simplified algorithm was used to illustrate shape feature-based tracking, but the second feature tracking unit 124 can use a filter called KCF (Kernelized Correlation Filter) for shape feature-based tracking. KCF performs a Fourier transform on the image and calculates it in the spectral space, thus reducing the computational cost. Furthermore, since KCF uses regression instead of difference for calculation, it is a method that can improve robustness.

[0093] Furthermore, shape feature tracking can also be achieved by extracting features using convolution operations instead of the spectral space, and then comparing them in the feature space. While this method may increase computational cost compared to using KCF, it can achieve higher tracking accuracy.

[0094] exist Figure 4In S109, the tracking management unit 125 calculates the mixing ratio of the tracking results of the color features obtained in S107 and the tracking results of the shape features obtained in S108. Here, refer to... Figure 12 The calculation of the mixing ratio is explained.

[0095] The blending rate is calculated based on the distance from the center coordinate 1201 of the frame image to the human body designated as the tracking object. For example, the blending rate of the blended shape features can be calculated using Equation 1 below. The blending rate of the shape features is a value in the range of 0.0 to 1.0.

[0096] The mixing rate of shape features = 1.0 - (distance to the tracked object / d1) × α… (Equation 1)

[0097] In Equation 1, d1 is the maximum distance from the center coordinate 1201 of the frame image to the frame boundary. Figure 12 In the example, the distance to the tracked object is d2 in the case of tracked object 1202 which exists near the outer periphery of the frame image, and d3 in the case of tracked object 1203 which exists near the center of the frame image.

[0098] α is a weighted coefficient corresponding to the characteristics of the tracked object, and can be set to, for example, 0.7. When hand and foot movements, such as in factory work, affect the tracking accuracy of shape features, the coefficient α can be set lower than 0.7. Furthermore, when the uniform is a characteristic color such as red, the coefficient α relative to the color feature's mixing rate can be set higher than 0.7. Thus, the coefficient α can be changed according to the specific characteristics of the tracked object.

[0099] The mixing ratio of color features can be calculated using Equation 2. The mixing ratio of shape features uses the value calculated using Equation 1.

[0100] The mixing rate of color features = 1.0 - the mixing rate of shape features... (Equation 2)

[0101] Furthermore, the calculation method for the mixing ratio of shape and color features is not limited to using Equations 1 and 2. Equation 1 simply requires a relationship that the greater the distance from the center coordinate 1201 of the frame image to the tracked object, the lower the mixing ratio of the shape features; it is not limited to a linear equation, but can be a quadratic or higher equation, or even a nonlinear one. Alternatively, the mixing ratio of the color features can be calculated first, similarly to Equation 2, by using the difference between 1.0 and the mixing ratio of the color features. In this case, the equation for calculating the mixing ratio of the color features becomes a relationship where the greater the distance to the tracked object, the higher the mixing ratio of the color features.

[0102] In S110, the tracking management unit 125, based on the mixing rate calculated in S109, mixes the tracking results regarding the first feature and the tracking results regarding the second feature. Here, refer to... Figure 13 This section explains the mixing of features.

[0103] Figure 13 This example illustrates how a layered distinction map 802 of color features and a layered distinction map 1140 of shape features are mixed to generate a layered distinction map 1301 of blended features. The tracking management unit 125 mixes the corresponding pixels of the layered distinction map 802 of color features and the layered distinction map 1140 of shape features using the following formula 3 to generate the layered distinction map 1301 of blended features.

[0104] Mixing feature = Color feature × Mixing rate of color feature + Shape feature × Mixing rate of shape feature… (Equation 3)

[0105] Figure 13 An example is shown where the color feature blending rate is 0.2 and the shape feature blending rate is 0.8. In the blending feature hierarchy map 1301, the peak position 1140 of the shape feature hierarchy map 1140 is greater than the peak position 803 of the color feature hierarchy map 802; therefore, peak position 1140 is determined as tracking result 1302. Tracking result 1302 is set as peak position 1302 of the blending feature hierarchy map 1301.

[0106] also, Figure 13 While an example of mixing a hierarchical differentiation map of the first feature and a hierarchical differentiation map of the second feature is shown, it is not limited to this. The tracking management unit 125 may also use other methods, such as using the position after proportionally allocating the peak positions of each hierarchical differentiation map according to a mixing rate as the detection position of the tracked object, to mix the tracking results of the first feature and the second feature.

[0107] exist Figure 4 In step S111, the tracking management unit 125 determines whether a tracking object is detected in the hierarchical differentiation map 1301, which is a mixing result. Figure 13 In the example, the tracking management unit 125 can determine that the peak position 1302 is the location of the tracked object. Furthermore, if the tracking management unit 125 does not detect a peak position in the hierarchical differentiation map 1301 of mixed features, for example, if there is no position in the hierarchical differentiation map 1301 greater than or equal to a predetermined threshold, it can determine that no tracked object has been detected.

[0108] If a tracked object is detected from the blending result (S111: Yes), the process proceeds to S112. If no tracked object is detected from the blending result (S111: No), the process proceeds to S113.

[0109] In S112, the output unit 13 outputs the tracking result. The output unit 13 displays the detection box overlapping the position of the tracked object detected in S111. The size of the detection box can be set, for example, to the size of the tracked object set in S105. If the tracking result is output, the process returns to S101, and tracking processing for the next motion picture frame begins.

[0110] In S112, the tracking management unit 125 sets the tracking marker to OFF. The process returns to S103, and new tracking targets are set through the processes in S103 to S106. Additionally, Figure 4 The processing described above represents an example of setting a new tracking object when tracking of the tracking object fails, but the timing of setting the tracking object is not limited to this. For example, the tracking object setting unit 122 may also perform human detection and reset the tracking object every predetermined number of frames or every predetermined time interval.

[0111] (Effects)

[0112] In the above embodiment, the image processing apparatus 1 sets the mixing ratio of the tracking results based on color features and the tracking results based on shape features based on the relative position of the tracked object relative to the camera. Specifically, in the outer periphery far from the center of the image, since color features such as the color of clothing are stable, the mixing ratio of color features is set to be higher than the mixing ratio of shape features. On the other hand, in the vicinity of the center of the image, hands and feet are difficult to see, and shape features are stable; therefore, the mixing ratio of shape features is set to be higher than the mixing ratio of color features.

[0113] In this way, by changing the mixing ratio of multiple features of the tracked object according to its position within the camera range, the image processing device 1 can track the tracked object with high precision. Specifically, when performing dynamic line analysis using images captured by a fisheye camera, the mixing ratio of color features is set higher in the peripheral portion far from the center of the camera range, thus improving the tracking accuracy of the tracked object.

[0114] <Other>

[0115] Furthermore, the above-described embodiments are merely illustrative examples of the structure of the present invention. The present invention is not limited to the specific embodiments described above, and various modifications can be made within the scope of its technical concept.

[0116] For example, in the above embodiment, the case where camera 2 is a fisheye camera was described, but it is not limited to fisheye cameras. Camera 2 is any imaging device capable of taking pictures from either an overhead view or a sideways view of the tracked object, depending on the position of the tracked object. In addition, camera 2 is not limited to being installed on the ceiling, as long as it is installed in a location where it can be viewed from above to take pictures of the tracked object.

[0117] Furthermore, in the above embodiment, the image processing device 1 sets the blending rate based on the distance from the center of the camera range to the tracked object, but it is not limited to this. The image processing device 1 may also estimate the position of the tracked object based on the distance measured by the camera 2 to the tracked object, or the distance between the top of the human head and the toes, and set the blending rate accordingly.

[0118] Furthermore, the image processing device 1 is not limited to the position of the tracked object. In cases where the appearance of the tracked object changes due to variations in posture, such as when a person is lying down, the mixing ratio can be set based on the orientation of the tracked object relative to the shooting surface of the captured image. The orientation of the tracked object relative to the shooting surface can be estimated, for example, based on the shape and size of the tracked object.

[0119] Furthermore, the image processing device 1 can also set the blending rate based on the difference from a pre-prepared background image. When the color features of the tracked object are significant relative to the background image, the blending rate can be set in a way that prioritizes the color features. For example, when tracking a person wearing a black shirt against a black background, the color features are not significantly represented, but when tracking a person wearing red or blue clothing against a black background, the color features are significantly represented. Therefore, when comparing with a pre-prepared background area, if the color features (red clothing, blue clothing) are significant relative to the background, the blending rate is set to prioritize the color features.

[0120] <Postscript 1>

[0121] (1) An image processing apparatus comprising:

[0122] The tracking object setting unit (122) sets the tracking object in the first frame of the moving image;

[0123] The first feature tracking unit (123) tracks the tracking object in the second frame based on the first feature of the tracking object set in the first frame;

[0124] The second feature tracking unit (124) tracks the tracking object in the second frame based on the second feature of the tracking object set in the first frame;

[0125] The tracking management unit (125) mixes the tracking results based on the first feature tracking unit and the tracking results based on the second feature tracking unit at a predetermined mixing ratio; and

[0126] The output unit (13) outputs the detection position of the tracked object in the second frame based on the mixing result of the tracking management unit.

[0127] <Appendix 2>

[0128] An image processing method, comprising:

[0129] To make the computer perform:

[0130] Tracking object setting step (S105): The tracking object is set in the first frame of the motion picture;

[0131] The first feature tracking step (S107) involves tracking the tracking object in the second frame based on the first feature of the tracking object set in the first frame.

[0132] The second feature tracking step (S108) involves tracking the tracking object in the second frame based on the second feature of the tracking object set in the first frame.

[0133] The tracking management steps (S109, S110) mix the tracking results from the first feature tracking step and the tracking results from the second feature tracking step at a predetermined mixing ratio; and

[0134] The output step (S112) outputs the detection position of the tracked object in the second frame based on the mixing result mixed in the tracking management step.

[0135] Explanation of reference numerals in the attached figures

[0136] 1: Image processing device; 2: Camera; 11: Image acquisition unit; 12: Processing unit; 121: Detection unit; 122: Tracking object setting unit; 123: First feature tracking unit; 124: Second feature tracking unit; 125: Tracking management unit; 13: Output unit.

Claims

1. An image processing apparatus comprising: The tracking object setting unit sets the tracking object in the first frame of the moving image; The first feature tracking unit tracks the tracking object in the second frame based on the color feature of the tracking object set in the first frame, i.e., the first feature. The second feature tracking unit tracks the tracking object in the second frame based on the shape feature, i.e., the second feature, of the tracking object set in the first frame. The tracking management department mixes the tracking results based on the first feature tracking department and the tracking results based on the second feature tracking department at a predetermined mixing rate; as well as The output unit, based on the mixing result obtained by the tracking management unit, outputs the detection position of the tracked object in the second frame. The first feature tracking unit calculates the color distance between the tracked object and its color in the second frame, and generates a likelihood map based on the color distance, representing the likelihood of the tracked object's location in the second frame. The second feature tracking unit calculates the difference between the shape of the image that offsets the position of the tracked object in the first frame and the shape of the image in the second frame, and generates a likelihood map representing the likelihood of the position of the tracked object in the second frame based on the difference.

2. The image processing apparatus according to claim 1, wherein, The tracking result based on the first feature is a likelihood map representing the likelihood of the location of the tracked object in the second frame, calculated based on the first feature. The tracking result based on the second feature is a likelihood map representing the likelihood of the location of the tracked object in the second frame, calculated based on the second feature. The output unit outputs the position with the highest likelihood in the likelihood map of the mixed result as the detection position of the tracked object in the second frame.

3. The image processing apparatus according to claim 1 or 2, wherein, The specified mixing ratio is set based on the position of the tracked object within the camera's field of view of the first frame.

4. The image processing apparatus according to claim 1 or 2, wherein, The specified mixing ratio is set based on the orientation of the tracked object relative to the shooting surface of the first frame.

5. The image processing apparatus according to claim 1 or 2, wherein, It also includes a detection unit that detects the tracked object from the first frame.

6. The image processing apparatus according to claim 1 or 2, wherein, The tracking object setting unit obtains the distance from the center position of the camera range of the first frame to the tracking object. The tracking management department sets the prescribed mixing rate based on the distance.

7. The image processing apparatus according to claim 1, characterized in that, The tracking management unit sets the specified mixing rate so that the greater the distance from the center of the camera range of the first frame to the tracked object, the greater the mixing rate of the color features.

8. The image processing apparatus according to claim 1, wherein, The second feature tracking unit uses KCF (Kernelized Correlation Filter) to generate a likelihood map representing the likelihood of the location of the tracked object in the second frame.

9. The image processing apparatus according to claim 1 or 2, wherein, The output unit outputs a detection box centered on the position of the tracked object in the second frame, which is the same size as the box surrounding the tracked object in the first frame.

10. An image processing method, comprising: To make the computer perform: The tracking object setting steps involve setting the tracking object in the first frame of the moving image; The first feature tracking step involves tracking the tracking object in the second frame based on the color feature of the tracking object set in the first frame, i.e., the first feature. The second feature tracking step involves tracking the tracking object in the second frame based on the shape features, i.e., the second features, of the tracking object set in the first frame. The tracking management step mixes the tracking results from the first feature tracking step and the tracking results from the second feature tracking step at a specified mixing ratio; as well as The output step, based on the mixing result from the tracking management step, outputs the detection position of the tracked object in the second frame. In the first feature tracking step, the color distance between the tracked object and the color in the first frame is calculated in the second frame. Based on the color distance, a likelihood map representing the likelihood of the tracked object's location in the second frame is generated. In the second feature tracking step, the difference between the shape of the image and the image that offset the position of the tracked object in the first frame is calculated in the second frame, and a likelihood map representing the likelihood of the position of the tracked object in the second frame is generated based on the difference.

11. A program product for causing a computer to perform the steps of the method of claim 10.

Images