Image processing device, image processing method, and imaging device

Abstract

To provide a technique for correctly identifying an object being a tracking object and another object and tracking the object even when there exists an object that has the similar appearance to the object being the tracking object.SOLUTION: An image processing device obtains a loss on the basis of a distance between images of a feature amount of a tracking object in each image, a distance between feature amounts of the tracking object and a non-tracking object in the image and an overlapping ratio of an image region of the tracking object to an image region of the non-tracking object in one or more images, performs learning of a model for extracting a feature amount of the object from the image on the basis of the loss, and performs tracking processing of tracking the tracking object in the image on the basis of the learned model.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technique for tracking an object in an image. [Background technology]

[0002] There are several techniques for tracking objects in an image, including those that use luminance or color information, those that use template matching, and those that use a deep neural network. One method that uses a deep neural network is called tracking by detection. In this method, an object in an image is detected and tracked by associating the object with a past object. In Non-Patent Document 1, this association is performed using features obtained from a neural network trained by metric learning. Metric learning is a method for learning a transformation into a space in which data with a higher similarity is distributed closer to each other in space. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-026108 A [Non-patent literature]

[0004] [Non-Patent Document 1] Nicolai Wojke, Alex Bewley, Dietrich Paulus: Simple online and realtime tracking with a deep association metric(ICIP2017) Summary of the Invention [Problem to be solved by the invention]

[0005] We consider a method of tracking by extracting features corresponding to each object using a neural network trained by metric learning and associating them with past features. However, in such a method, if the posture of the object to be tracked changes or it is occluded, the object to be tracked may be mistakenly associated with other objects that look similar, resulting in failure of tracking.

[0006] The present invention provides a technology for correctly identifying an object to be tracked from other objects and tracking the object, even when an object that looks similar to the object to be tracked is present. [Means for solving the problem]

[0007] One aspect of the present invention is characterized by comprising a calculation means for calculating a loss based on the distance between images of feature amounts of a tracked object in each image, the distance between the feature amounts of the tracked object and non-tracked object in an image, and the overlap rate between image areas of the tracked object and image areas of non-tracked objects in one or more images, a learning means for learning a model for extracting feature amounts of an object from an image based on the loss, and a tracking means for performing a tracking process to track the tracked object in an image based on the model learned by the learning means. Effect of the Invention

[0008] According to the configuration of the present invention, even if there is an object that looks similar to the object to be tracked, the object to be tracked can be correctly identified from the other objects and tracked. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram showing an example of the functional configuration of the learning device 100. [Diagram 2] 4 is a flowchart of a learning process of a learning model by the learning device 100. [Diagram 3] A diagram showing the relationship between a CNN for object detection and a CNN for feature extraction. [Figure 4] 5A to 5C are diagrams illustrating a process for acquiring a feature amount of a tracking target object. [Diagram 5] FIG. 13 is a diagram for explaining the process for determining d1, d2, r, and m. [Figure 6] An image showing the distance between features in two-dimensional space. [Figure 7] FIG. 2 is a block diagram showing an example of the functional configuration of a tracking device 200. [Figure 8] 4 is a flowchart of a tracking process of a tracking target object by the tracking device 200. [Figure 9] FIG. 11 is a diagram showing an example of the configuration of correct answer data according to the first modification example. [Figure 10] 10A to 10C are diagrams illustrating a method for acquiring feature amounts of a tracking target object and a non-tracking target object according to Modification 2. [Figure 11] A diagram showing how to calculate IoU1 and IoU2. [Figure 12] FIG. 2 is a block diagram showing an example of the hardware configuration of a computer apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [First embodiment] First, we will explain an image processing device as a learning device that learns a learning model used to correctly identify a tracking target (tracking target object) and other objects (non-tracking target objects) in continuously captured images and track the tracking target object.

[0012] An example of the functional configuration of a learning device 100 according to this embodiment is shown in the block diagram of Fig. 1. The learning process of a learning model by the learning device 100 having the example functional configuration shown in Fig. 1 will be described with reference to the flowchart of Fig. 2.

[0013] In step S101, the acquisition unit 110 acquires a first image and correct answer data (first correct answer data) of the first image. The first correct answer data is created in advance (known) for the tracking target object in the first image, and includes area information indicating the image area of ​​the tracking target object in the first image, and a label unique to the tracking target object. For example, when the image area is a rectangular area, the area information indicating the image area of ​​the tracking target object is information including the center position of the rectangular area (the position of the tracking target object), the width and the height of the rectangular area. Note that here, when the image area of ​​the tracking target object is a rectangular area, the center position of the rectangular area is set as the position of the tracking target object, but any position of the rectangular area may be set as the position of the tracking target object.

[0014] The acquisition unit 110 also acquires a second image captured following the first image, and the correct answer data (second correct answer data) of the second image. The second correct answer data, like the first correct answer data, is created in advance (known) for the tracking target object in the second image, and includes area information indicating the image area of ​​the tracking target object in the second image, and a label unique to the tracking target object. The same label is assigned to the same tracking target object in the first image and the second image.

[0015] The first image and the second image are, for example, an image of a first frame in a moving image and an image of a second frame following the first frame, respectively. Also, for example, the first image and the second image are, for example, a still image of interest in a plurality of still images captured periodically or irregularly, and a still image captured after the still image of interest, respectively. Also, processing such as cutting out a part of the acquired image may be performed as necessary.

[0016] In step S102, the object detection unit 120 inputs the first image to a CNN (Convolutional Neural Network) for object detection, which is a learning model for object detection, and executes arithmetic processing of the CNN for object detection. As a result, the object detection unit 120 acquires, for each object in the first image, region information indicating the image region of the object and a score of the object as a detection result of the object. For example, when the image region is a rectangular region, the region information indicating the image region of the object is information including the center position of the rectangular region, the width and height of the rectangular region. In addition, the score of the object is a score with a value range of 0 to 1, and the larger the score, the higher the object-likeness. Examples of technologies for detecting objects in images include "Liu, SSD: Single Shot Multibox Detector: ECCV2016" and the like.

[0017] In step S103, the feature extraction unit 130 inputs the first image into a feature extraction CNN, which is a learning model for feature extraction, and executes the computation process of the feature extraction CNN. As a result, the feature extraction unit 130 acquires a feature map (first feature map) that represents the feature of each region when the first image is equally divided vertically and horizontally. Then, the feature extraction unit 130 identifies a corresponding position on the first feature map that corresponds to the center position of the image region of the tracking target object indicated by the first correct answer data, and acquires the feature at the identified corresponding position as the "feature of the tracking target object". In addition, the feature extraction unit 130 identifies a corresponding position on the first feature map that corresponds to the center position of the image region indicated by the region information acquired in step S102, and acquires the feature at the identified corresponding position as the "feature of the non-tracking target object".

[0018] Similarly, the feature extraction unit 130 inputs the second image to a feature extraction CNN, which is a learning model for feature extraction, and executes arithmetic processing of the feature extraction CNN. In this way, the feature extraction unit 130 acquires a feature map (second feature map) that indicates the feature of each region when the second image is equally divided vertically and horizontally. The feature extraction unit 130 then identifies a corresponding position on the second feature map that corresponds to the center position of the image region of the tracking target object indicated by the second correct answer data, and acquires the feature at the identified corresponding position as the "feature of the tracking target object."

[0019] Here, the process for acquiring the feature amount of the tracking target object will be described with reference to Fig. 4. Fig. 4 shows a state where a first image having a size of 640 pixels horizontal x 480 pixels vertical is superimposed with a first feature amount map, which is a map in which the feature amounts of each divided area when the first image is divided into 24 divided areas horizontal x 16 divided areas vertically are registered. The first feature amount map is a map in which (24 x 16) feature amounts with the number of channels C (any natural number) are registered. Since the position of the upper left corner of the first image is the origin (0, 0), the position (x, y) on the first image is in the range of 0≦x≦639, 0≦y≦439, and the position (p, q) on the first feature amount map (position of the divided area unit) is in the range of 0≦p≦23, 0≦q≦15.

[0020] Position 402 is the position (x, y)=(312, 276) of the tracking target object indicated by the first correct answer data. At this time, the feature extraction unit 130 calculates (312x24 / 640)=11.7, and sets 11 obtained by deleting the decimal part of 11.7 as p, and calculates (276x16 / 480)=9.2, and sets 9 obtained by deleting the decimal part of 9.2 as q. In this way, the feature extraction unit 130 obtains the position (11, 9) on the first feature map corresponding to the position (312, 276) of the tracking target object on the first image. A rectangle 403 indicates the position (11, 9) on the first feature map corresponding to the position (312, 276) of the tracking target object on the first image. Then, the feature extraction unit 130 acquires the feature at position 403 (the 1x1xC array element at position 403) as the feature of the tracking target object in the first image.

[0021] The feature extraction unit 130 also performs the same process on the second image to obtain the feature of the tracking target object in the second image. That is, the feature extraction unit 130 obtains a position on the second feature map corresponding to the position of the tracking target object indicated by the second ground truth data, and obtains the feature at the obtained position (the 1x1xC array element at the obtained position) as the feature of the tracking target object in the second image.

[0022] In addition, when identifying the corresponding position on the second feature map that corresponds to the center position of the image region indicated by the region information acquired in step S102, the above method of finding (p, q) from (x, y) is used.

[0023] Here, the relationship between the CNN for object detection and the CNN for feature extraction will be described with reference to FIG. 3. The CNN for object detection and the CNN for feature extraction may be implemented in any configuration. For example, as shown in FIG. 3(a), the CNN for object detection and the CNN for feature extraction may be provided separately, and each may be configured to operate independently on the input image (first image, second image). Also, as shown in FIG. 3(b), a configuration common to the CNN for object detection and the CNN for feature extraction may be provided as a shared CNN, and the configuration not included in the shared CNN in the CNN for object detection may be re-defined as the CNN for object detection, and the configuration not included in the shared CNN in the CNN for feature extraction may be re-defined as the CNN for feature extraction. In this case, the input image is input to the shared CNN, and the shared CNN inputs the processing result for the input image to the CNN for object detection and the CNN for feature extraction. The CNN for object detection operates using the processing result output from the shared CNN as input, and the CNN for feature extraction operates using the processing result output from the shared CNN as input.

[0024] Next, in step S104, the loss calculation unit 140 uses the feature amounts calculated in step S103 to calculate a feature amount distance d1, which is the Euclidean distance between the feature amount of the tracking target object in the first image and the feature amount of the tracking target object in the second image. The loss calculation unit 140 also uses the feature amounts calculated in step S103 to calculate a feature amount distance d2, which is the Euclidean distance between the feature amount of the tracking target object in the first image and the feature amount of the non-tracking target object in the first image. The distance between the feature amounts is the distance between each of the feature amounts in the feature amount space to which each of the feature amounts belongs, and a method for calculating the distance between the feature amounts is well known, so a description thereof will be omitted.

[0025] Next, in step S105, the loss calculation unit 140 calculates parameters of a loss function. In this embodiment, the triplet loss given by the following (Equation 1) is used as the loss function.

[0026] loss=max(rd1-d2+m,0) … (Formula 1) In this loss function, loss occurs when d1 is large and d2 is small, so it is possible to acquire features such that the distance between features for the same object is smaller than the distance between features for different objects.

[0027] In this (Equation 1), r and m are parameters of the loss function, r is a distance parameter, and m is a margin parameter. When r is set large, learning is performed so that the distance between feature amounts of the same object becomes small, and when m is set large, learning is performed so that the distance between feature amounts of different objects becomes large. In this embodiment, these parameters are dynamically set using the ground truth data and the object area detected in step S102, thereby learning features with higher object identification performance.

[0028] The loss function to be used is not limited to a specific function as long as distance learning can be performed. For example, contrastive loss or softmax loss may be used as the loss function.

[0029] Here, the process for determining d1, d2, r, and m will be described with reference to a specific example shown in Fig. 5. Fig. 5(a) shows a first image, and Fig. 5(b) shows a second image.

[0030] 5(a) includes a tracking target object 501 and non-tracking target objects 502 and 503. A rectangle 506 is an image area of ​​the tracking target object 501 defined by the first ground truth data. A rectangle 507 is an image area of ​​the non-tracking target object 502 detected in step S102. A rectangle 508 is an image area of ​​the non-tracking target object 503 detected in step S102.

[0031] 5(b) includes a tracking target object 504 and a non-tracking target object 505. A rectangle 509 is an image area of ​​the tracking target object 504 defined by the second ground truth data, and a rectangle 510 is an image area of ​​the non-tracking target object 505 detected in step S102.

[0032] In such a case, the feature extraction unit 130 calculates an inter-feature distance d1 between a feature at a corresponding position on the first feature map corresponding to the "position of the tracking target object 501 in the first image" indicated by the first ground truth data, and a feature at a corresponding position on the second feature map corresponding to the "position of the tracking target object 504 in the second image" indicated by the second ground truth data.

[0033] The feature extraction unit 130 also selects one of the non-tracking target objects 502 and 503 (non-tracking target object 502 in FIG. 5(a)) as a selected object, and calculates a feature distance d2 between a feature at a corresponding position on the first feature map corresponding to the position detected in step S102 for the selected object and a feature at a corresponding position on the first feature map corresponding to the "position of the tracking target object 501 in the first image" indicated by the first correct answer data. When multiple non-tracking target objects are included in the first image, the method of selecting a selected object from the multiple non-tracking target objects is not limited to a specific selection method. For example, the non-tracking target object closest (closest) to the position of the tracking target object is selected as the selected object.

[0034] Furthermore, the feature extraction unit 130 obtains a distance parameter r using the following (Equation 2), and obtains a margin parameter m using the following (Equation 3).

[0035] r=1 … (Formula 2) m=m0+IoU … (Formula 3) Here, m0 is an arbitrary real number. IoU (Intersection over Union) is the overlapping ratio of the areas of two image regions. In the case of Fig. 5, the overlapping ratio of the areas of image region 506 of tracking target object 501 and image region 507 of non-tracking target object 502 selected as the selected object is calculated as IoU.

[0036] In step S106, the loss calculation unit 140 calculates the loss (loss) by calculating (Equation 1) using d1, d2, r, and m obtained in the above processing.

[0037] FIG. 6 is an image diagram showing the inter-feature distance in a two-dimensional space when the loss is calculated by the above-mentioned method according to the present embodiment. Feature 601 is the feature of the tracked object in the first image, and feature 602 is the feature of the tracked object in the second image. d1 is the inter-feature distance between feature 601 and feature 602. Feature 603 and feature 604 are the feature of the non-tracked object in the first image. d2 is the inter-feature distance between feature 601 and feature 604, and d2' is the inter-feature distance between feature 601 and feature 603. Moreover, m, m0, and IoU are margin parameters shown in (Equation 3).

[0038] In this method, when there is no overlap between the image area of ​​the tracking target object and the image area of ​​the non-tracking target object in the first image, the feature distance d2' is trained to be greater than the sum of the feature distance d1 and the margin parameter m0. On the other hand, when there is overlap between the image area of ​​the tracking target object and the image area of ​​the non-tracking target object in the first image, the feature distance d2 is trained to be greater than the sum of the feature distance d1 and the margin parameter m. For this reason, it is possible to train so that the feature distance becomes greater for a non-tracking target object having a high IoU with the image area of ​​the tracking target object, that is, a non-tracking target object that is close to the tracking target object and has a high risk of erroneous switching of tracking.

[0039] The loss function is not limited to the above format as long as the parameters are changed based on the IoU with the image region of the tracking target object. When calculating the loss, multiple sets of the first image and the second image may be prepared, the above process may be performed for each set to obtain the loss for each set, and the average of the losses obtained for each set may be used as the final loss.

[0040] In step S107, the learning unit 150 updates parameters (weight values, etc.) in the CNN for feature extraction so as to minimize the loss calculated in step S106, thereby learning the CNN for feature extraction. Note that the learning unit 150 may simultaneously learn the CNN for object detection in addition to learning the CNN for feature extraction. The parameters such as the weights may be updated, for example, using the backpropagation method.

[0041] Next, an image processing device as a tracking device that performs processing for tracking a tracking target object in an image using the above-mentioned trained CNN will be described. An example of the functional configuration of a tracking device 200 according to this embodiment is shown in the block diagram of Fig. 7. The tracking process of a tracking target object by the tracking device 200 having the example of the functional configuration shown in Fig. 7 will be described according to the flowchart of Fig. 8.

[0042] In step S201, the image acquisition unit 210 acquires one captured image. The one captured image is the most recently captured image among the captured images captured in succession, and may be, for example, the captured image of the latest frame in a moving image, or may be the most recently captured still image among still images captured periodically or irregularly.

[0043] In step S202, the object detection unit 220 inputs the captured image acquired in step S201 to the object detection CNN and executes the calculation process of the object detection CNN. As a result, the object detection unit 220 acquires, for each object in the captured image acquired in step S201, area information indicating the image area of ​​the object and a score of the object as a detection result of the object.

[0044] In step S203, the feature extraction unit 230 inputs the captured image acquired in step S201 into the feature extraction CNN and executes the arithmetic processing of the feature extraction CNN to acquire a feature map of the captured image. Then, for each object acquired in step S202, the feature extraction unit 230 specifies a corresponding position on the feature map that corresponds to the center position of the image area indicated by the area information of the object, and acquires the feature at the specified corresponding position as the "feature of the object."

[0045] If the captured image acquired in step S201 is the first captured image acquired since the start of the tracking process (the image captured at the initial time), the process proceeds to step S204. On the other hand, if the captured image acquired in step S201 is the second or subsequent captured image acquired since the start of the tracking process (the image is not the image captured at the initial time), the process proceeds to step S207.

[0046] In step S204, the feature amount storage unit 240 assigns a unique ID (object ID) to each object for the feature amount of the object acquired in step S203. For example, object IDs 1, 2, 3, ... are assigned to objects in descending order of score.

[0047] Then, in step S205 (if the process proceeds from step S204 to step S205), feature amount storage unit 240 stores in tracking device 200 the feature amounts of each object to which an object ID was assigned in step S204.

[0048] In step S207, the feature matching unit 250 selects one of the features (features at the current time) of each object acquired in step S203 that has not been selected as a selected feature. Then, the feature matching unit 250 acquires n (n is a natural number) feature amounts (stored features) for each object from the features stored in the tracking device 200 in order of closest to the current time. Then, the feature matching unit 250 performs feature matching. DepartmentThe feature matching unit 250 calculates the inter-feature distance between each of the acquired stored features and the selected feature (the method of calculating the inter-feature distance is the same as above). Note that the feature matching unit 250 may calculate, for each object, one stored feature (for example, an average value of the n stored features) from the n stored features acquired for the object, and calculate the inter-feature distance between the one stored feature calculated for the object and the selected feature.

[0049] In step S208, the feature matching unit 250 identifies the smallest inter-feature distance among the inter-feature distances calculated in step S207. The feature matching unit 250 then assigns the object ID assigned to the object of the stored feature corresponding to the identified smallest inter-feature distance to the object of the selected feature. If the "object ID assigned to the object of the stored feature corresponding to the smallest inter-feature distance among the inter-feature distances calculated in step S207" has been previously assigned to another object, the feature matching unit 250 issues a new object ID and assigns it to the object of the selected feature. The object IDs may be assigned in descending order from the object with the highest score, but this is not limiting.

[0050] By performing the process of step S207 and the process of step S208 on the feature amounts of all objects acquired in step S203, it is possible to assign object IDs to all objects detected in step S202. Note that the above process for assigning object IDs to feature amounts at the current time is an example, and is not intended to be limiting. For example, the Hungarian algorithm may be used. Also, in steps S207 and S208, the following process may be performed.

[0051] In step S207, the feature matching unit 250 selects one of the features (features at the current time) of each object acquired in step S203 that has not been selected as a selected feature. Then, the feature matching unit 250 calculates the inter-feature distance between the feature (stored feature) of the object previously stored in the tracking device 200 and the selected feature (the method of calculating the inter-feature distance is the same as above).

[0052] In step S208, the feature matching unit 250 identifies an inter-feature distance that is less than the threshold value among the inter-feature distances calculated in step S207. When the feature matching unit 250 identifies a plurality of inter-feature distances that are less than the threshold value, it identifies the smallest inter-feature distance among the identified plurality of inter-feature distances. The feature matching unit 250 then assigns to the object of the selected feature the object ID that is assigned to the object of the stored feature that was saved most recently among the stored features corresponding to the identified inter-feature distances. When there is no inter-feature distance less than the threshold value among the inter-feature distances calculated in step S207, the feature matching unit 250 issues a new object ID and assigns it to the object of the selected feature.

[0053] In other words, if the inter-feature distance between the feature at the current time and a feature of interest previously stored in tracking device 200 is less than a threshold, the feature at the current time is assigned the object ID of the object in the feature of interest in order to match the object in the feature at the current time with the object in the feature of interest.

[0054] Then, in step S205 (if the process proceeds from step S208 to step S205), feature amount storage unit 240 stores in tracking device 200 the feature amounts of each object to which an object ID was assigned in step S208.

[0055] Furthermore, the feature matching unit 250 outputs the object ID stored in the tracking device 200 in step S205 and the image area of ​​the object to which the object ID is assigned as a tracking result. The tracking result may be displayed by the tracking device 200 or another device by superimposing a frame indicating the image area of ​​the object included in the tracking result on the captured image together with the object ID of the object, or may be transmitted to an external device. Note that the process according to the flowchart in FIG. 8 is performed for each captured image to be subjected to tracking process.

[0056] In this manner, in this embodiment, learning is performed so that the distance between the features of a non-tracking target object becomes larger as the non-tracking target object is closer to the tracking target object in the captured image and is more likely to cause an erroneous switching of tracking. This makes it possible to acquire features with higher object identification performance, and to perform tracking with high accuracy even when there are objects that look similar.

[0057] <Variation 1> In this modification, a case will be described in which correct answer data is assigned to each object (tracking target object and / or non-tracking target object) in an image. The correct answer data according to this modification will be described with reference to FIG. Fig. 9(a) shows an example of the configuration of correct answer data as a table in which the center positions (horizontal position coordinates, vertical position coordinates), widths, heights, and labels of image regions 904, 905, and 906 of objects 901, 902, and 903 in the image of Fig. 9(b) are registered. Note that it is assumed that at least one object with the same label exists in each of the first and second images.

[0058] When calculating the loss in the loss calculation unit 140, an object with a certain label is regarded as a tracking target object, and an object with any other label is regarded as a non-tracking target object. The above calculation may also be performed for objects with different labels, and the final loss may be determined by taking the average of these.

[0059] <Variation 2> In this modification, a method for acquiring features of a tracking target object and a non-tracking target object by feature extraction unit 130 will be described with reference to Fig. 10 as an example. In image 1001, region 1002 is an image region of the tracking target object indicated by the correct answer data, and regions 1003 and 1004 are image regions of the non-tracking target objects detected by object detection unit 120. Image 1005 is an image within region 1002, image 1006 is an image within region 1003, and image 1007 is an image within region 1004.

[0060] The feature extraction unit 130 converts (resizes, etc.) the images 1005 to 1007 into images of a specified image size (32 pixels x 32 pixels in this case). That is, the feature extraction unit 130 converts the image 1005 into an image 1008 of 32 pixels x 32 pixels, the image 1006 into an image 1009 of 32 pixels x 32 pixels, and the image 1007 into an image 1010 of 32 pixels x 32 pixels.

[0061] Then, the feature extraction unit 130 inputs the image 1008 into the CNN for feature extraction and executes the arithmetic processing of the CNN for feature extraction, thereby acquiring the feature of the object corresponding to the image 1008. The feature extraction unit 130 also inputs the image 1009 into the CNN for feature extraction and executes the arithmetic processing of the CNN for feature extraction, thereby acquiring the feature of the object corresponding to the image 1009. The feature extraction unit 130 also inputs the image 1010 into the CNN for feature extraction and executes the arithmetic processing of the CNN for feature extraction, thereby acquiring the feature of the object corresponding to the image 1010. The feature of the object may be acquired as a vector with an arbitrary number of dimensions.

[0062] By performing such processing on the first image and the second image, it is possible to obtain the feature amounts of the object in each image, and thereafter, the same processing as in the first embodiment may be performed.

[0063] <Variation 3> In this modification, the distance parameter r and the margin parameter m are calculated according to (Equation 4) and (Equation 5), respectively.

[0064] r=1+IoU1·IoU2 (Formula 4) m=m0 (formula 5) Here, m0 is an arbitrary positive real number. The calculation method of IoU1 and IoU2 will be described with reference to Fig. 11. Fig. 11(a) shows the first image, and (b) shows the second image.

[0065] 11(a), the first image includes a tracking target object 1101 and a non-tracking target object 1102 (if the first image includes multiple non-tracking target objects, the non-tracking target object closest to the tracking target object 1101). IoU1 is the IoU between an image region 1105 of the tracking target object 1101 and an image region 1106 of the non-tracking target object 1102.

[0066] 11(b), the second image includes a tracking target object 1103 and a non-tracking target object 1104 (if the second image includes multiple non-tracking target objects, the non-tracking target object closest to the tracking target object 1103). IoU2 is the IoU between an image region 1107 of the tracking target object 1103 and an image region 1108 of the non-tracking target object 1104.

[0067] In other words, if there is a non-tracked object that has an overlapping portion with the image area of ​​the tracked object in both the first image and the second image, r takes a value greater than 1. By calculating the loss in this way, Proximity In the case where a non-tracking target object exists that is not being tracked, learning can be performed so that the distance between the features becomes small. This makes it possible to acquire features that can robustly identify the tracking target object even when the tracking target object is occluded by another object or occludes another object.

[0068] In addition, the loss function is not limited to the above form, as long as it changes parameters when a non-tracked object exists in both the first image and the second image and has an overlapping portion with the image area of ​​the tracked object.

[0069] <Variation 4> In this modification, when calculating the loss, multiple sets of a first image and a second image are prepared, and the above process is performed for each set to obtain the loss for each set. The loss obtained for each set is weighted according to the score of the non-tracking target object corresponding to the set, and the sum of these weighted losses is used as the final loss. In this modification, the loss is calculated according to the following (Equation 6).

[0070]

number

[0071] Here, N is the total number of sets, and loss i is the loss calculated for the i-th set. Also, p i is the score of the non-tracked object selected when calculating d2 in (Equation 1) for the i-th set. When the loss is calculated in this way, non-tracked objects with high scores, i.e., objects with a high risk of incorrectly switching tracking, contribute more to the loss. This allows learning to emphasize cases that are important to tracking performance, making it possible to perform efficient learning. Note that the weighting of the loss is not limited to the above format, as long as it is changed depending on the score.

[0072] <Variation 5> In this modification, contrast loss is used as the loss function. Contrastive loss is expressed by the following (Equation 7).

[0073]

number

[0074] Here, d is the distance between the features of the features corresponding to the two objects, and y is 1 when the two objects are the same object, and 0 when the objects are different objects. In this loss function, d is trained to be small when the two objects are the same object, and d is trained to be large when the objects are different objects. In addition, r and m are distance parameters and margin parameters, respectively, and as in the first embodiment, when r is set large, training is performed so that the distance between the features of the same object becomes small, and when m is set large, training is performed so that the distance between the features of different objects becomes large. When setting the parameters, for example, (Equation 2) and (Equation 3) can be used as in the first embodiment.

[0075] <Variation 6> In the first embodiment, CNN is used as the learning model. However, the model applicable to the learning model is not limited to CNN, and other models (for example, machine learning models) may be used.

[0076] In the first embodiment and its modified example, an example of a method of calculating the loss based on the distance between images of the feature amount of the tracking target object in each image, the distance between the feature amounts of the tracking target object and the non-tracking target object in each image, and the overlap rate between the image area of ​​the tracking target object and the image area of ​​the non-tracking target object in one or more images has been described. However, the method of calculating the loss using these pieces of information is not limited to the above method.

[0077] In the first embodiment, the learning device 100 and the tracking device 200 are described as separate image processing devices, but they may be stored in one image processing device. In other words, both the process described as being performed by the learning device 100 and the process described as being performed by the tracking device 200 may be performed by one image processing device. In such a case, for example, an imaging device equipped with an imaging unit that captures moving images or captures still images periodically or irregularly, and an image processing device that can execute both the process described as being performed by the learning device 100 and the process described as being performed by the tracking device 200 can be configured. In such an imaging device, the operation of the learning device 100 (learning of the CNN for object detection and the CNN for feature extraction) can be performed using the captured image captured by the imaging device itself. Then, the imaging device can perform the tracking process of the object in the captured image captured by itself after the operation using the learned CNN for object detection and CNN for feature extraction.

[0078] In addition, in the first embodiment, the Euclidean distance is used as the distance between the features. However, the distance between the features may be calculated by any method and is not limited to a specific calculation method (any type of distance may be used as the distance between the features).

[0079] [Second embodiment] Each functional unit shown in Fig. 1 and Fig. 7 may be implemented by hardware or software (computer program). In the latter case, a computer device capable of executing such a computer program is applicable to the above-mentioned learning device 100 and tracking device 200. An example of the hardware configuration of a computer device applicable to the learning device 100 and tracking device 200 will be described with reference to the block diagram of Fig. 12.

[0080] The CPU 1201 executes various processes using computer programs and data stored in the RAM 1202 and the ROM 1203. As a result, the CPU 1201 controls the operation of the entire computer device, and executes or controls the various processes described as being executed by the learning device 100 and the tracking device 200.

[0081] The RAM 1202 has an area for storing computer programs and data loaded from the ROM 1203 or the external storage device 1206. The RAM 1202 also has an area for storing data received from the outside via the I / F 1207. The RAM 1202 also has a work area used when the CPU 1201 executes various processes. In this way, the RAM 1202 can provide various areas as needed.

[0082] The ROM 1203 stores setting data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and the like.

[0083] The operation unit 1204 is a user interface such as a keyboard, a mouse, or a touch panel, and the user can input various instructions to the CPU 1201 by operating it.

[0084] The display unit 1205 has a liquid crystal screen or a touch panel screen, and can display the processing results by the CPU 1201 as images, characters, etc. For example, the results of the tracking process described above can be displayed on the display unit 1205. The display unit 1205 may also be a projection device such as a projector that projects images and characters.

[0085] The external storage device 1206 is a large-capacity information storage device such as a hard disk drive. The external storage device 1206 stores an OS (operating system) and computer programs and data for causing the CPU 1201 to execute or control the various processes described as being performed by the learning device 100 and the tracking device 200. The data stored in the external storage device 1206 includes the data described as being stored in the learning device 100 and the tracking device 200, the data treated as known data in the above description, data related to CNN, and the like. The computer programs and data stored in the external storage device 1206 are loaded into the RAM 1202 as appropriate under the control of the CPU 1201, and become targets for processing by the CPU 1201.

[0086] The I / F 1207 is a communication interface for performing data communication with an external device. For example, the I / F 1207 can be connected to an imaging device capable of capturing moving images and still images, a server device that stores moving images and still images, or a network to which these devices are connected. In this case, moving images and still images captured by the imaging device and moving images and still images supplied from the server device are supplied to the computer device via the I / F 1207, and the supplied moving images and still images are stored in the RAM 1202 or the external storage device 1206.

[0087] The CPU 1201 , RAM 1202 , ROM 1203 , operation unit 1204 , display unit 1205 , external storage device 1206 , and I / F 1207 are all connected to a system bus 1208 .

[0088] The hardware configuration shown in FIG. 12 is an example of a hardware configuration of a computer device that can be applied to the learning device 100 and the tracking device 200, and can be modified / altered as appropriate.

[0089] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) and the like used in each of the above embodiments and each of the modified examples are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0090] In addition, any or all of the embodiments and modifications described above may be used in appropriate combination. In addition, any or all of the embodiments and modifications described above may be used selectively.

[0091] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0092] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0093] 100: Learning device 110: Acquisition unit 120: Object detection unit 130: Feature extraction unit 140: Loss calculation unit 150: Learning unit

Claims

1. a calculation means for calculating a loss based on a distance between images of feature amounts of a tracked object in each image, a distance between feature amounts of a tracked object and a non-tracked object in each image, and an overlap rate between an image area of ​​the tracked object and an image area of ​​a non-tracked object in one or more images; A learning means for learning a model for extracting a feature amount of an object from an image based on the loss; A tracking means for performing a tracking process for tracking a tracking target object in an image based on the model learned by the learning means; An image processing device comprising:

2. The calculation means is The feature amounts of the tracked object and the non-tracked object in the first image are obtained from a feature amount map that is a result of the calculation of the model to which the first image is input, and the feature amount of the tracked object in a second image subsequent to the first image is obtained from the feature amount map that is a result of the calculation of the model to which the second image is input.

2. The image processing device according to claim 1,

3. The calculation means is An image area of ​​an object in a first image is resized, and the result of the calculation of the model with the resized image area input is acquired as a feature amount of the object in the first image; an image area of ​​an object in a second image subsequent to the first image is resized, and the result of the calculation of the model with the resized image area input is acquired as a feature amount of the object in the second image.

2. The image processing device according to claim 1,

4. 4. The image processing device according to claim 1, wherein the calculation means calculates a loss based on a distance between images of feature amounts of the tracked object in each image, a distance between the feature amounts of the tracked object and the non-tracked object in the image, an overlap rate between an image area of ​​the tracked object and an image area of ​​the non-tracked object in a first image, and an overlap rate between an image area of ​​the tracked object and an image area of ​​the non-tracked object in a second image subsequent to the first image.

5. the calculation means calculates the loss for each set of images, and calculates a final loss by summing the losses calculated for each set of images weighted according to a score indicating an object likelihood obtained in the detection of the non-tracked object; The learning means learns the model based on the final loss.

5. The image processing apparatus according to claim 1, wherein the image processing apparatus is a computer.

6. The calculation means calculates the loss for each set of images, and calculates an average of the losses calculated for each set of images as a final loss; The learning means learns the model based on the final loss.

5. The image processing apparatus according to claim 1, wherein the image processing apparatus is a computer.

7. 4. The image processing apparatus according to claim 2, wherein the non-tracking target object in the first image is the non-tracking target object that is closest to the tracking target object in the first image.

8. An imaging unit that captures an image; An image processing device according to any one of claims 1 to 7, An imaging device comprising:

9. An image processing method performed by an image processing device, comprising: a calculation step in which a calculation means of the image processing device calculates a loss based on a distance between images of feature amounts of the tracked object in each image, a distance between feature amounts of the tracked object and non-tracked object in each image, and an overlap rate between an image area of ​​the tracked object and an image area of ​​a non-tracked object in one or more images; a learning step in which a learning means of the image processing device learns a model for extracting a feature amount of an object from an image based on the loss; a tracking step in which a tracking means of the image processing device performs a tracking process for tracking a tracking target object in an image based on the model learned in the learning step; An image processing method comprising:

10. A computer program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 7.

Images