Information processing device, information processing method and program

Abstract

To make it possible to track a specific object.SOLUTION: An information processing device that tracks a specific object from images captured at multiple points in time includes: retention means for retaining a feature amount of a tracking target on the basis of a learned model for detecting the position of a predetermined object in an input image; acquisition means for acquiring feature amounts of objects in a plurality of images on the basis of the learned model; detection means for detecting a candidate object similar to the tracking target on the basis of the feature amount of the tracking target and the feature amounts of the objects acquired from the plurality of images; and identification means for identifying a correspondence relationship between the candidate object detected in a first image among the plurality of images and the candidate object in a second image captured at a point in time different from that of the first image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a technique for tracking a specific subject within an image. [Background technology]

[0002] There are techniques for tracking a specific subject in an image, such as those that use brightness or color information or template matching, but in recent years, techniques that use Deep Neural Networks (hereinafter abbreviated as DNN) have been attracting attention as a highly accurate tracking technique. For example, Non-Patent Document 1 is one method for tracking a specific subject in an image. An image containing the target to be tracked and an image serving as a search range are input to a Convolutional Neural Network (hereinafter abbreviated as CNN) with the same weights. Then, the cross-correlation between the respective feature amounts obtained from the CNN is calculated to identify the position of the target to be tracked in the image within the search range. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP2013-219531 Public Relations [Non-patent literature]

[0004] [Non-Patent Document 1] Bertinetto, “Fully-Convolutional Siamese Networks for Object Tracking”,arXiv 2016 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Non-Patent Document 1, when an object similar to the tracking target exists in the image, the cross-correlation value with the similar object becomes high, which may cause an error in tracking the similar object as the tracking target. In addition, in Patent Document 1, when an object similar to the tracking target exists in the vicinity of the tracking target, the positions of the tracking target and the similar object are predicted. However, in the method shown in Patent Document 1, since only the position of the tracking target is used for prediction, there is a possibility that the tracking target may be lost when the tracking target exists in a position away from the predicted position or when the tracking target and the similar object are close to each other.

[0006] The present invention has been made in consideration of such problems, and has an object to track a specific object. [Means for solving the problem]

[0007] The information processing device of the present invention that solves the above-mentioned problems is an information processing device that tracks a specific object from images captured at multiple times, and is characterized in having: a storage means for storing features of the tracked target based on a trained model that detects the position of a specific object in an input image; an acquisition means for acquiring features of the object in multiple images based on the trained model; a detection means for detecting a candidate object similar to the tracked target based on the features of the tracked target and the features of the object acquired from the multiple images; and an identification means for identifying a correspondence between the candidate object detected in a first image of the multiple images and the candidate object in a second image captured at a time different from that of the first image. Effect of the Invention

[0008] According to the present invention, a specific object can be tracked. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing device; [Diagram 2] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Diagram 3] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 4] A flowchart showing a processing procedure executed by a tracking target determination unit. [Diagram 5] A flowchart showing a processing procedure executed by an object detection unit. [Figure 6] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 7] A flowchart showing a processing procedure executed by a tracking unit. [Figure 8] FIG. 1 shows an example in which a tracking target is occluded. [Figure 9] FIG. 13 is a diagram showing an example of detecting the position of a tracking target in an image. [Figure 10] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 11] FIG. 13 is a diagram showing an example of occlusion determination; [Figure 12] FIG. 1 shows an example of an image in which multiple candidate objects are detected. [Figure 13] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 14] 1 is a flowchart showing a processing procedure executed by an information processing device; [Figure 15] FIG. 13 is a diagram showing an example of an acquired template image and a search range image. [Figure 16] A diagram showing an example of a map output by a trained model. [Figure 17] A diagram showing an example of training data used in a learning model. [Figure 18] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 19] FIG. 1 is a block diagram showing an example of a functional configuration of an information processing device; [Figure 20] 1 is a flowchart showing a processing procedure executed by an information processing device; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> <Execution phase processing using trained models> An information processing device according to an embodiment of the present invention will be described with reference to the drawings. Note that the same reference numerals in the drawings perform the same operation, and therefore the description will be omitted. Also, the components shown in this embodiment are merely examples, and are not intended to limit the scope of the present invention to only those.

[0011] In this embodiment, an example will be described in which the tracking target and an object similar to the tracking target are simultaneously tracked, and tracking is continued stably even in a situation where there are many objects similar to the tracking target or a situation where the tracking target is blocked by another object. That is, the purpose is to stably track each object even when similar objects exist.

[0012] 1 is a hardware configuration diagram of an information processing device 1 according to this embodiment, which tracks a specific object from images captured at multiple times. A CPU H101 controls the entire device by executing a control program stored in a ROM H102. A RAM H103 temporarily stores various data from each component. The RAM H103 also develops the program, making it executable by the CPU H101.

[0013] The storage unit H104 stores data to be processed in this embodiment, and saves data to be tracked. As a medium for the storage unit H104, a HDD, a flash memory, various optical media, etc. can be used. The input unit H105 is composed of a keyboard, a touch panel, a dial, etc., and receives input from a user, and is used when setting a tracking target. The display unit H106 is composed of a liquid crystal display, etc., and displays a subject and a tracking result to a user. In addition, this device can communicate with other devices such as a shooting device via the communication unit H107.

[0014] FIG. 2 is a block diagram showing an example of a functional configuration of the information processing device 1. The information processing device 1 has an image acquisition unit 201, a tracking target determination unit 202, a storage unit 203, an object detection unit 204, and a tracking unit 205, and each of the components is connected to a storage unit 206. The storage unit 206 may be in an external device or may be included in the information processing device 1. Each of the functional components will be briefly described. The image acquisition unit 201 acquires an image of a predetermined object captured by an imaging device. The predetermined object is, for example, an object such as a person or a vehicle, and is an object that has some individual differences. In the following embodiment, tracking of a person will be described as a specific example. The tracking target determination unit 202 determines an object to be a tracking target (target object) from among objects included in the image. The storage unit 203 stores the feature amount of an object that is a candidate for a tracking target from an initial image. The object detection unit 204 detects the position of an object from images captured at multiple times. The tracking unit 205 identifies and tracks a tracking target from images captured at multiple times.

[0015] 3 is a flowchart showing the process flow of this embodiment. In the following description, each process (step) is represented by adding an S to the beginning, and the notation of the process (step) is omitted. However, the information processing device does not necessarily have to perform all the processes described in this flowchart. The processes executed by the CPU H101 are each shown as a functional block.

[0016] In S301, the image acquisition unit 201 acquires an image (initial image) of a predetermined object. The image acquisition unit 201 may acquire an image captured by an imaging device connected to the information processing device, or may acquire an image stored in the storage unit H104. The process from S301 to S303 aims to set a target object to be tracked using the initial image.

[0017] In S302, the tracking target determination unit 202 determines an object to be tracked (target object) from the image acquired in S301. There may be one or more tracking targets. In this embodiment, an example of selecting one tracking target will be described. Here, a trained model that detects the position of a specific object is used to acquire the position of an image feature indicating a specific object from the image, and a partial image including the target object is determined. For the trained model, for example, a model that has previously learned image features about a specific object such as a person or a vehicle is used. The learning method will be described later. When one object is detected from the image, it is set as the tracking target. When the specific object is not detected from the image, for example, the image of the next frame may be input. When multiple objects are acquired, candidates for the tracking target are output, and the tracking target is determined by a method specified in advance. Here, the tracking target (target object) in the acquired image is determined according to an instruction specified by the input unit H105.

[0018] A specific method for determining the tracking target is a method for determining the tracking target by touching a subject displayed on the display unit H105. Note that the tracking target may be determined by automatically detecting a main subject or the like in an image, other than being specified by the input unit H105. An example of a method for automatically detecting a main subject in an image is Japanese Patent No. 6556033. Also, the tracking target may be determined based on both the specification by the input unit H105 and the result of object detection in the image. Examples of a technology for detecting an object in an image include "Liu, SSD: Single Shot Multibox Detector. In: ECCV2016".

[0019] FIG. 12 shows the result of detecting candidates for the tracking target from within an image. Person 1303, person 1305, and person 1307 in FIG. 12 are candidates for the tracking target. Frames 1303, 1305, and 1307 are bounding boxes (hereinafter referred to as BB) showing the positions of the detected candidates. The user can determine the tracking target by touching any of the candidates BB shown on the display unit 106 or selecting it with a dial or the like. As described above, there are various means for determining the tracking target, but the present embodiment does not limit the means for designating the tracking target.

[0020] In S303, the storage unit 203 stores the feature amount of the tracking target from the image including the determined tracking target based on the trained model. A detailed flowchart of S303 for the feature amount storage process is shown in Fig. 4. The storage unit 202 generates and stores a template feature amount representing the tracking target based on the image obtained by the image acquisition unit 201 and a bounding box (hereinafter referred to as BB) indicating the position of the tracking target obtained by the tracking target determination unit 202.

[0021] First, in S401, the storage unit 203 acquires information on the position in the image of the tracking target determined by the tracking target determination unit 202. The information on the position of the tracking target acquired here is hereinafter referred to as a Bounding Box (hereinafter BB). Note that the information on the position of the tracking target uses information input by the user on the central position of the tracking target when the tracking target is determined in S302, or a result of detecting a predetermined position (e.g., center of gravity) of the tracking target by a learning model.

[0022] Next, in S402, the storage unit 202 acquires a template image obtained by extracting an image showing the tracking target to a predetermined size based on the position of the tracking target in the image. That is, the periphery of the area obtained in S401 is cut out from the initial image as a template image, and resized to a predetermined size. The predetermined size may be matched to the size of the input image of the trained model.

[0023] Thereafter, in S403, the holding unit 202 inputs a template image indicating the tracking target to a trained model that detects the position of a predetermined object in the input image, thereby acquiring the feature amount of the tracking target. Here, the image resized in S402 is input to the CNN (trained model). The CNN is trained in advance to obtain feature amounts that make it easy to distinguish between tracking targets and non-tracking targets. The training method will be described later. Here, the CNN is composed of convolution and nonlinear transformations such as Rectified Linear Unit (hereinafter, ReLU) and Max Pooling. ReLU and Max Pooling described here are merely examples. Leaky ReLU, Sigmo score function, etc. may be used instead of ReLU, and Average Pooling, etc. may be used instead of Max Pooling. This embodiment is not limited to these. Thereafter, in S404, the holding unit 202 holds the feature amount of the tracking target obtained in S403 as a template feature amount indicating the tracking target. The above processing is the processing of the setting phase of the tracking target.

[0024] Next, in S304, the image acquisition unit 201 acquires images captured at multiple times in order to perform tracking processing. In the following processing, a process of detecting a tracking target set in a first image from a second image captured at a time different from that of the first image will be described. In addition, it is assumed that the first image and the second image are captured so as to include the tracking target as much as possible.

[0025] In S305, a candidate object similar to the tracking target is detected based on the feature amount of the tracking target and the feature amount of the object acquired from the multiple images. A flowchart explaining the process executed by the object detection unit 204 in S305 is shown in Fig. 5. Note that the process from S304 onwards is a process for images captured after the image in which the tracking target is determined, and is a process for detecting the tracking target from the image.

[0026] First, in S501, the object detection unit 204 acquires a search range image (partial image) indicating an area in which to search for a tracking target from a current image (second image). Here, the search range image is acquired based on the detection position of the previous tracking target or candidate object. That is, in the second image, a partial image of a predetermined size is extracted from an area corresponding to the vicinity of a candidate object detected from the first image (past image). The size of the search area may be changed according to the speed of the object and the angle of view of the image. In addition, the search area may be the entire search image or the periphery of the previous position of the tracking target. By setting a partial area, rather than the entire input image, as the search range, the processing speed is improved and tracking correspondence errors are reduced.

[0027] Thereafter, in S502, the object detection unit 204 extracts an input image to be input to the trained model from the search range image. The object detection unit 204 cuts out a search range area from the search range image and resizes it. The size of the search range is determined as a constant multiple of the size of the BB to be tracked. By obtaining features from images of the same size, features with less noise can be obtained. Based on the determined search area, a region is cut out and resized to be equivalent to the resize ratio in S402.

[0028] Furthermore, in S503, the object detection unit 204 inputs the extracted search range image to a trained model (CNN) that detects the position of a predetermined object in the input image, thereby acquiring the feature amount of each search range image. That is, the image of the cut-out area is input to the CNN. The feature amount of each search range image indicates the feature amount of an object present in each search range image. Note that the CNN in S503 has a part or all of the same weights as the CNN in S403. With this CNN, for example, if a certain search range image includes an obstruction that obstructs a person, a feature amount indicating the obstruction can be acquired. Also, if another partial image does not include a person but includes an animal, a feature amount indicating the animal is obtained.

[0029] Furthermore, in S504, the object detection unit 204 obtains the cross-correlation between the feature amount of the tracking target and the feature amount of an object present in the current search range image obtained in S503. The cross-correlation is an index that indicates the similarity between detected objects. Here, an object similar to the tracking target (object of the same type) is called a candidate object. In other words, an object whose cross-correlation is greater than a predetermined value is a candidate object. The candidate objects include either or both of the tracking target and the non-tracking target. As a specific example, if the tracking target is a person, the cross-correlation of the search range image having a feature amount indicating a person will be high.

[0030] Furthermore, in S505, the object detection unit 204 detects the position of the candidate object in the current image. Since some or all of the weights of the CNN in S503 and the CNN in S403 are the same, the value of cross-correlation is large at positions where the probability that the candidate object exists in the search range is high. Therefore, it is possible to detect the position of the candidate object from the search range image where the value of cross-correlation is equal to or greater than a threshold value. That is, based on the cross-correlation obtained in S504, a position where the cross-correlation is greater than a predetermined value is detected as the position of the candidate object. For a position where the cross-correlation is smaller than a predetermined value, it is considered that the possibility of the tracking target existing is low. Here, further, based on the position of this candidate object, a BB that further surrounds the candidate object is obtained. First, the position of the BB is determined based on the search range image that showed a high response in the cross-correlation.

[0031] An example of the processing result of S305 is shown in Fig. 9. Map 901 shows a map obtained based on cross-correlation. The tracking target is a person 902, and a cell 904 near the center of the person 902 shows a high cross-correlation value. If this correlation value is equal to or greater than a threshold, it can be estimated that the person 902 is located in cell 904. On the other hand, the width and height of the BB may be learned in advance so that the CNN can estimate them (described later). Also, the width and height of the BB of the tracking target obtained in S302 may be used as is.

[0032] In S306, the tracking unit 205 identifies a correspondence between a candidate object detected in a first image among the multiple images and a candidate object in a second image captured at a different time from the first image. By identifying the correspondence between objects detected at multiple times, it is possible to track objects that have a correspondence. In addition, by updating the feature amount and position of the tracking target based on the image in which the tracking target is detected, it is possible to track the tracking target more stably. A flowchart explaining the process executed by the tracking unit 205 is shown in FIG. 7.

[0033] First, in S701, the tracking unit 205 acquires combinations (candidates for correspondence) of candidate objects detected from images captured at past times and stored in advance in the storage unit 206, and candidate objects detected from images captured at the current time. Here, past candidate objects and current candidate objects are associated so that all combinations can be made. Here, candidate objects detected from past images are assigned labels of tracked / non-tracked objects. Note that when there is one tracked target, the object identified as the tracked target among the past candidate objects may be associated with each of the current candidate objects.

[0034] In S702, the tracking unit 205 identifies a combination (correspondence) in which the acquired similarity is equal to or greater than a threshold value. A high similarity between a past candidate and a current candidate indicates that the past candidate and the current candidate are likely to be the same object. There are several methods for associating. For example, there is a method of prioritizing association of candidates with high similarity, a method using the Hungarian algorithm, and the like. The method of association is not limited here. Here, for a candidate object other than the tracking target in the first image, the same object is identified based on the similarity with the candidate object in the second image. By tracking other objects similar to the tracking target object in this way, it is possible to suppress association of the tracking target with other objects. Therefore, it is possible to perform stable tracking. By performing appropriate association in this way, the past tracking target and the current tracking target can be recognized as the same object.

[0035] For example, the similarity L between past candidate c1 and current candidate c2 is calculated as follows. Here, BB is a vector that brings together the four variables (center coordinate value x, center coordinate value y, width, height) of each candidate BB, and f indicates the features of each candidate. The features are extracted from the feature map obtained from CNN where each candidate is located. W1 and W2 are empirically obtained coefficients, where W1>0 and W2>0. In other words, the closer the features are, the higher the similarity, and the closer the detection positions and the sizes of the detection areas are, the higher the similarity.

[0036]

number

[0037] Next, in S703, the tracking unit 205 identifies the tracking target based on the association result. As a result of the association obtained in S702, the current candidate associated with the past tracking target can be identified as the tracking target. Candidate objects other than the tracking object are given information indicating that they are not tracking targets. If there is no current candidate object whose similarity with the feature amount of the past tracking target is greater than a predetermined threshold, the tracking target may be outside the angle of view or may be occluded by another object. In that case, it may be notified that the tracking target has not been identified.

[0038] Finally, in S704, the storage unit 206 holds the feature amount of the tracking target in the second image and the feature amount of the candidate object in the second image. If the tracking target is identified from the current image, the feature amount of the tracking target is updated. If a candidate object having a similarity to the feature amount of the tracking target in the first image greater than a predetermined threshold is detected from the second image, the feature amount acquired from the second image is held as the feature amount of the tracking target. If a candidate object having a similarity to the feature amount of the tracking target greater than a predetermined threshold is not detected from the second image, the feature amount acquired from the first image is held as the feature amount of the tracking target. If the tracking target is not detected from the current image, the feature amount and position of the tracking target in the past image are held. Also, the feature amount in which the label of the tracking target / non-tracking target is added to the current candidate object is stored. The BB (position and size) and features of the tracking target and its candidates are updated. By holding the feature amount and the determination result for the candidate object similar to the tracking target, more stable tracking can be performed.

[0039] In S307, the image acquisition unit 201 judges whether or not to end the tracking process. If the tracking process is to be continued, the process returns to S304, and if the tracking process is to be ended, the process proceeds to the end. The end determination is made, for example, when a user's end instruction is obtained or when the image of the next frame cannot be acquired. If the image of the next frame can be acquired, the process proceeds to S304. The above is the process in the execution step of the tracking process. Next, the learning process will be described.

[0040] <Learning steps> Next, we will show a method for training a trained model (specifically, a CNN) that estimates the position of an object in an image. Note that the trained model used here has already been trained to some extent for object classification tasks (e.g., to detect people but not animals), and we will consider an example in which the trained model is trained to be able to identify individuals based on the external features of a given object. This makes it possible to track a specific object.

[0041] For example, suppose there is a person A wearing red clothes and a person B wearing yellow clothes. A trained model that simply detects people may be trained to ignore the color of the clothes in the task of person detection because the color of the clothes is not necessarily a necessary feature. However, when detecting (tracking) only person A, it is necessary to learn a feature that distinguishes person A from person B. In this case, the color of the clothes becomes an important feature and may be a feature necessary for identifying an individual. In this embodiment, learning is performed so that the feature amount of the tracking target object can be recognized from objects in the same category in a manner that distinguishes it from other objects in the same category. Note that an example of the functional configuration of the information processing device 2 at the time of learning is shown in FIG. 13. The information processing device 2 includes a ground truth acquisition unit 1400, a template image acquisition unit 1401, a search range image acquisition unit 1402, a tracking target estimation unit 1403, a loss calculation unit 1404, a parameter update unit 1405, a parameter storage unit 1406, and a storage unit 1407.

[0042] The storage unit 1407 stores images captured at multiple times and GT information indicating the position and size of the tracking target for each of the images. Here, information input by the user as the center position (or BB indicating the area) of the tracking target object for each image is stored as GT information. The method of generating GT information may be a method other than GT assignment by the user. For example, the result of detecting the position of the tracking target object using another trained model may be used. The GT acquisition unit 1400, template image acquisition unit 401, and search range image acquisition unit 1402 each acquire images stored in the storage unit 1407.

[0043] A Ground Truth (hereinafter, GT) acquisition unit 1400 acquires the correct position of the object to be tracked in the template image and the correct position of the object to be tracked in the search range image by acquiring GT information. The BB of the object to be tracked in the template image acquired in the template image acquisition unit 1401 and the BB of the object to be tracked in the search range image acquired in the search range image acquisition unit 1402 are acquired. Specifically, as shown in FIG. 17, in an image 1704, information indicating that the object 1705 to be the object to be tracked is the object to be tracked is given, and information indicating that the object is not the object to be tracked is given to the other areas. For example, the area of ​​the object to be tracked 1705 is labeled with a binary real number of 1, and the other areas are labeled with 0.

[0044] The template image acquisition unit 1401 acquires an image in which a tracking target exists as a template image. Note that the template image may include a plurality of objects of the same category. The search range image acquisition unit 1402 acquires an image to be used to search for a tracking target. That is, the image is an image from which the feature amount of a specific object to be tracked can be acquired. For example, the template image acquisition unit 1401 selects an arbitrary frame from a series of sequence images, and the image acquisition unit 1402 before searching selects another frame from the same sequence images that was not selected by the template image acquisition unit 1401.

[0045] The tracking target estimation unit 1403 estimates the position of the tracking target in the search range image. Based on the template image obtained by the template image acquisition unit 1401 and the search range image obtained by the search range image acquisition unit 1402, the tracking target estimation unit 1403 estimates the position of the tracking target in the search range image.

[0046] The loss calculation unit 1404 calculates the loss based on the tracking result obtained by the tracking target estimation unit 1403 and the position of the tracking target in the search range image obtained by the GT acquisition unit 1404. The closer the estimation result is to the teacher data, the smaller the loss becomes. Note that the correct answer for the position of the tracking target in the search range image is obtained based on the GT information acquired by the GT acquisition unit.

[0047] The parameter update unit 1405 updates the parameters of the CNN based on the loss obtained by the loss calculation unit 1404. Here, the parameters are updated so that the loss value converges. When the total loss value converges or the loss value becomes smaller than a predetermined value, the parameter set is updated and learning is terminated.

[0048] The parameter storage unit 1406 stores the CNN parameters updated by the parameter update unit 1405 in the storage unit 206 as learned parameters.

[0049] A flowchart of the learning process will be described with reference to FIG. 14. First, in S1500, the GT acquisition unit 1400 acquires GT information, and based on the GT information, acquires the correct position of the object to be tracked in the template image (BB of the tracked object) and the correct position of the tracked object in the search range image. In S1501, the template image acquisition unit 1401 acquires a template image. For example, an image as shown in FIG. 15(a) is acquired. An object 1601 in FIG. 15(a) is the tracked object, a partial image 1602 is the BB of the tracked object acquired in the GT acquisition unit 1400, and a partial image 1603 indicates an area to be cut out as a template. That is, here, the template image acquisition unit 1401 acquires the partial image 1603 as the template image.

[0050] In S1502, the template image acquisition unit 1401 cuts out an area to be the template from the template image and resizes it to a predetermined size. The size of the cut-out area is determined based on the BB of the tracking target, as a constant multiple of the size of the BB, for example.

[0051] In S1503, the tracked target estimation unit 1403 inputs the template image generated in S1502 to a learning model (CNN) to obtain a CNN feature amount of the template.

[0052] In S1504, the search range image acquisition unit 1402 acquires a search range image. The partial image that becomes the search range is acquired as a partial image that includes the tracking target based on the position and size of the tracking target object. An example of an image that becomes the search range is shown in FIG. 15(b). In FIG. 15(b), object 1604 indicates the tracking target, partial image 1605 indicates the BB of the tracking target, and partial image 1606 indicates the search range area. The search range image 1606 includes an object similar to the tracking target object.

[0053] In S1505, the search range image acquisition unit 1402 cuts out and resizes the search range area from the search range image. The size of the search range is determined to be a constant multiple of the size of the BB of the tracking target, and resized according to the magnification at which the template was resized in S1502 (the size of the tracking target after the template resizing is resized so that it is approximately the same as the size of the tracking target after the search range resizing).

[0054] In S1506, the tracking target estimation unit 1403 inputs the search range image generated in S1506 to a learning model (CNN) to obtain a CNN feature amount of the search range.

[0055] In S1507, the tracking target estimation unit 1403 estimates the position of the tracking target in the search range image. The tracking target estimation unit 1403 calculates the cross-correlation indicating the similarity between the CNN feature of the tracking target obtained in S1506 and the CNN feature of the search range obtained in S1506, and outputs it as a map. Based on the cross-correlation, the tracking target is estimated by indicating the position where the cross-correlation is equal to or greater than a threshold. The map showing the estimation result is shown in FIG. 16(a). Map 1701 is the map obtained by the cross-correlation, and areas 1702 and 1703 indicate the places where the cross-correlation value is high. In this way, by taking the cross-correlation, the cross-correlation value of the position where it is highly likely that an object similar to the tracking target exists becomes high. On the other hand, the position of the tracking target that is the correct answer obtained by the GT acquisition unit 1400 is 1705 in FIG. 16(b). In other words, 1702 indicates the position of the tracking target, so a desirable value is estimated, but 1703 has a high cross-correlation value even though it is not the tracking target, so it can be said to have estimated an undesirable value. The purpose of the learning step is to update the weights so that the cross-correlation value at the position of the tracking target is high and the cross-correlation value at positions other than the tracking target is low.

[0056] In S1508, the loss calculation unit 1404 calculates a loss related to the inferred position of the tracking target and a loss related to the size of the tracking target. The loss related to the position is calculated to proceed with learning so that the cross-correlation value of the position of the tracking target shows a high value. The Ground Truth (hereinafter, GT) acquisition unit 1400 acquires the BB of the tracking target in the template image acquired by the template image acquisition unit 1401 and the BB of the tracking target in the search range image acquired by the search range image acquisition unit 1402.

[0057] If the map 1701 obtained by the processing in S1507 is Cinf and the GT map 1704 is Cgt, the loss function can be written as in formula (1-2). Formula (1-2) is the average of the squares of the differences for each pixel between the maps Cin and Cgt, and the loss is small when the tracked target is correctly estimated, and the loss is large when a non-tracked target is estimated to be the tracked target or when the tracked target is estimated to be the non-tracked target.

[0058]

number

[0059] Similarly, the size-related loss is calculated according to equation (1-3).

[0060]

number

[0061] Loss W and Loss H are the losses related to the estimated width and height of the tracked object, respectively. W gt and H gt The width and height values ​​of the target are embedded in the position of the target. By calculating the loss using equations (1-3) and (1-4), W i n, H in Even in this case, the learning proceeds so that the width and height of the tracked target can be inferred from the position of the tracked target. By combining all the losses, we obtain Equation (1-5). Loss=Loss C +Loss w +Loss H Here, the loss is described in the form of Mean Squared Error (MSE), but the loss is not limited to MSE. Smooth-L1, etc. may also be used. There is no limitation on the calculation formula for the loss. Also, the loss function for the position and the loss function for the size may be different.

[0062] In S1509, the parameter update unit 1405 (learning unit) updates the parameters of the CNN based on the loss calculated in S1508. The parameter update is performed based on back propagation using Momentum SGD or the like. Note that while the output of the loss function for one image has been described, in actual learning, the loss value of formula (1-2) is calculated for the scores estimated for multiple various images. The connection weighting coefficient between layers of the learning model is updated so that the loss values ​​for multiple images are all smaller than a predetermined threshold.

[0063] In S1510, the parameter storage unit 1406 stores the CNN parameters updated in S1509 in the storage unit 206. In the inference step, inference is performed using the parameters stored in S1510, thereby making it possible to correctly track the tracking target.

[0064] In S1511, the parameter update unit 1405 determines whether or not to end the learning. The learning is determined to be ended when the loss value obtained by the formula (1-2) becomes smaller than a predetermined threshold value.

[0065] <Effects of the First Embodiment> This embodiment is characterized by tracking a tracking target and simultaneously tracking an object similar to the tracking target. By simultaneously tracking an object similar to the tracking target, erroneous tracking of similar objects is reduced, as will be described with reference to FIG. 8. Images 801, 802, and 803 are images obtained at times t=0, t=1, and t=2, respectively. A person 804 and a person 805 are shown in the image, of which the tracking target is person 804 and the similar object is person 805.

[0066] First, consider the case where only the tracking target 804 is tracked. In this case, the object 804 that was correctly tracked at time t=0 is occluded by the object 810 at time t=1. When occlusion occurs, the feature amount of the object 804 is likely to be detected as being less object-like due to the occlusion. Since the feature amount detected for the object 810 is highly object-like, it is highly likely that the tracking target at time t=1 will be regarded as the object 810, and erroneous tracking of the object 810 as the tracking target will begin.

[0067] Next, consider the case where not only the tracking target 804 but also a similar object 805 is tracked at the same time. There are two past tracking target candidates at time t=1, namely, object 804 and object 805. On the other hand, the newly obtained tracking target candidate at t=1 is only object 808 because object 809 is occluded. In this case, when comparing the similarity between past candidates 804 and 805 and object 808, the similarity between object 805 and object 808 is higher than the similarity between object 804 and object 808. The reason for the high similarity is that the CNN features associated with each candidate are learned to distinguish between objects and that the position and size of the BB change gradually over time. Therefore, it can be seen that the past candidate associated with the current candidate 808 is not 804 but 805. Here, the latest feature amount is updated to the feature amount of object 808 for object 805, but the feature amount obtained at t=0 is retained for object 804 that was not detected at t=1. Next, at time t=2, the similarity between the candidates is also calculated. The past candidates at time t=2 are object 804 and object 808. Meanwhile, there are two new candidates obtained at t=2, candidate 811 and candidate 812. These two candidate objects are not occluded, so desirable features can be acquired. When the similarity is calculated, the similarity between object 808 and object 811, and object 804 and object 812 are high, while the similarity between 808 and 812 and between 806 and 811 are low. Therefore, the tracking target 806 is associated with 812, and the tracking target can be tracked correctly.

[0068] <Variation 1-1> Online Metric Learning In Modification 1-1, in equation (1-1) in the first embodiment, the weight W2 for the feature amount is successively updated using the feature amounts of the tracking target and similar objects obtained in time series.

[0069] for example,

[0070]

number

[0071] where f target is the feature of the tracked target obtained at each time, and f distractоr are the features of similar objects obtained at each time.

[0072] By updating the weights using the features of the tracked target and similar objects as in equation (1-2), it is possible to calculate the similarity by assigning a larger weight to the feature dimension that makes it easier to distinguish between the tracked target and similar objects. Therefore, even if the features of the tracked target and similar objects are close to each other in the feature space, it becomes easier to distinguish between the tracked target and similar objects.

[0073] <Variation 1-2> Offline Metric Learning In Modification 1-2, the transformation for finding the similarity between feature quantities in formula (1-1) in embodiment 1 is calculated in advance by metric learning. If the function for transforming the feature quantities is F, formula (1-1) can be expressed as formula (1-7).

[0074]

number

[0075] Transformation F is a structure in which one or more layers of Neural Networks are connected, and can be trained in advance using triplet loss, etc. By training transformation F using triplet loss, it is possible to learn a transformation that makes the distance closer for the same object in the past and present, and makes the distance farther for different objects. For details on the method of training using triplet loss, see "Wang, Learning Fine-grained Image Similarity with Deep Ranking, In: CVPR2014".

[0076] <Embodiment 2> Obstruction determination In the second embodiment, an occlusion determination process is further performed in the tracking target identification process of S306 in FIG. 7 in the first embodiment. By performing the occlusion determination, even if the tracking target is occluded, switching of tracking to another similar object is suppressed. A process that differs from the first embodiment will be described in detail. Note that the hardware configuration is the same as that of the first embodiment. FIG. 18 shows an example of the functional configuration of an information processing device 1' in the second embodiment. The configuration is basically the same as that in FIG. 2 in the first embodiment, and an occlusion determination unit 207 that newly performs occlusion determination is added. The functional configurations with the same reference numerals perform the same process as in the first embodiment. The occlusion determination unit 207 determines the occlusion relationship between objects based on a partial image of the candidate object detected from the image. The tracking unit 205' further tracks the tracking target based on the determination result of the occlusion determination unit 297.

[0077] Next, the process executed by the information processing device 1' in this embodiment will be described. The flowchart of this embodiment corresponds to FIG. 3 and FIG. 10(A)(B). The basic process is the same as that of the first embodiment, and only the process of S306 is different. Therefore, the difference in S306 will be described in detail below, and the description of the other processes will be omitted. In addition, in S305, a candidate object similar to the target object is detected based on the feature amount of the tracking target. At this time, if the tracking target is occluded by another object, if the object occluding the tracking target is an object similar to the tracking target, it is detected as a candidate object. In that case, the tracking target is associated with the position of the occluded similar object by the occlusion determination process, but since the original tracking feature is maintained at the timing when the occlusion is removed, it can be tracked again. On the other hand, if the tracking target is occluded by an obstacle such as a wall, the occluded tracking target is not detected as a candidate object in S305. In this case, in the occlusion determination process at the later stage, it is determined that there is no candidate object that can be associated with the tracking target that was detected immediately before being occluded the previous time, and the feature amount of the tracking target is stored in S303. After that, tracking can be resumed when the occlusion is removed and detection becomes possible again.

[0078] 10(A) is a flowchart illustrating the tracking target identification process S306 including the occlusion determination process. First, in S701, the tracking unit 205' obtains the similarity between candidates for past time stored in advance in the storage unit 206 and candidates for the current time obtained by the object detection unit 204. The process of S701 is performed in the same manner as S701 in the first embodiment. Next, in S702, the tracking unit 205' performs association based on the similarity between the past candidates and the current candidate. The process of S702 is also the same as the process of S702 in the first embodiment.

[0079] In S1002, the occlusion determination unit 207 determines whether or not there is an occlusion region where the candidate object is occluded, based on the position of the candidate object in the image (second image) currently being processed. That is, the occlusion determination is performed for each candidate object in the current image. The occlusion determination process in S1002 will be described in more detail with reference to FIG. 10(B). Here, the occlusion determination is performed particularly for a candidate (called a target object) for which no corresponding candidate is found in S702. First, in S10021, the occlusion determination unit 207 determines whether or not the correspondence is established for all candidate objects detected in the past in S702. If the correspondence between all candidate objects detected from the past image (first image) and the candidate objects detected from the current image has been completed, the process proceeds to S10025. If there is a past candidate object (target object) among the candidate objects detected from the past image whose similarity to the candidate object detected from the current image is equal to or less than a threshold, the process proceeds to S10022. That is, if the process proceeds to S10022, there is a possibility that a candidate object that is occluded exists. In S10022, the occlusion determination unit 207 acquires information indicating the degree of overlap between the current candidate object (target object) and another candidate BB. Here, the intersection of union (hereinafter, IoU) is calculated as an index indicating the degree of overlap between objects. Specifically, for each partial image (BB) of the candidate object detected from the current image, assuming that area A of object A and area B of object B, the IoU between object A and object B is calculated as area (A∩B) / (A∪B). The higher the IoU, the higher the degree of overlap between the objects. The other candidate whose IoU exceeds a threshold value is set as an occluder of the candidate. At this time, the state of the candidate object A is determined to be "occluded". Next, in S10024, the occlusion determination unit 207 updates the position of the candidate determined to be occluded based on the position of the occluder. For example, it may be updated as in formula (2-1). p s =(1-α)p o +αp s Formula (2-1) Here, p s is the position of the candidate, and p о is the position of the occluder. α is an empirically set value.

[0080] In S703, the tracking unit 205 specifies the correspondence between the candidate object in the first image and the candidate object in the second image based on the occlusion determination result. That is, the position of the tracking target object in the second image is specified. In S702, if the candidate object specified as the tracking target object previously was specified in the current image, the position of the tracking target in the current image is specified. In S702, if the tracking target previously was not specified from the candidate object in the current image, occlusion determination is performed in S1002. If it is determined that the tracking target is occluded in the current image, the occluder is specified, and the position of the tracking target is updated based on the formula (2-1). On the other hand, the feature amount of the tracking target is not updated. In S704, the storage unit 206 stores the position and feature amount of the tracking target specified by the tracking unit 205. By the above process, even if occlusion occurs, the position of the tracking target may be updated while retaining the feature amount of the tracking target, so that the tracking target can be tracked again after the occlusion is eliminated.

[0081] <Variation 2-1> In the modified example 2-1, the occlusion determination is performed by a neural network. An example of performing the occlusion determination by a neural network is "Zhou, Bi-box Regression for Pedestrian Detection and Occlusion, In: ECCV2018". In this example, in S1002, the tracking unit 205 estimates the BB of the object and simultaneously estimates the non-occluded area (visible area) of the object area. Then, when the ratio of the area where occlusion occurs in the object area exceeds a predetermined threshold, it is possible to determine that there is an occlusion.

[0082] <Effects of the second embodiment> FIG. 11 shows the effect of updating the position of the candidate to the position of the occluder by such occlusion determination and updating the position of the candidate based on the position of the occluder.

[0083] In FIG. 11, the images obtained at times t=0, 1, 2, and 3 are images 1211, 1212, 1213, and 1214, respectively. The target to be tracked is 1216. At time t=0, there are two target to be tracked, 1216 and similar object 1215, and both are tracked. At time t=1, 1216 is hidden by similar object 1217, so the only candidate at time t=1 is similar object 1217. At this time, if the IoU between 1216 and 1217 exceeds the threshold and it is determined that there is an occlusion, the position of 1216 is updated to match the position of 1217 according to formula (2-1). At time t=2, the occlusion has not been resolved, so the position of 1216 is updated to match the position of 1218, which is an occluder. At time t=3, the occlusion is resolved, and there are three candidates, 1219, 1220, and 1221. In this case, the correct matching results are 1218 and 1219, and 1216 and 1220. However, if the occlusion determination is not performed and the position of 1216 is not updated to match the positions of 1217 and 1218, 1216 will be present in the vicinity of candidate 1221 at time t=3. Therefore, there is a high possibility that 1216 will be matched with the newly obtained candidate 1221, not with 1220, which will cause erroneous tracking. On the other hand, if the position of candidate 1216 is updated according to equation (2-1), the position of 1216 will be close to the position of 1220, and 1216 and 1220 can be matched. Therefore, it is possible to reduce erroneous tracking.

[0084] Third embodiment: Single object tracking (combination with online learning tracking method) In the third embodiment, in contrast to a tracking method using online learning, multiple candidates are simultaneously tracked, so that each object is stably tracked even when similar objects exist. The hardware configuration is the same as that of the first embodiment. FIG. 19 shows an example of the functional configuration of an information processing device 3 in this embodiment. The configuration is basically the same as that of FIG. 2 in the first embodiment, and a learning unit 1902 that performs online learning is newly added. The tracking unit 1901 specifies the position of the tracking target by inputting a current image to a trained model. The learning unit 1902 updates the connection weighting parameters of the trained model that estimates the position of the object based on the position of the tracking target estimated in the current image. The trained model used here is MDNet ("Nam, Learning Multi-Domain Convolutional Neural Networks for Visual Tracking, In: CVPR2016"). In MDNet, an image is input to a CNN (trained model) to obtain a feature value indicating an object. Furthermore, each of the acquired features is input to a Fully Connection layer (hereinafter, FC layer), and it is determined whether the input feature value is a feature value of the tracking target. The FC layer is trained online so that it outputs a higher value for an object that is more likely to be a tracking target. Online training trains the FC layer in the initial frame and at intervals of several frames thereafter. Below, a description of the same processes as in the first embodiment will be omitted, and a detailed description will be given of processes that differ from those in the first embodiment.

[0085] FIG. 20 shows the process executed by the information processing device 3 in this embodiment. The process from S301 to S304 is the same as the process from S301 to S304 in the first embodiment. In S304, a search range is set from an acquired image. The search range image is determined based on the past positions and sizes of candidate objects. In S305, the search range image obtained in S304 is input to a trained model, so that each feature amount acquired from the search range image is input to an FC layer, and an object whose obtained resemblance (similarity) to a tracking target exceeds a threshold is acquired as a candidate object. The trained model uses the MDNet described above. In S2001, the tracking unit 1901 identifies the position of the tracking target from among the candidate objects. In S2002, the learning unit 1902 updates the parameters of the trained model based on the determination result of the tracking target.

[0086] In such a tracking method using online learning, it is possible to reduce erroneous tracking by simultaneously tracking multiple candidates in the same manner as in the first embodiment.

[0087] <Embodiment 4> Multiple object tracking In the fourth embodiment, a case where a plurality of tracking target objects are set instead of one will be described. Even when a plurality of similar objects are tracked, by simultaneously tracking candidate objects detected in the past, the tracking target can be stably tracked even if it is lost once. The hardware configuration is the same as that of the first embodiment. The information processing device that executes this embodiment has a functional configuration similar to that of the information processing device 1 of the first embodiment, but there is a difference in the processing of the tracking target determination unit 202 and the tracking unit 205. The tracking target determination unit 202 determines a plurality of objects as tracking targets. The tracking target is determined by the same method as that of the first embodiment. All objects shown in a certain image may be acquired as tracking targets. The tracking unit 205 tracks each detected object for a plurality of tracking targets. Specifically, the CNN features of a plurality of candidate objects are held, and the similarity between the candidate objects at time t and time t+1 is also used to perform correspondence.

[0088] Next, the process executed by the information processing device 1 in this embodiment will be described. The flowchart of this embodiment corresponds to FIG. 3. In S301, the image acquisition unit 201 acquires an image (initial image) of a predetermined object. In S302, the tracking target determination unit 202 determines a plurality of objects to be tracked from the image acquired in S301. In S303, the storage unit 203 stores the feature amounts of a plurality of tracking targets from an image including the determined tracking targets based on a trained model. Here, the Detect-Track method is used for the trained model. (“Feichtenhofer, Detect to Track and Track to Detect, In: ICCV 2017”.) In Detect-Track, object detection is performed using CNN for each continuous time-series frame. Next, in S304, the image acquisition unit 201 acquires images captured at a plurality of times in order to perform tracking processing. In S305, the object detection unit 204 detects the position of a candidate object from temporally consecutive images obtained by the image acquisition unit 201 based on the trained model. First, the object detection unit 204 detects a candidate object using a CNN (trained model) for each consecutive time-series frame. That is, the CNN feature at time t and the CNN feature at time t+1 are acquired. Next, the position of the candidate object is detected by calculating the cross-correlation between the CNN feature obtained at time t and the CNN feature obtained at time t+1. In S306, the tracking unit 205 identifies multiple tracking targets from the current image (t+1). Here, first, the tracking unit 205 estimates the change in BB ΔBB (change in BB position and change in BB size) for each object. That is, the tracking unit 205 estimates the change in BB by comparing BB(t+1) with BB(t)+ΔBB(t). Here, it is found that objects having similar changes in BB position and BB size are the same object, and therefore each object can be associated with each other. Next, the tracking unit 205 calculates the distance between the CNN features of the candidate object at time t and time t+1 that have been associated based on the formula (1-1) to calculate the similarity. If there is an association where the similarity is greater than a predetermined value, it is associated with the previous detection result and tracked. Note that the associations may be determined in descending order of relatively high similarity.If there is no correspondence with a similarity greater than a predetermined value, the current detection result (feature amount and position) is held without being associated with the previous detection result.

[0089] Here, if the number of objects detected at time t is 2 and the number of objects detected at time t+1 is 1, then of the two objects obtained at time t, the object that is the same as the object at time t+1 is considered to be an object with high similarity. By associating objects with high similarity with each other, it is possible to reduce erroneous tracking. However, there may be cases where an object detected at time t is not detected at time t+1 due to occlusion or the like. At this time, if at least one or more candidate objects exist at time t+1 in addition to the object to be tracked, erroneous tracking of a candidate object located close to the object may begin. Therefore, in S306, the CNN features of multiple objects that are candidate objects may be held, and the similarity with the feature amount of the candidate object held at the time of similarity calculation may be calculated. If the object to be tracked is occluded, the correspondence cannot be identified, but tracking can be resumed when the occlusion is removed.

[0090] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or a CPU, MPU, etc.) of the system or device reads and executes the program. The program may also be provided by recording it on a computer-readable recording medium. [Explanation of symbols]

[0091] 1. Information processing device 201 Image Acquisition Unit 202 Tracking target determination unit 203 Generation part 204 Object detection unit 205 Tracking part 206 Memory section

Claims

1. An information processing device that tracks a specific object from images captured at multiple times, Based on the trained model, the system identifies a tracking target specified by the user from one or more past images, and an acquisition means for acquiring feature amounts of each of candidate objects including an object selected based on the tracking target, and feature amounts of each of the candidate objects in a current image; a detection means for detecting a candidate object similar to the tracking target from the current image and a candidate object similar to an object selected based on the tracking target, based on features of each of the candidate objects in the one or more previous images and features of each of the candidate objects in the current image; a tracking means for tracking the tracking target using a feature amount of the tracking target in the current image determined based on a result of the detection process; An information processing device comprising:

2. 2. The information processing apparatus according to claim 1, further comprising: an updating unit that updates the feature amount of the tracking target based on the result of the detection process.

3. when the candidate object is detected from the current image, the similarity of which with respect to the feature amount of the tracking target in the past image is greater than a predetermined threshold, the updating means updates the feature amount acquired from the current image as the feature amount of the tracking target; 3. The information processing device according to claim 2, wherein if the candidate object having a similarity to the feature of the tracking target greater than a predetermined threshold is not detected from the current image, the feature acquired from the past image is updated as the feature of the tracking target.

4. 4. The information processing device according to claim 2, wherein the updating means updates the position of the tracking target in the current image when the candidate object having a similarity to the feature amount of the tracking target in the past image that is greater than a predetermined threshold is detected in the current image, and does not update the position of the tracking target when the candidate object having a similarity to the feature amount of the tracking target that is greater than a predetermined threshold is not detected in the current image.

5. 5. The information processing apparatus according to claim 1, further comprising a storage unit that stores a feature quantity of an object designated by a user as the feature quantity of the tracking target.

6. The method further comprises extracting means for extracting a partial image for detecting the candidate object from the current image based on the position of the candidate object detected from the one or more previous images; 6. The information processing apparatus according to claim 1, wherein the acquiring means acquires the feature amount of the object from the current image based on the partial image of the current image extracted by the extracting means.

7. 7. The information processing apparatus according to claim 6, wherein said extracting means extracts a partial image of a predetermined size from an area of ​​said current image corresponding to the vicinity of said candidate object detected from said past image.

8. 8. The information processing device according to claim 1, further comprising: a learning unit that updates parameters of the trained model based on training data indicating a position of the tracking target in an image.

9. The tracking system further includes a calculation means for acquiring a loss for an estimated position of an object similar to the tracking target in the image based on teacher data indicating the position of the tracking target in the image, The information processing device according to claim 8 , wherein the learning means updates parameters of the trained model based on the loss acquired by the calculation means.

10. The method further comprises: determining whether or not there is an occlusion area in which the candidate object is occluded based on the position of the candidate object in the current image; 10. The information processing device according to claim 1, wherein the detection means identifies a correspondence between the candidate object in the one or more past images and the candidate object in the current image based on the result of the determination by the determination means.

11. 11. The information processing device according to claim 1, wherein when the determination means determines that the tracking target in the current image is occluded, the detection means identifies an object occluding the tracking target in the current image as the tracking target.

12. 12. The information processing device according to claim 10, wherein the determining means determines whether or not the occluded area exists for an object among the candidate objects in the past image whose similarity to the candidate object in the current image is less than a predetermined threshold.

13. 13. The information processing device according to claim 12, wherein the determination means determines, for a target object among the candidate objects in the past image whose similarity to the candidate object in the current image is smaller than a predetermined threshold, the degree of overlap with the area of ​​the target object based on the area of ​​the candidate object in the current image, and if the degree of overlap is greater than the predetermined threshold, determines that the target object is occluded.

14. the determining means determines, when the target object is occluded, the candidate object occluding the target object as an occluder; The information processing apparatus according to claim 13 , wherein the specifying means specifies the position of the occluder as the position of the target object.

15. A program for causing a computer to function as each of the means included in the information processing device according to any one of claims 1 to 14.

16. An information processing method for tracking a specific object from images captured at multiple times, the method comprising: based on a trained model, selecting from one or more past images features of a tracking target designated by a user and candidate objects including an object selected based on the tracking target; and feature amounts of the following features: an acquisition step of acquiring feature quantities of each of the candidate objects in the current image; a detection step of detecting a candidate object similar to the tracking target from the current image and a candidate object similar to the object selected based on the tracking target, based on features of each of the candidate objects in the one or more previous images and features of each of the candidate objects in the current image; a tracking step of tracking the tracking target using a feature amount of the tracking target in the current image determined based on a result of the detection process.

Images