Image processing device, image processing method, and computer program for executing the method

Abstract

When tracking a specific subject within an image, the system minimizes computational load while maintaining high accuracy in tracking the subject. [Solution] The image processing device comprises an image acquisition unit that acquires input images in a time series, an object detection unit that creates object region information from the input images, similar region detection unit that creates similar region information from the input images, and a storage unit that stores the object region information and similar region information. The object detection unit creates object region information using the object region information and similar region information of past frames stored in the storage unit. Furthermore, the object detection unit and the similar region detection unit are configured using a neural network.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, and particularly to an image processing apparatus suitable for tracking a specific subject in an image with high accuracy while suppressing the amount of calculation when tracking the specific subject.

Background Art

[0002] In recent years, image processing technologies have been utilized in various fields such as surveillance systems, autonomous vehicles, and sports analysis. In particular, a technology for tracking a specific subject in a video plays an important role. Further, with the development of image processing technologies, a technology for efficiently and accurately tracking a specific subject in an image is required in the fields of surveillance cameras, robotics, and video analysis.

[0003] As technologies for tracking a specific subject in an image, those using conventional luminance or color information and template matching have been known. On the other hand, in recent years, a method using Deep Neural Network (hereinafter simply abbreviated as "DNN"), which is a type of deep learning, has attracted attention as a high-precision tracking technology.

[0004] For example, Non-Patent Document 1 describes a method for tracking a specific subject within an image. In this method, the image containing the target to be tracked and the image representing the search area are input into a Convolutional Neural Network (hereinafter simply abbreviated as "CNN") with identical weights. The cross-correlation between the features obtained from these inputs is then calculated to identify the location of the target within the search area image. Recently, Vision Transformer (Non-Patent Document 2), which applies the structure of a Transformer to image recognition, has emerged, further improving the accuracy of image recognition. Here, a Transformer is a deep learning model architecture characterized by its ability to efficiently capture the relationships between important parts within the input data using a Self-Attention Mechanism. While such tracking methods can accurately identify the location of the target to be tracked, they are prone to errors such as tracking the wrong target when similar objects overlap on the screen.

[0005] To avoid this, there are methods, such as the one described in Patent Document 1, that create a histogram from the color features and depth information of the detected object's region, examine its changes, and determine whether or not the object is occluded, thereby improving tracking accuracy. In addition, the method described in Non-Patent Document 4 uses a DNN to individually infer the region of each target to be tracked, and performs object tracking based on the inference results obtained from past frames and image features. [Prior art documents] [Patent Documents]

[0006] [Patent Document 1] U.S. Patent Application Publication No. 2018 / 0012078 [Non-patent literature]

[0007] [Non-Patent Document 1] Luca Bertinetto, et al., “Fully-Convolutional Siamese Networks for Object Tracking”, [online], 30 Jun 2016, [Retrieved December 25, 2020], arXiv, Internet<URL:https: / / arxiv.org / pdf / 1606.09549> [Non-Patent Document 2] Alexey Dosovitskiy, et al., “An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale”, [online], 22 Oct 2020, [Retrieved December 25, 2024], arXiv, Internet<URL: https: / / arxiv.org / pdf / 2010.11929> [Non-Patent Document 3] Liang-Chieh Chen, et al., “DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected”, [online], 2 Jun 2016, [Retrieved December 25, 2024], arXiv, Internet<URL: https: / / arxiv.org / pdf / 1606.00915> [Non-Patent Document 4] Yuanyou Xu, et al., “A Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation”, [online], 22 Oct 2020, [Retrieved December 25, 2024], arXiv, Internet<URL: https: / / arxiv.org / abs / 2308.13266> [Overview of the project] [Problems that the invention aims to solve]

[0008] The method described in Non-Patent Document 4 involves saving a map with multiple channels over the past few frames in order to track multiple targets simultaneously. Therefore, the transfer time of the map is added, and additional calculations must be performed using the map as input. Consequently, this method has the drawback of tending to be computationally intensive.

[0009] The object of the present invention is to provide an image processing device that can track a specific subject in an image with high accuracy while reducing the amount of computation required. [Means for solving the problem]

[0010] The image processing apparatus of the present invention preferably comprises an image acquisition unit that acquires input images in a time series, an object detection unit that creates object region information from the input images, similar region detection unit that creates similar region information from the input images, and a storage unit that stores the object region information and similar region information, wherein the object detection unit creates object region information using the object region information and similar region information of past frames stored in the storage unit. [Effects of the Invention]

[0011] According to the present invention, it is possible to provide an image processing device that can accurately track a specific subject in an image while reducing the amount of computation required. [Brief explanation of the drawing]

[0012] [Figure 1] This is a functional configuration diagram of the image processing apparatus according to Embodiment 1. [Figure 2] This is a hardware and software configuration diagram of an image processing device. [Figure 3]It is a flowchart showing a series of processes for obtaining a target region and a similar region related to a tracking target of an image processing apparatus according to Embodiment 1. [Figure 4] It is a diagram for explaining the state of detecting a tracking target from an image. [Figure 5] It is a diagram for explaining the flow of processing related to a target region and a similar region regarding a tracking target of an image processing apparatus according to Embodiment 1. [Figure 6] It is a diagram for explaining the flow of learning processing related to a target region and a similar region regarding a tracking target of an image processing apparatus according to Embodiment 1. [Figure 7] It is a functional configuration diagram of an image processing apparatus according to Embodiment 2. [Figure 8] It is a diagram for explaining the flow of processing related to a target region and a similar region regarding a tracking target of an image processing apparatus according to Embodiment 2. [Figure 9] It is a functional configuration diagram of an image processing apparatus according to Embodiment 3. [Figure 10] It is a diagram for explaining the flow of learning processing related to a target region and a similar region regarding a tracking target of an image processing apparatus according to Embodiment 3. [Figure 11] It is a diagram for explaining the flow of processing related to a target region and a similar region regarding a tracking target of an image processing apparatus according to Embodiment 4.

Modes for Carrying Out the Invention

[0013] Hereinafter, each embodiment according to the present invention will be described with reference to FIGS. 1 to 11.

[0014] 〔Embodiment 1〕

[0015] Hereinafter, Embodiment 1 according to the present invention will be described with reference to FIGS. 1 to 6.

[0016] This embodiment relates to an image processing device that realizes the function of detecting and tracking a person from a video or a series of still image frames (hereinafter also simply referred to as "input image"). The scope of application of the present invention is not limited to the category of objects to be detected and tracked, but in this embodiment, an example in which the target is a person will be described. In the person tracking technology of this embodiment, person tracking is realized by detecting the same person appearing in consecutive frames. The situation dealt with by this embodiment is particularly assumed to be the shooting of a sporting event, where the clothing and direction of movement of people are similar and they frequently come into close proximity and cross paths. In such a case, if people with similar visual characteristics such as position and color are considered to be the same person across each frame, it is easy to make a mistake in tracking. In this embodiment, such failures in person tracking will be referred to as false tracking.

[0017] In this invention, in order to suppress the occurrence of such false tracking, we focus on objects that look similar to the target being tracked. We aim to improve tracking accuracy by using information about the regions in which the target being tracked and objects that look similar to the target were located in past frames to track objects, thereby suppressing the occurrence of false tracking.

[0018] First, the configuration of the image processing apparatus according to Embodiment 1 will be described using Figures 1 and 2.

[0019] Figure 1 is a functional configuration diagram of an image processing apparatus according to Embodiment 1.

[0020] Figure 2 is a hardware and software configuration diagram of the image processing device.

[0021] As shown in Figure 1, the image processing apparatus according to this embodiment consists of an image acquisition unit 101, a target detection unit 102, a similar region detection unit 103, and a storage unit 110.

[0022] The image acquisition unit 101 is a functional unit that acquires input images. Specifically, it sequentially acquires image frames of a video or a series of still images of a person from the storage unit 110. The image acquisition unit 101 also has the function of acquiring specific information or regions from the image frames. The image acquisition unit 101 may acquire images not only from the storage unit 110, but also from an imaging device connected to the image processing device 100, or from other external devices.

[0023] The target detection unit 102 is a functional unit that detects a specific person and their region from the image acquired by the image acquisition unit 101, similar region information calculated from past frames on the time axis, and the target region.

[0024] The similarity region detection unit 103 is a functional unit that detects regions of objects similar to the tracked object. The similarity region detection unit 103 can be implemented, for example, by a neural network.

[0025] The memory unit 110 is a functional unit that stores data and programs handled by the image processing device.

[0026] The image processing device 100 consists of a CPU 201, ROM 202, main memory 203, communication device 204, input / output device 205, display device 206, and auxiliary storage device 210 as hardware, as shown in Figure 2.

[0027] The CPU (Central Processing Unit) 201 controls each part of the image processing device and executes programs loaded into the main memory 203 to implement its functions.

[0028] ROM (Read Only Memory) 202 is a non-volatile semiconductor memory device that stores programs such as firmware and BIOS that cannot be rewritten.

[0029] Main memory 203 is implemented using RAM (Random Access Memory) and is a volatile semiconductor memory device that temporarily stores programs and work data referenced by the CPU 201.

[0030] The communication device 204 is a device that connects the image processing device 100 to a network, and the communication device 204 can communicate with other devices such as a camera or a server.

[0031] The input / output device 205 includes a keyboard, touch panel, mouse, printer, etc., and accepts user input. It is used for setting tracking targets and outputting information, among other things.

[0032] The display device 206 is, for example, a liquid crystal display, and displays images, subjects, and tracking results to the user.

[0033] The auxiliary storage device 210 is, for example, a large-capacity storage device such as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The auxiliary storage device 210 in this embodiment has an image acquisition program 221, a target detection program 222, and a similar area detection program 223 installed. The image acquisition program 221, target detection program 222, and similar area detection program 223 are computer programs that implement the functions of the image acquisition unit 101, target detection unit 102, and similar area detection unit 103, respectively. It also stores image data, setting data, and tracking results that are the target of processing in this embodiment. The image data includes time-series video frames and still image data. The metadata for each frame includes the object (e.g., shooting time, camera position). The configuration data includes the initial position, size, or category of the target to be tracked (e.g., person, vehicle), thresholds, and learning parameters. The data showing the tracking results includes the detected coordinates of the target to be tracked (e.g., position and size of the bounding box), the label or identifier of the target to be tracked, a list of similar regions, and features. As a device for storing programs and data, the media of the auxiliary storage device 210 may be an HDD, SSD, optical media, or flash memory card.

[0034] Next, a series of processes for determining the target region and similar region related to the tracking target of the image processing apparatus according to Embodiment 1 will be described using Figures 3 to 5.

[0035] Figure 3 is a flowchart showing a series of processes for determining the target region and similar region related to the tracking target of the image processing device according to Embodiment 1.

[0036] Figure 4 illustrates how the tracking target is detected from the image.

[0037] Figure 5 is a diagram illustrating the processing flow for the target region and similar region related to the tracking target of the image processing device according to Embodiment 1.

[0038] The image acquisition unit 101 of the image processing device 100 sequentially acquires image frames of stored video and continuous still images of people (S101).

[0039] Next, the image acquisition unit 101 acquires the target region and similar region information, which is information about similar regions calculated over the past N (N is a natural number) frames (S102). Here, a similar region is a region of an object similar to the tracking target (details will be described later). In this embodiment, N=3 is used for explanation, but any number of frames can be used as long as it is one or more frames determined in advance. However, the number of N frames for which similar region information is used and the target region are fixed in advance during training and are not changed during calculation by the target detection unit 102 and the similar region detection unit 103.

[0040] Next, the object detection unit 102 of the image processing device 100 detects a specific person and their region from the image acquired in S101 and S102, similar region information calculated from past N frames, and the target region (S103). Any method can be used to detect an object within the image; for example, a known technique is the method shown in Non-Patent Document 1.

[0041] Next, the target detection unit 102 of the image processing device 100 stores information about the detected target area in the storage unit 110 (S104).

[0042] Next, the similarity region detection unit 103 of the image processing device 100 detects similar regions relating to the region of an object having similar features to the tracking target and calculates similarity region information (S105). The process of calculating similarity region information will be explained in detail later.

[0043] Next, the similarity region detection unit 103 of the image processing device 100 stores the similarity region information calculated in S105 in the storage unit 110 (S106).

[0044] Next, the image processing device 100 determines whether the video frame to be processed has finished (S110). If it has finished (S110:YES), it terminates the processing; otherwise, if it has not finished (S110:NO), it returns to S101 and repeats the processing.

[0045] Next, using Figures 4 and 5, we will explain the process of detecting a specific tracking target within an image, as well as related concepts such as similar regions and similar region information.

[0046] Figure 4 shows how a specific tracking target was detected within the image. Figure 4 shows images 310, 311, 312, 313, and 314 in chronological order. In image 310, object regions 310-1 and 310-2 are detected; in image 311, object region 311-1 is detected; in image 312, object region 312-1 is detected; and in image 313, object region 313-1 is detected.

[0047] Here, when a person is an object, the object region is described as the region including the head, torso, limbs, etc. of the person as the tracking target, as shown in object region 310-1. However, the tracking target may consist of only a part of the torso, and is not limited to a region including the entire body.

[0048] Here, the target region is defined as the object region when the tracking target is a person in object region 311-1. Furthermore, when the tracking target is a person in object region 311-1, the similar region is defined as the object region of a person having similar characteristics to the tracking target as shown in object region 310-2, and the similar region information is information related to that similar region.

[0049] Similar region information is used for training and inference by inputting the RGB image 401 in Figure 5. Details of training and inference using similar region information will be explained later.

[0050] A similarity region is, for example, if the tracking target is a person, a region that possesses the characteristics of the person (shape and color). This can be the entire person, as in object region 301-2, or a part of the person (head, torso, etc.). In this case, the target region is also included in the set of similarity regions. That is, the target region itself is considered a similarity region, and the information of the target region is used as similarity region information. This makes it easier to train and to perform inference. Further details will be explained later.

[0051] The detection of the target region in S103 of Figure 3 is a type of semantic segmentation recognition task and can be realized using known methods such as those described in Non-Patent Literature 3. Non-Patent Literature 3 describes a technique for identifying object boundaries using a deep convolutional network.

[0052] Figure 5 shows how the target detection unit 102 calculates information about the target region. The neural network NN1 that determines the region to be tracked is a mechanism with the function described in Non-Patent Literature 3. When the RGB image 401 and the target region information 410, 411, 412 and similar region information 417, 418, 419 of the past N frames are input to the neural network NN1, it detects where the region of the person in question corresponds in the image and outputs the target region information 413.

[0053] On the other hand, the similarity region detection unit 103 is composed of a neural network NN2 that determines regions similar to the tracking target, and is a mechanism with the same function as described in Non-Patent Document 3. When an RGB image 401 is input, it detects and integrates regions of people similar to the tracking target and outputs similarity region information 422. In this embodiment, the target region information and similarity region information are calculated with a likelihood score of 0 if they are estimated to be the region of the object, and 1 otherwise. When outputting the region of a person similar to the tracking target as the score itself, the detected map is used as similarity region information 422. It is also conceivable that the detected result is an HxWxC feature, and that these features are integrated by adding them in the channel direction to obtain similarity region information 422. However, in the case of the first video frame, since there are no past frames, initial target region information and similarity region information are generated and input as target region information and similarity region information for the past N frames. The generation of initial target region information and similarity region information will be explained later. The neural network 402 is pre-trained to produce such an output for the input image (the training of target region information and similar region information will be described later). While the target region information, in ideal cases, could output the entire body of the tracked object, the actual results may include noise, such as parts of the limbs being missing, as seen in target region information 413.

[0054] In addition to the RGB image 401, a derivative configuration is also conceivable in which a 2.5-dimensional depth image is acquired separately using a dedicated sensor or the like. In this case, the depth image is used as the input, and the 4-channel information, which is concatenated with the 3-channel RGB image 401, is used for learning and recognition as the image input instead of the RGB image. This makes it possible to improve the accuracy of information about occluded areas.

[0055] Here, we will explain how to generate initial target area information.

[0056] If the target area information for the tracking target cannot be obtained, the initial target area information input to the target detection unit 102 in the initial frame will use a map with all zeros. Alternatively, if the target area can be obtained in advance from the first video frame, N copies will be created and used as the initial target area information. As an example, a method for creating the initial target area using the segmentation of the tracking target in the first video frame will be described. The segmentation information of the tracking target in the first frame is calculated in advance using the technology described in Non-Patent Literature 2, and the obtained segmentation map is copied for N frames. Then, the copied segmentation map is input to the target detection unit 102 as the initial target area information.

[0057] Next, we will explain how to generate initial similarity region information.

[0058] The similar region information obtained by inputting the initial input image to the similar region detection unit 103 is copied for N frames (in this embodiment, N=3) to become the initial similar region information. When the target detection unit 102 calculates the target region information from the initial frame, the initial frame, the initial target region information generated by the above method, and the similar region information obtained by this method are input to the target detection unit 102.

[0059] Next, we will explain the detection and inference of target region information and similar region information.

[0060] In this embodiment, the visible area of ​​an object is estimated as the target area information and similar area information. In particular, in this embodiment, the area is described as being colored to represent the entire body of a person, but the information is not limited to this format as long as it indicates where the target is located. For example, it could be a map where only the torso is 1 and the rest is 0, or a map where a specific value is placed near where the tracked target is located. It could also be a map of the central area of ​​the object. The central area of ​​the object referred to here could be in a form that estimates a region in the shape of a Gaussian function.

[0061] The target detection unit 102 learns to infer target region information for the region of the target being tracked. On the other hand, the similar region detection unit 103 learns to infer similar region information for regions of objects that have similar features to the target being tracked, as performed in step S105. Similar features are learned to respond according to the characteristics of objects that could be tracked, such as whether or not they have a human-like shape, whether or not they have similar colors, whether or not they have similar postures, etc. Similar region information is also described as the whole body region of a similar person, similar to the target region, but it is not limited to this format as long as it is information that indicates which region the target is located in.

[0062] Next, we will explain how to train the target detection unit and the similarity region detection unit using Figure 6.

[0063] Figure 6 is a diagram illustrating the processing flow for learning the target region and similar region related to the tracking target of the image processing device according to Embodiment 1.

[0064] The learning method for the target detection unit 102 is as follows.

[0065] The neural network NN1, as described in Non-Patent Document 3, outputs a map 404 of target region information based on three types of information. One is an RGB image 401, which is an input image prepared for training (hereinafter referred to as the "training image"). The other is target region information 410, 411, 412 from past frames, and similar region information 417, 418, 419. Non-Patent Document 3 describes a method for estimating the foreground region of a specific category of object, but here, a target region training value 431 is provided, and NN1 is trained so that a map similar to the training value 431 can be obtained through estimation. The training value for the target region is provided as information representing the entire region of a person, or information representing regions related to the same attribute (category), such as a part of the whole body. For example, information representing the entire region of a person uses a map where 1 is used for the region including the head, torso, and limbs, and 0 is used for the background and other parts. However, the information representing a region is not limited to values ​​of 0 or 1. Specifically, the output map 404 is compared with the target values ​​431, and the loss value 432 is calculated using known methods such as cross-entropy and squared error. The weight parameters of the neural network NN1 are adjusted using backpropagation or the like so that the loss value gradually decreases. (For this process, a method similar to that in Non-Patent Document 3 can be used. A sufficient amount of training images and target values ​​must be provided for the training to be meaningful. Since creating target values ​​for overlapping object regions is costly, it is also conceivable to create training data using computer graphics or by using image synthesis methods that cut out and superimpose object images.)

[0066] On the other hand, the learning method for the similarity region detection unit 103 is as follows.

[0067] The neural network NN2, as shown in Non-Patent Document 3, outputs similarity region information 422 based on the RGB image 401, which is the training image. If the target being tracked is a person, the region belonging to the person category is used as the target value 435. In this case, the target value 435 of the region belonging to the person category also includes the target region's target value 431. This allows the similarity region detection unit 103 to learn to respond to regions that resemble people. In addition, if a region has features similar to the target being tracked, it is also conceivable to assign a target value 435 that represents a part of the object. The neural network NN2 is trained using such target values ​​435. Specifically, the output result map 408 and the target values ​​435 are compared, and the loss value 432 is calculated using known methods such as cross-entropy and squared error. The weight parameters of the neural network NN2 are adjusted using backpropagation or the like so that the loss value gradually decreases. By including the training values ​​431 of the target detection unit 102 in the training values ​​435 of the similar region information, it becomes unnecessary to provide the similar region detection unit 103 with template images or time-series information to determine which target to track in advance. As a result, the similar region detection unit 103 can learn to solve recognition problems like those described in Non-Patent Document 3, thus improving the efficiency of learning.

[0068] As described above, according to this embodiment, the similarity region detection unit 103 only needs to learn to find regions that have a similar human-like quality to the person being tracked, thus solving the Semantic Segmentation problem as seen in Non-Patent Document 3, etc. Therefore, there is no need to identify the features of each instance, and the effect of more stable learning can be obtained.

[0069] Furthermore, the target detection unit 102, which detects the target to be tracked, can be processed with fewer computing resources compared to the method described in Non-Patent Document 4, which individually labels each instance and feeds back region information. If the region information and similar region information are each represented as a 1-channel map per frame, then, compared to Non-Patent Document 4, which has information with as many channels as there are object IDs, it becomes possible to detect the target to be tracked using an even smaller amount of information. Therefore, the image processing device of this embodiment can achieve both efficient use of computing resources and improved accuracy.

[0070] [Embodiment 2]

[0071] Hereinafter, Embodiment 2 of the present invention will be described with reference to Figures 7 and 8.

[0072] In this embodiment, the image processing apparatus of Embodiment 1 is modified to include a function for extracting features from an image, enabling the detection of the target being tracked and the detection of similar regions. This embodiment will be described primarily for its differences from Embodiment 1.

[0073] First, the configuration of the image processing apparatus according to Embodiment 2 will be described using Figure 7.

[0074] Figure 7 is a functional configuration diagram of the image processing apparatus according to Embodiment 2.

[0075] As shown in Figure 7, the image processing apparatus 100 according to Embodiment 2 has a feature extraction unit 120 added to the functional configuration of Embodiment 1. The feature extraction unit 120 is a functional unit that calculates image features based on the RGB image 401.

[0076] Next, using Figure 8, we will explain the processing flow for the target region and similar region related to the tracking target of the image processing device.

[0077] Figure 8 is a diagram illustrating the processing flow for the target region and similar region related to the tracking target of the image processing device according to Embodiment 2.

[0078] As shown in Figure 8, the similarity region detection unit 103 of this embodiment calculates similarity region information 422 as described in Embodiment 1 based on the image features calculated by the feature extraction unit 120 for the RGB image 401. On the other hand, the target detection unit 102 calculates target region information based on the feature quantities calculated by the feature extraction unit 120, as well as target region information 410, 411, 412 and similarity region information 417, 418, 419 calculated from past frames.

[0079] In initial frames where target region information and similar region information cannot be obtained in advance, initial target region information and initial similar region information are calculated, as in Embodiment 1, and input to the target detection unit 102 along with the feature quantities.

[0080] The feature extraction unit 120 is composed of a neural network NN3. For training the feature extraction unit 120, the target region information (teacher value 431) and similar region information (teacher value 435) shown in Figure 6 are used for training the neural network NN3, similar to the loss values ​​used in training in Embodiment 1. In Embodiment 1, the target detection unit 102 and the similar region detection unit 103 calculated loss values ​​separately, but in this embodiment, the loss values ​​are calculated and training is performed simultaneously. This ensures that the feature extraction unit 120 learns image features that include information about both the target region and the similar region. When calculating loss values ​​simultaneously, the weight parameters of the neural network NN2 are adjusted using backpropagation or the like so that the weighted sum of the two types of loss values ​​gradually decreases. The first type of loss value is the loss value of the target region information and the teacher value output by the target detection unit 102, and the second type of loss value is the loss value of the similar region information and the teacher value output by the similar region detection unit 103. Besides simultaneous calculation, there are also methods such as alternating training in each iteration.

[0081] As described above, according to this embodiment, target region information and similar region information can be calculated based on the image feature quantities calculated by the common feature extraction unit 120, thereby reducing the amount of computation. Furthermore, since the target detection unit 102 and the similar region detection unit 103 can be trained simultaneously or alternately (end to end) using each loss value, training can be performed more efficiently.

[0082] [Embodiment 3]

[0083] Hereinafter, Embodiment 3 of the present invention will be described with reference to Figures 9 and 10.

[0084] In this embodiment, the image processing apparatus of Embodiment 1 is modified to include a function that converts the target area information and similar area information of the tracking target into compressed data, thereby enabling the detection of the tracking target and its similar areas. This embodiment will also be described focusing on the differences from Embodiment 1.

[0085] First, the configuration of the image processing apparatus according to Embodiment 3 will be described using Figure 9.

[0086] Figure 9 is a functional configuration diagram of the image processing apparatus according to Embodiment 3.

[0087] As shown in Figure 9, the image processing apparatus 100 according to Embodiment 3 has a region information compression unit 130 added to the functional configuration of Embodiment 1. The region information compression unit 130 is a functional unit that converts target region information and similar region information into data-compressed information.

[0088] Next, using Figure 10, we will explain the processing flow for the target region and similar region related to the tracking target of the image processing device.

[0089] Figure 10 is a diagram illustrating the processing flow for learning the target region and similar region related to the tracking target of the image processing device according to Embodiment 3.

[0090] The process by which the target detection unit 102 and the similarity region detection unit 103 calculate target region information and similar region information using the RGB image 401 as input is performed by neural networks NN1 and NN2, as in Embodiment 1.

[0091] In this embodiment, the target region information and similar region information calculated based on past frames are further transformed using a neural network NN4 that converts them into features with less information. Based on the target region information and similar region information, the neural network NN4 calculates the respective target information and similar region information. Subsequently, the target information and similar region information are input to the neural network NN1 of the target detection unit 102 and used for calculating the target region information 413. For example, the target information and similar region information are 1x1xC tensors with a size smaller than the width and height of the target region information and similar region information. This tensor is an arbitrary value obtained by the neural network through learning and optimization. The number of dimensions C of the transformed features is a value smaller than the area of ​​the target region information and similar region information. By setting a sufficiently small number of dimensions C, even if the computation time from the input layer onward of the neural network NN4 and the neural network NN1 of the target detection unit 102 is added together, the computation time can be reduced compared to when the target region information and similar region information are input. As a result, the total computation time of the neural networks NN1, NN2, and NN4 can be kept smaller than in Embodiment 1.

[0092] The target information and similar information may, for example, be features with half the width and height of the target region information, or features with a width and height of 1 and multiple elements in the channel method. The neural network NN4 is trained by preparing and inputting N frames of target region information and similar region information training values ​​431 and 435 corresponding to previously prepared past frames. The calculated target information and similar information are input to the neural network NN1 of the target detection unit 102 along with the RGB image 401, and the loss value is calculated using the calculated target region information and the training value 431 of the target region information. The parameter update method of the neural network NN4 using this loss value is the same as that of the target detection unit 102 described in Embodiment 1.

[0093] As described above, according to this embodiment, the target region information and similar region information can be handled with an even smaller amount of information than in Embodiment 1, and the computational load of the target detection unit 102 can be reduced. Furthermore, since the target detection unit 102 and the region information compression unit 130 can learn simultaneously or alternately (end to end) using each loss value during learning, it is possible to learn in a way that allows for more efficient conversion of information regarding the target region and regions similar to the target region.

[0094] [Embodiment 4]

[0095] Hereinafter, Embodiment 4 of the present invention will be described with reference to Figure 11.

[0096] Embodiment 2 described an image processing device that takes an RGB image as input to a neural network that performs feature extraction, performs learning and inference, and tracks an object. This embodiment further inputs another image to the neural network that performs feature extraction, calculates template features, and uses them to calculate target region information and similar region information. This embodiment will mainly be described in terms of the differences from Embodiment 2.

[0097] The following explanation uses Figure 11 to describe the processing flow for the target region and similar region related to the tracking target of the image processing device.

[0098] Figure 11 is a diagram illustrating the processing flow for the target region and similar region related to the tracking target of the image processing device according to Embodiment 4.

[0099] In this embodiment, the feature extraction unit 120 newly inputs an RGB image 440 obtained at a different time from the RGB image 401 (hereinafter referred to as the "template image") 440 into the neural network NN3 of the feature extraction unit 120 to calculate template features 441 and image features 442 (feature extraction). The target region information 410, 411, 412 and similar region information 417, 418, 419 calculated from the RGB image 401 and past frames are input into the neural network NN1 of the target detection unit 102, as in the previously described embodiment 2. Furthermore, in this embodiment, the template features 441 are also input into the neural network NN1 of the target detection unit 102 at the same time. Then, based on the template features, target region information and similar region information, the neural network NN1 calculates target region information 413.

[0100] The learning of the feature extraction unit 120, the target detection unit 102, and the similarity region detection unit 103 is the same as in Embodiment 2. The image processing device 100 calculates the target region information and similarity region information of the target detection unit 102 and the similarity region detection unit 103 simultaneously or alternately, calculates the loss value between these and the target value of the target region information 413 and the target value of the similarity region information 422, and performs learning. The weighting of the loss value and the learning method of the neural network are the same as in Embodiment 2.

[0101] As described above, according to this embodiment, feature extraction can be performed on common images in the same way as in Embodiment 2, and target area information and similar area information can be calculated, thereby reducing the amount of computation. Furthermore, it is possible to detect the tracked target while retaining template features of a past frame using a template image of the tracked target. Therefore, it is possible to obtain the effect of detecting the tracked target more robustly, similar to the tracking methods seen in Non-Patent Document 1, etc. As an example, consider a complex situation in a sports scene such as soccer where another person overlaps with the tracked target. In this case, a template image in which the tracked target is not crowded is retained in advance. The template features calculated from the retained template image do not include the features of similar people, so it is easy to capture the features of the tracked target only. If the tracked target can be detected based on such template features, the risk of mistakenly detecting an adjacent person as the tracked target can be reduced because the features of other people (such as posture and color) are not mixed in. This makes it possible to achieve more robust tracking target detection.

[0102] (Composition 1) An image acquisition unit that acquires input images in chronological order, A target detection unit creates target area information from the input image as target area information, A similar region detection unit creates similar region information from the input image, which is information about one or more similar regions that are regions of an object similar to the tracking target. An image processing apparatus comprising a storage unit for storing the target region information and the similar region information, The image processing apparatus is characterized in that the target detection unit creates the target region information using the target region information and similar region information of past frames stored in the storage unit.

[0103] (Configuration 2) The image processing apparatus according to configuration 1, characterized in that the similarity region detection unit estimates the region of the attribute to which the tracking target belongs, and creates the similarity region information including the target region of the tracking target.

[0104] (Composition 3) Furthermore, it has a feature extraction unit, The target detection unit extracts image features to create the target region information, An image processing apparatus according to either configuration 1 or configuration 2, characterized in that the image feature quantities extracted by the similarity region detection unit to create the similarity region information are extracted by the same feature extraction unit.

[0105] (Composition 4) Furthermore, it has a region information compression unit that compresses and converts the target region information or the similar region information into region information of a smaller size, The image processing apparatus according to any one of configurations 1 to 3, characterized in that the target detection unit creates the target region information and the similar region information based on the region information compressed and converted by the region information compression unit.

[0106] (Composition 5) The image processing apparatus according to configuration 3, characterized in that the target detection unit calculates the target region information based on the image feature quantities obtained by the feature extraction unit based on the first input image and an input image at a different time from the first input image, the image feature quantities obtained based on the first input image, the target region information of past frames and the similar region information.

[0107] (Composition 6) The image processing apparatus according to any one of configurations 1 to 5, characterized in that the target detection unit and the similar region detection unit are composed of a neural network.

[0108] (Composition 7) The image processing apparatus according to configuration 3, characterized in that the feature extraction unit is composed of a neural network.

[0109] (Composition 8) The image processing apparatus according to configuration 4, characterized in that the region information compression unit is composed of a neural network.

[0110] (Method 1) An image processing method for tracking a specific object within an image using an image processing device, The image processing device performs an input image acquisition step of acquiring input images in a time series, The image processing device performs an object detection step of creating information about the target region of the object to be tracked from the input image as target region information, The image processing device performs a similar region detection step of creating similar region information from the input image, which is information about one or more similar regions that are regions of an object similar to the tracking target. The image processing apparatus has a storage step for storing the target region information and the similar region information, The image processing method is characterized in that the target detection step creates target region information using the target region information and similar region information of past frames stored by the storage step.

[0111] (Program 1) A computer program that causes a computer to perform each step described in Method 1. [Explanation of Symbols]

[0112] 100...Image processing device, 101...Image acquisition unit, 102...Target detection unit, 103...Similar region detection unit, 110...Storage unit, 120...Feature extraction unit, 130...Region information compression unit, NN1, NN2, NN3, NN4... Neural networks, 201...CPU, 202...ROM, 203...Main memory, 204...Communication device, 205...Input / output device, 206...Display device, 210...Auxiliary storage device

Claims

1. An image acquisition unit that acquires input images in chronological order, A target detection unit creates target area information from the input image as target area information, A similar region detection unit creates similar region information from the input image, which is information about one or more similar regions that are regions of an object similar to the tracking target. An image processing apparatus comprising a storage unit for storing the target region information and the similar region information, The image processing apparatus is characterized in that the target detection unit creates the target region information using the target region information and similar region information of past frames stored in the storage unit.

2. The image processing apparatus according to claim 1, characterized in that the similarity region detection unit estimates the region of the attribute to which the tracking target belongs and creates the similarity region information including the target region of the tracking target.

3. Furthermore, it has a feature extraction unit, The target detection unit extracts image features to create the target region information, The image processing apparatus according to claim 1, characterized in that the image feature quantities extracted by the similarity region detection unit to create the similarity region information are extracted by the same feature extraction unit.

4. Furthermore, it has a region information compression unit that compresses and converts the target region information or the similar region information into region information of a smaller size, The image processing apparatus according to claim 1, characterized in that the target detection unit creates the target region information and the similar region information based on the region information compressed and converted by the region information compression unit.

5. The image processing apparatus according to claim 3, wherein the target detection unit calculates the target region information based on the image feature quantities obtained by the feature extraction unit based on the first input image and an input image at a different time, the image feature quantities obtained based on the first input image, the target region information of past frames and the similar region information.

6. The image processing apparatus according to claim 1, characterized in that the target detection unit and the similar region detection unit are configured by a neural network.

7. The image processing apparatus according to claim 3, characterized in that the feature extraction unit is configured by a neural network.

8. The image processing apparatus according to claim 4, characterized in that the region information compression unit is configured by a neural network.

9. An image processing method for tracking a specific object within an image using an image processing device, The image processing device performs an input image acquisition step of acquiring input images in a time series, The image processing device performs an object detection step of creating information about the target region of the object to be tracked from the input image as target region information, The image processing device performs a similar region detection step of creating similar region information from the input image, which is information about one or more similar regions that are regions of an object similar to the tracking target. The image processing apparatus has a storage step for storing the target region information and the similar region information, The image processing method is characterized in that the target detection step creates target region information using the target region information and similar region information of past frames stored by the storage step.

10. A computer program for causing a computer to perform each of the steps described in claim 9.

Images