Information processing device, information processing method

Abstract

To provide an information processing device, an information processing method, and a computer program for stable obtainment of a detection result without depending on the number of detected candidates.SOLUTION: An information processing device 1 includes: first inference means which infers an attribute of an object in an input image and a weight according to the attribute of each of regions in the input image; and second inference means which infers the attribute of the object on the basis of the attribute and the weight inferred by the first inference means.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to a technique for inferring the attributes of objects in an image. [Background technology]

[0002] In recent years, machine learning technologies, particularly Deep Learning, have seen rapid development in areas ranging from image recognition and speech recognition to machine translation. Among these, object detection using Deep Learning has become a crucial task, with applications in various fields such as surveillance cameras, medical image recognition, and autonomous driving.

[0003] One technique for improving object detection performance is to ensemble multiple results. For example, in Random Forest, multiple decision trees are prepared, and the results predicted by each decision tree are ensembled to produce the final result, which suppresses overfitting of each decision tree and improves performance. This ensemble concept can also be applied to Deep Learning. For example, Non-Patent Document 2 presents a method where a single image is input into multiple different neural networks to obtain multiple object detection frames. Then, the coordinate values ​​of the obtained object detection frames are weighted by the confidence level of object detection and ensembled to improve object detection performance.

[0004] On the other hand, methods using multiple neural networks like this are difficult to apply to tasks requiring real-time performance because the computational complexity increases as the number of models increases. Non-patent document 1 uses attention to learn the relationships between multiple object detection candidate frames inferred by a neural network from a single image. By ensembling the estimated multiple object detection candidate frames based on attention, the object detection performance is improved. Rather than using multiple neural networks to create the ensemble, the performance is improved while suppressing the decrease in speed by ensembling multiple object detection candidates obtained from a single neural network. BACKGROUND ART NON-PATENT LITERATURE

[0005] NON-PATENT LITERATURE 1 Han Hu et al, “Relation Network for Object Detection”, The IEEE Conference on Computer Vision and Pattern Recognition 2018. NON-PATENT LITERATURE 2 Solovyev et al, “Weighted Boxes Fusion: combining boxes for object detection models.” SUMMARY OF THE INVENTION PROBLEM TO BE SOLVED BY THE INVENTION

[0006] In the method disclosed in Non-Patent Literature 1, weighting for object detection candidates is learned as attention. That is, in order to perform weighting, it is necessary to extract object detection candidates once, and the performance of the ensemble changes depending on the number of detected candidates. If only one candidate is detected, it is difficult to perform ensemble. The present invention provides a technique for stably obtaining detection results without depending on the number of detected candidates. MEANS FOR SOLVING THE PROBLEM

[0007] One aspect of the present invention includes: first inference means for inferring a size of an object in an input image and a weight related to the size of each region in the input image; second inference means for inferring the size of the object based on the size of the object inferred by the first inference means and the weight; comprising 、 The size of the object refers to the size of the rectangular region containing the object. The first inference means infers the attributes of an object based on a map obtained by the element-wise product of a map of the attributes of an object in the input image, which is obtained based on the cross-correlation between the features of an object in the input image and features in an image different from the input image, and a map representing the weights related to the attributes of each region in the input image. . EFFECT OF THE INVENTION

[0008] According to the configuration of the present invention, detection results can be obtained stably regardless of the number of detected candidates. [Brief explanation of the drawing]

[0009] [Figure 1] A block diagram showing an example of the hardware configuration of the information processing device 1. [Figure 2] A block diagram showing an example of the functional configuration of the information processing device 1. [Figure 3] A flowchart of the processing performed by the information processing device 1. [Figure 4] A diagram showing an example of a neural network configuration. [Figure 5] A diagram showing an example of the configuration of each layer. [Figure 6] (a) is a block diagram showing an example of the functional configuration of the integration unit 204, and (b) is a flowchart showing the details of the processing in step S304. [Figure 7] A diagram showing an example of input images and map sets. [Figure 8] A block diagram showing an example of the functional configuration of the information processing device 2. [Figure 9] A flowchart of the processing performed by the information processing device 2. [Figure 10] A flowchart of the processing performed by the information processing device 1. [Figure 11] (a) is a diagram showing an example of a template image, and (b) is a diagram showing an example of a search image. [Figure 12] A diagram showing an example of a map based on cross-correlation. [Figure 13] A flowchart of the processing performed by the information processing device 2. [Figure 14] (a) is a diagram showing an example configuration of map 1401, and (b) is a diagram showing an example configuration of map 1406. [Modes for carrying out the invention]

[0010] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention to the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, the same or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] This embodiment describes a case in which a task is performed to infer "object attributes" such as the position and size of objects in an input image. First, an example of the hardware configuration of the information processing device 1 that performs training of a neural network to realize such a task will be explained using the block diagram in Figure 1.

[0012] The CPU 101 executes various processes using computer programs and data stored in the ROM 102 and RAM 103. In doing so, the CPU 101 controls the operation of the entire information processing device 1 and executes or controls the various processes described as being performed by the information processing device 1.

[0013] ROM102 stores configuration data for the information processing device 1, computer programs and data related to the startup of the information processing device 1, computer programs and data related to the basic operation of the information processing device 1, and so on.

[0014] RAM 103 has an area for storing computer programs and data loaded from ROM 102 and memory unit 104, and a work area used by CPU 101 when executing various processes. Furthermore, RAM 103 has an area for storing data received from the outside via I / F 105. In this way, RAM 103 can provide various areas as appropriate.

[0015] The memory unit 104 is a large-capacity information storage device such as a hard disk drive, flash memory, or various optical media. The memory unit 104 stores the OS (operating system), computer programs and data that cause the CPU 101 to execute or control various processes described as being performed by the information processing device 1. The computer programs and data stored in the memory unit 104 are loaded into the RAM 103 as appropriate according to the control of the CPU 101 and become the target of processing by the CPU 101.

[0016] I / F105 functions as a communication interface for data communication with external devices via wired and / or wireless networks such as LANs and the Internet.

[0017] CPU101, ROM102, RAM103, memory unit104, and I / F105 are all part of the system bus. 106 It is connected to [the device]. Note that the configuration shown in Figure 1 is an example of a hardware configuration applicable to the information processing device 1, and can be modified or changed as appropriate.

[0018] Figure 2 shows an example of the functional configuration of the information processing device 1. In this embodiment, each functional unit in Figure 2 is assumed to be implemented by software (computer program). In the following, the functional units in Figure 2 may be described as the main processing units, but in reality, the functions of the functional units are realized by the CPU 101 executing the corresponding computer program. Note that the functional units shown in Figure 2 may also be implemented in hardware.

[0019] The process of "training a neural network that infers the position and size of an object as attributes from an input image," which is performed in such an information processing device 1, will be explained according to the flowchart in Figure 3.

[0020] In step S301, the image acquisition unit 201 acquires an input image that includes the object to be detected. In this embodiment, the object to be detected is a person and a dog, but it is not limited to these. Furthermore, the method of acquiring the input image is not limited to a specific method. For example, the image acquisition unit 201 may acquire one image as the input image from a group of images stored in the storage unit 104, or it may acquire an image captured by the imaging device as the input image via the I / F 105. Alternatively, the image acquisition unit 201 may acquire one image as the input image via the I / F 105 from a group of images stored in an external device such as a server device.

[0021] In the following section, to provide a more detailed explanation, we will describe the case in which the image acquisition unit 201 acquires the input image 701 shown in Figure 7(a) in step S301. The input image 701 includes a person 702 and a dog 703 as objects to be detected.

[0022] In step S302, the GT acquisition unit 202 acquires GroundTruth (hereinafter referred to as GT), which is the training data for the person 702 and the dog 703 contained in the input image 701 acquired by the image acquisition unit 201. The method of acquiring GT is not limited to a specific method, similar to that for the input image; it may be acquired from the storage unit 104 or from an external device via the I / F 105.

[0023] For example, the GT acquisition unit 202 acquires the GT map 704 shown in Figure 7(b) as a GT indicating the center positions of the person 702 and the dog 703 included in the input image 701 acquired by the image acquisition unit 201. Each element in the GT map 704 (each rectangle in Figure 7(b)) corresponds to each pixel or each image region (one image region consists of two or more pixels) in the input image 701. Element 705 is the element corresponding to the center position of the person 702 in the input image 701, and element 706 is the element corresponding to the center position of the dog 703 in the input image 701. Among the elements in the GT map 704, the values ​​of elements 705 and 706 are "1", and the values ​​of elements other than elements 705 and 706 are "0".

[0024] For example, the GT acquisition unit 202 acquires the GT map 707 in Figure 7(c) as a GT indicating the sizes of the person 702 and the dog 703 included in the input image 701 acquired by the image acquisition unit 201. Each element in the GT map 707 (each rectangle in Figure 7(c)) corresponds to each pixel or each image region in the input image 701. In the GT map 707, the value of the element that is positionally corresponding to element 705 is the size (height and width) of the person 702, and the frame 708 represents that size. Also, in the GT map 707, the value of the element that is positionally corresponding to element 706 is the size (height and width) of the dog 703.

[0025] Furthermore, if an element in a map has multiple values, for example, two values ​​such as height and width, the map may be composed of a map of elements with height values ​​and a map of elements with width values.

[0026] For example, the GT acquisition unit 202 acquires the GT map 709 shown in Figure 7(d) as a GT representing the category of person 702 included in the input image 701 acquired by the image acquisition unit 201. Each element in the GT map 709 (each rectangle in Figure 7(d)) ​​corresponds to each pixel or each image region in the input image 701. In the GT map 709, the value of an element belonging to a region 710 within a frame 708 corresponding to the size of person 702 is "1", and the value of an element not belonging to region 710 is "0".

[0027] For example, the GT acquisition unit 202 acquires the GT map 711 shown in Figure 7(e) as a GT representing the category of dogs 703 included in the input image 701 acquired by the image acquisition unit 201. Each element in the GT map 711 (each rectangle in Figure 7(e)) corresponds to each pixel or each image region in the input image 701. In the GT map 711, the value of elements belonging to region 712 within a frame corresponding to the size of the dog 703 is "1", and the value of elements not belonging to region 712 is "0".

[0028] In step S303, the inference unit 203 inputs the input image 701 into the neural network and performs calculations on the neural network. As a result, the inference unit 203 obtains inference maps as examples shown in Figures 7(f), (g), (h), (i), (j), and (k) as inference results related to the object to be detected. Each element in these inference maps (each rectangle in Figures 7(f), (g), (h), (i), (j), and (k)) corresponds to each pixel or each image region in the input image 701.

[0029] An example of the neural network configuration used by the inference unit 203 is explained using Figure 4. The neural network in Figure 4 is a hierarchical neural network having four layers: Layer 1 (401), Layer 2 (402), Layer 3 (403), and Layer 4 (404). Each layer has convolution, batch-normalization, and rectified linear unit (ReLU) operations, as shown in Figure 5. The configuration in Figure 5 is merely an example; Leaky ReLU or the Sigmoid function may be used instead of ReLU, or a combination of MaxPooling and AveragePooling may be used.

[0030] Map 713 in Figure 7(f) is an inference map obtained as the output of the neural network for the input image 701. The value of each element in map 713 (inference value) indicates the likelihood that the element is the central position of the object. In map 713, as the neural network learns, the value of the element corresponding to the central position of the object approaches 1, while the values ​​of the other elements approach 0.

[0031] The map 715 in Figure 7(g) is an inference map obtained as the output of the neural network for the input image 701, and the value of each element in map 715 is the inferred size of the object inferred for that element. Figure 7(g) shows the size frame indicated by the value (inferred size) of each element. The inferred value for each element changes according to the size of the object corresponding to that element.

[0032] Map 717 in Figure 7(h) is an inference map obtained as the output of the neural network for the input image 701, and the value of each element in map 717 is such that the element is "category is " person This indicates the likelihood that an element belongs to the domain of an object.

[0033] Map 718 in Figure 7(i) is an inference map obtained as the output of the neural network for the input image 701, and the value of each element in map 718 indicates the likelihood that the element is "an element belonging to the region of objects whose category is "dog".

[0034] Map 719 in Figure 7(j) is an inference map (weighted map) obtained as the output of a neural network to which the input image 701 is input. The value of each element in map 719 is the weight for the size of the object corresponding to that element.

[0035] Map 720 in Figure 7(k) is an inference map (weighted map) obtained as the output of a neural network to which the input image 701 is input. The value of each element in map 720 is the weight for the category of the object corresponding to that element.

[0036] In step S304, the integration unit 204 performs ensemble integration based on maps 715, 717, 718, 719, and 720. An example of the functional configuration of the integration unit 204 is shown in Figure 6(a). Details of the processing in step S304 will be explained according to the flowchart in Figure 6(b). The following explanation will focus on processing related to people, but the same process is applied to dogs. In step S602, the weighting unit 601 calculates the element product of map 715 and map 719 according to the following (Equation 1) to obtain map S'.

[0037] S'=S·A S … (Formula 1) Here, S represents map 715, and A Srepresents map 719, and "·" represents element-wise product. In step S603, the average processing unit 602 performs spatial averaging processing for averaging each region in map S' in the spatial direction according to the following (Equation 2).

[0038] S A =SpatialAverage(S’) … (Equation 2) Here, SpatialAverage(X) is a function that averages, for each element in map X, an element group including said element and its surrounding elements, and SpatialAverage(S') is defined by the following (Equation 2-1).

[0039]

Mathematical Expression

[0040] … (Equation 2-1) Here, S A i represents the value of the i-th element in map S A , and S' j represents the value of the j-th element in map S'. In addition, "j∈Peripheral i " represents the index j of N elements including the i-th element and its surrounding elements. In other words, S A i is the average (spatial average) value of the values of N elements including the i-th element and its surrounding elements in map S'. N is the number of elements for which spatial average is calculated, and is determined empirically in advance. In step S604, the weighting unit 601 calculates the element-wise product of map 717 and map 720 according to the following (Equation 3) to obtain map O'.

[0041] O’=O·AO … (Equation 3) Here, O represents map 717 and AO represents map 720. In step S 605 , the average processing unit 602 performs spatial averaging processing for averaging map O' in the spatial direction according to the following (Equation 4).

[0042] O A =SpatialAverage(O') … (Formula 4) As a result, O A i (Map O) A The value of the i-th element in map O' is the average (spatial average) of the values ​​of the N elements in map O' that include the i-th element and its surrounding elements.

[0043] The integration unit 204 then outputs map 713 as the final inference result of the central position of the object in the input image, and map S A This outputs the final inference result of the size of the object in the input image, and maps O A This outputs the final inference result for the object category in the input image.

[0044] Returning to Figure 3, in step S305, the loss calculation unit 205 calculates the loss based on the difference between the inference result output by the integration unit 204 and the GT map acquired in step S302. In this embodiment, the loss with the corresponding GT map is calculated for each of the object's position, object's size, and object's category.

[0045] First, let's explain the process of calculating the loss for the object's position. In this embodiment, the neural network learns the task of object detection. Therefore, it is necessary for the neural network to learn so that it can correctly detect the object's position.

[0046] Therefore, the loss calculation unit 205 trains the neural network so that the map 713, which is the output of the neural network for the input image 701, approaches the corresponding GT map 704. If the output of Layer 4 (404) (map 713) of the neural network that takes the input image 701 as input is Cinf and the GT map 704 is Cgt, then the loss for the position of the object is Loss C This can be calculated according to the following (Equation 5).

[0047]

number

[0048] … (Formula 5) Σ represents calculating the sum over all elements of Cinf(Cgt), and N represents the number of elements in Cinf(Cgt). In other words, the loss calculation unit 205 calculates "the squared difference between the values ​​of the elements in the pair" for each pair of elements that are positionally corresponding in Cinf and Cgt, and the average of the squared values ​​obtained for all pairs is used as the Loss. C This is how it is calculated. When the value of the neural network's output, Cinf, deviates from the value of the GT map 704, Cgt, the loss increases, and as the values ​​of Cinf and Cgt approach each other, the loss decreases. Since learning proceeds in the direction of reducing the loss, as learning progresses, the output of the neural network approaches Cgt.

[0049] For example, suppose in map 713, the element corresponding to the center position of person 702 shows a high value. In this case, the element corresponding to the center position of person 702 in the corresponding GT map 704 705 Since it shows a high value, it indicates that GT and the inference result are close in terms of the central position of person 702, and in this case, the loss LossC will be small.

[0050] On the other hand, suppose that in map 713, the element corresponding to the center position of dog 703 shows a low value. In this case, since element 706, which corresponds to the center position of dog 703 in the corresponding GT map 704, shows a high value, it indicates that there is a discrepancy between the GT and the inference result regarding the center position of dog 703, and in this case, the loss C It will get bigger.

[0051] Furthermore, the formula for calculating the loss based on the difference between the inferred position of the object's center and the GT is not limited to the above-mentioned sum of squared errors formula; for example, the loss can also be calculated using a loss function such as Cross-Entropy.

[0052] Next, we will explain the process of calculating the loss with respect to the size of the object. The loss calculation unit 205 calculates the map S described above. A The loss is based on the difference between the corresponding GT map Sgt and the result. S This is calculated according to (Equation 6) below.

[0053]

number

[0054] … (Formula 6) Σ is S A This represents finding the sum over all elements of (Sgt), where N is S A This represents the number of elements in (Sgt). δ() is a function that returns 1 for elements in the map where an object exists, and 0 for elements where no object exists.

[0055] Map S A The closer the value of GT map Sgt is to that, the greater the loss. S As the value of decreases, as learning progresses, map S A This will bring it closer to the GT map Sgt, allowing for more accurate inference of object sizes.

[0056] Note: Map S A The formula for calculating the loss based on the difference between the GT map Sgt is not limited to the above formula of the sum of squared errors; for example, formulas using L1 error or Smooth L1 error may also be used.

[0057] Thus, Map S A By comparing it with the GT map Sgt, it becomes possible to perform training while considering ensemble integration. As a result, the accuracy of object size inference with ensemble integration is improved.

[0058] Next, we will explain the process of calculating the loss for the object category. The loss calculation unit 205 calculates the map O described above. AThe loss is based on the difference between the corresponding GT map Ogt and the result. O This is calculated according to (Equation 7) below.

[0059]

number

[0060] … (Formula 7) Σ is O A This represents finding the sum over all elements of (Ogt), where N is O A This represents the number of elements in (Ogt). δ() is a function that returns 1 for elements in the map where an object exists, and 0 for elements where an object does not exist.

[0061] Then the loss calculation unit 205 calculates the three losses as shown in (Equation 8) below. C Loss S Loss O The linear sum of these is calculated as the loss in the neural network.

[0062] Loss = α Loss C +βLoss S +γLoss O … (Formula 8) Here, α, β, and γ are hyperparameters used during training and are determined empirically.

[0063] In step S306, the update unit 206 trains the neural network by updating its parameters to reduce the loss obtained in step S305. The neural network parameters consist of two types: weights for elements such as convolutions and weights related to the neural network architecture, and both are updated. The parameter updates are performed using backpropagation, such as Momentum SGD.

[0064] While the above explanation focused on calculating the loss for a single image, actual training involves calculating the loss for multiple different images. The neural network parameters are then updated so that the losses calculated for all of these images fall below a threshold.

[0065] In step S308, the update unit 206 determines whether the learning termination condition has been met. Here, it is determined that the "learning termination condition has been met" if the Loss becomes smaller than the threshold. Note that the learning termination condition is not limited to a specific condition. For example, it may be determined that the "learning termination condition has been met" if the difference between the previous Loss and the current Loss (the amount of change in Loss) becomes less than the threshold. Alternatively, it may be determined that the "learning termination condition has been met" if the number of learning iterations (the number of repetitions of steps S301 to S306) exceeds the threshold. Furthermore, two or more conditions may be combined, and it may be determined that the "learning termination condition has been met" if all or any one of the conditions is met.

[0066] As a result of this judgment, if the learning termination condition is met, the process proceeds to step S309; ​​otherwise, the process proceeds to step S303.

[0067] In step S309, the update unit 206 stores the neural network parameters learned through the above process in the storage unit 207. The storage unit 207 may be the same as the storage unit 104, another storage unit of the information processing device 1, or an external device that is communicatively connected to the information processing device 1 via the I / F 105.

[0068] Next, we will describe an information processing device 2 that uses a neural network trained by information processing device 1 to infer the attributes (position, size, category) of objects in an input image. Information processing device 2 may be the same device as information processing device 1, or it may be a separate device from information processing device 1.

[0069] The hardware configuration of the information processing device 2 may be the same as that of the information processing device 1, as shown in Figure 1, or it may be a different configuration. When the configuration of Figure 1 is applied to the information processing device 2, the CPU 101 executes various processes using computer programs and data stored in the ROM 102 and RAM 103. In this way, the CPU 101 controls the operation of the entire information processing device 2 and executes or controls the various processes that the information processing device 2 performs. The ROM 102 stores setting data for the information processing device 2, computer programs and data related to the startup of the information processing device 2, computer programs and data related to the basic operation of the information processing device 2, etc. The memory unit 104 stores the OS (operating system), computer programs and data for causing the CPU 101 to execute or control the various processes that the information processing device 2 performs, etc. Other aspects are the same as those of the information processing device 1.

[0070] Figure 8 shows an example of the functional configuration of the information processing device 2. In this embodiment, each functional unit in Figure 8 is assumed to be implemented by software (computer program). In the following, the functional units in Figure 8 may be described as the main processing units, but in reality, the functions of the functional units are realized by the CPU 101 executing the computer program corresponding to the functional unit. Note that the functional units shown in Figure 8 may also be implemented by hardware. The "inference processing (object detection task) for inferring the attributes of objects in the input image" performed in such an information processing device 2 will be explained according to the flowchart in Figure 9.

[0071] In step S901, the image acquisition unit 801 acquires an input image containing the object to be detected. Similar to step S301 above, the object to be detected here is a person and a dog, but it is not limited to these. Also, the method of acquiring the input image is not limited to a specific method. To provide a concrete explanation, we will describe the case in step S901 where the image acquisition unit 801 acquires the input image 701.

[0072] In step S902, the inference unit 802 first reads the neural network parameters stored in the memory unit 207. Then, the inference unit 802 constructs a neural network (a neural network trained by processing according to the flowchart in Figure 3) based on the read parameters. The inference unit 802 then inputs the input image 701 into the constructed neural network and performs calculations on the neural network. As a result, the inference unit 802 obtains maps as the output of the neural network (inference results related to the object to be detected), as exemplified in Figures 7(f), (g), (h), (i), (j), and (k), similar to step S303 above. The following describes the processing related to people, but the same process is performed for dogs.

[0073] In step S903, the integration unit 803 uses the maps 715 and 719 acquired in step S902 to perform processing according to the above (Equation 1), (Equation 2), and (Equation 2-1), thereby processing map S A This is obtained as the inference result for the size of the object. The integration unit 803 also uses the maps 717 and 720 obtained in step S902 to perform processing according to the above equations (3) and (4), thereby obtaining map O A This is obtained as the inference result for the object category. In other words, the image acquisition unit 801, inference unit 802, and integration unit 803 shown in Figure 8 operate in the same way as the image acquisition unit 201, inference unit 203, and integration unit 204 shown in Figure 2, respectively.

[0074] In step S904, the post-processing unit 804 identifies elements in the map 713 that have a value greater than or equal to a threshold as target elements, and maps S A In this process, the value of the element that is positionally corresponding to the target element (size of the object) is identified.

[0075] In step S905, the post-processing unit 804 performs Non-Maximum Suppression (NMS) processing on each of the frame (object detection frame) with the object size identified in step S904, and acquires one frame as the object detection frame. For example, map 715 is map S A Let's assume this is the case. In this instance, Non-Maximum Suppression (NMS) processing is performed on multiple frames 716 for person 702 in map 715, and one frame is obtained as the detection frame for person 702. Similar processing is performed on multiple frames for dog 703 to obtain the detection frame for dog 703.

[0076] Furthermore, the post-processing unit 804 generates a map O for the person. A Map O obtained for the dog, which is the element that is spatially corresponding to the target element. A The system identifies the element with the largest value among the elements that are positionally corresponding to the target element. The post-processing unit 804 then obtains the category corresponding to the map to which the identified element belongs as the object detection category.

[0077] The post-processing unit 804 then outputs the object's detection position (center position of the detection frame, position of any of the four corners of the detection frame, etc.), the size of the object's detection frame (height and width), and the object's detection category as estimated attributes for the object. The output destination of the object's attributes is not limited to a specific destination; for example, they may be displayed on a screen (not shown) of the information processing device 2, or transmitted to an external device via the I / F 105. The information processing device 2 may also perform further processing using the object's attributes.

[0078] Thus, according to this embodiment, by learning the weights for the inference results of object attributes as a two-dimensional feature map, it becomes possible to perform a stable ensemble regardless of the number of detected candidates.

[0079] <Modification 1 of the First Embodiment> In the following modifications and embodiments, including this modification, the differences from the first embodiment will be explained, and unless otherwise specified below, they will be the same as the first embodiment. In this modification, in addition to the maps exemplified in Figures 7(f), (g), and (j), map 1406 exemplified in Figure 14(b) and a corresponding weighting map are also obtained. Map 1406 shows the size of an object, similar to the map in Figure 7(g). However, the values ​​of each element in map 1406 do not represent the width and height of the frame as in Figure 7(g), but rather the distances 1402, 1403, 1404, and 1405 from the element to the top, bottom, left, and right edges of the object's region, respectively, as shown in Figure 14(a). Map 1401 in Figure 14(a) is the GT map corresponding to map 1406. Each element (each rectangle) in the maps in Figures 14(a) and (b) corresponds to each pixel or each image region in the input image.

[0080] In this embodiment, in step S302, the inference unit 203 obtains, from the neural network that has received the input image 701, the map 1406 exemplified in Figure 14(b) and the corresponding weight map, in addition to the maps exemplified in Figures 7(f), (g), and (j). Then, in step S304, the integration unit 204 calculates the element-wise product of the map of the object's size (width, height) and the weight map corresponding to that map as follows.

[0081] W1 = W·A W1 H1 = H·A H1 Here, W is a map in which only the width value (one of the attribute elements "height" and "width") that make up the attribute "size" is used from the values ​​of each element (height, width) of the object's size map. Also, A W1 This is weighted map A S This map uses only the weight for width in the values ​​of each element (weights for height and width). Furthermore, H is a map that uses only the value for height (one of the attribute elements "height" and "width" that make up the attribute "size") in the values ​​of each element (height, width) of the object size map. Also, A H1 This is weighted map A SThis map uses only the weight for height in the values ​​of each element (weights for height and width).

[0082] Furthermore, the integration unit 204 calculates the element product of a map showing the distance from each element to the top, bottom, left, and right edges of the object (such as the map 1406 mentioned above, hereinafter referred to as the top, bottom, left, and right edge map) and a weighting map corresponding to that map, as follows.

[0083] W2 = (L + R) · A W2 H2 = (T + B) · A H2 Here, L is a map in which only the "distance from the element to the left edge" value is used for each element in the top, bottom, left, and right edge map (distance from the element to the top, bottom, left, and right edges, respectively). R is a map in which only the "distance from the element to the right edge" value is used for each element in the top, bottom, left, and right edge map, and T is a map in which only the "distance from the element to the top edge" value is used for each element in the top, bottom, left, and right edge map. Also, B is a map in which only the "distance from the element to the bottom edge" value is used for each element in the top, bottom, left, and right edge map. Also, A W2 This is a weighted map for the map obtained by the element sum of map L and map R, and A H2 This is a weighted map for the map obtained by the elemental sum of map T and map B. The integration unit 204 then defines the map S' of object sizes as follows.

[0084] S' = ((W1 + W2) / 2, (H1 + H2) / 2) This means that each element of map S' has the value of each element in the map obtained by (W1+W2) / 2 and the value of each element in the map obtained by (H1+H2) / 2.

[0085] Thus, a map of object attributes in an input image is constructed based on a map obtained by the element-wise product of a channel-specific map of object attributes in the input image and a map representing the weights of those attributes for each region in the input image.

[0086] Then, from such a map S', map S is created in the same manner as in the first embodiment. A We obtain the map O in the same manner as in the first embodiment. A This is determined. The process then proceeds in the same manner as in the first embodiment. The inference unit 802 and the integration unit 803 perform the same processing as the inference unit 203 and the integration unit 204 in steps S902 and S903, respectively.

[0087] Thus, in this modified example, not only is the spatial ensemble described in the first embodiment performed, but multiple inference results of different qualities are also ensembled in the channel direction. By ensembling using feature vectors of two dimensions or more, not only in the spatial direction but also in the channel direction, it is possible to further improve the accuracy of ensemble integration.

[0088] <Modification 2 of the first embodiment> In the first embodiment, the elemental product of the maps was calculated using (Equation 1) and (Equation 3), and the resulting map was obtained by performing spatial averaging on the map obtained from the elemental product using (Equation 2) and (Equation 4) as the result of ensemble integration for the map of object attributes. However, the method for obtaining the ensemble integration result is not limited to the method described in the first embodiment. For example, taking the map S of the size of an object as an example, the map S' may be obtained using the following (Equation 9) instead of (Equation 1).

[0089] S'=δ·S·A S … (Formula 9) Here, δ is a binary map where the element values ​​are either 1 or 0. Map A S If the value of the element of interest in map A is greater than or equal to the threshold, then the value of the element in map δ corresponding to that element of interest will be 1. S If the value of the element of interest in is less than the threshold, then the value of the element in map δ corresponding to that element of interest is 0. In other words, (Equation 9) is the weighted map A SThis represents the element-wise product of the binarized map obtained by binarizing the original value and map S. Alternatively, as shown in (Equation 10) below, the weighted map may be transformed nonlinearly and then used again as the weighted map to obtain map S'.

[0090]

number

[0091] … (Equation 10) Here, "i∈Peripheral" represents the index of a specified number of elements, including the i-th element and its surrounding elements. Also, instead of the spatial average in (Equation 2-1), the function in (Equation 11) below is used for S A i You may ask for it.

[0092] S A i =max(S' i ) … (Formula 11) Here, max(S' i ) is a function that returns the maximum value among the value of the i-th element in map S' and the values ​​of each element around that i-th element. Alternatively, instead of the spatial average in (Equation 2-1), use the following function (Equation 12) to calculate S A i You may ask for it.

[0093] S A i =median(S' i ) … (Equation 12) Here, median(S' i This function returns the value of the i-th element in map S' and the median of the values ​​of each element surrounding that i-th element.

[0094] [Second Embodiment] In this embodiment, we will first describe the learning of the neural network architecture for performing an object tracking task that detects a specific target from an input image. The processing performed by the information processing device 1 to perform such learning will be explained according to the flowchart in Figure 10.

[0095] In step S1001, the image acquisition unit 201 acquires an image containing the tracking target as a template image. The GT acquisition unit 202 acquires attributes such as the position and size of the tracking target within the template image as GT. An example of a template image is shown in Figure 11(a). The template image 1101 includes the tracking target 1103, and the GT is, for example, the position and size of the rectangular region 1104 that includes the tracking target 1103.

[0096] In step S1002, the image acquisition unit 201 extracts a partial image containing the tracking target from the template image based on the position and size indicated by the GT, and resizes the extracted partial image to an appropriate size (for example, the size of the template image). In the example in Figure 11(a), the image within the rectangular region 1102, which is obtained by a constant factor enlarging the rectangular region 1104 having the position and size indicated by the GT in the template image 1101, is extracted as a partial image.

[0097] In step S1004, the image acquisition unit 201 acquires a search image that is different from the image acquired in step S1001 and includes the tracking target. For example, the image acquisition unit 201 acquires a search image from a different frame in the image sequence to which the image acquired in step S1001 belongs. An example of a search image is shown in Figure 11(b). The search image 1105 includes the tracking target 1103.

[0098] In step S1005, the image acquisition unit 201 extracts a partial image from the search image based on the position and size indicated by the GT, and resizes the extracted partial image to an appropriate size (for example, the size of the search image). In the example in Figure 11(b), the image within the rectangular region 1106, which is obtained by a constant factor enlarging the rectangular region having the position and size indicated by the GT in the search image 1105, is extracted as a partial image.

[0099] In step S1003, the inference unit 203 inputs the resized partial image (first partial image) from step S1002 into the neural network and performs computational processing on the neural network to obtain the features of the target to be tracked in the first partial image.

[0100] In step S1006, the inference unit 203 performs the steps S 1005 The resized partial image (second partial image) is input into a neural network, and the neural network performs calculations to obtain features from the second partial image.

[0101] In step S1007, the inference unit 203 calculates the cross-correlation between the features acquired in step S1003 and the features acquired in step S1006, and identifies (infers) the location and size of regions in the search image that are similar to (highly correlated with) the features of the target being tracked. The inference unit 203 then generates an inference location map and an inference size map, similar to those in the first embodiment.

[0102] In the search image, elements in the "inference position map" corresponding to the locations of regions with features similar to (highly correlated with) the features of the tracked target hold a value representing the likelihood of the tracked target's central position, for example, the correlation value obtained for that feature. Similarly, elements in the "inference size map" corresponding to the locations of regions with features similar to (highly correlated with) the features of the tracked target in the search image hold the size of that region.

[0103] Figure 12 shows an example of a map based on the cross-correlation obtained in step S1007. Figure 12(a) shows map 1201, which indicates the center position of the tracked target inferred in the search image. Element 1202 in map 1201 corresponds to the center position of the tracked target, and learning proceeds so that the inferred center position becomes element 1202.

[0104] Figure 12(b) shows map 1204, which indicates the size of the tracked object inferred in the search image. Map 1207 in Figure 12(c) is a weighted map for map 1204 and, as in the first embodiment, is obtained as output from the neural network. The value of each element in map 1207 is the weight for the size of the object corresponding to that element.

[0105] Then, similar to the first embodiment, the neural network is trained by updating the neural network parameters through the processes in steps S304 to S306. This training process is repeated, and when the training completion condition is met, the process proceeds to step S309 via step S308, and the neural network parameters are stored in the memory unit 207.

[0106] Next, the processing performed by the information processing device 2 to detect a specific target from the input image in an object tracking task will be explained according to the flowchart in Figure 13. In Figure 13, processing steps similar to those shown in Figure 10 are given the same step numbers.

[0107] In step S1001, the image acquisition unit 801 acquires an image containing the tracking target as a template image. In step S1004, the image acquisition unit 801 acquires a different image from the one acquired in step S1001, which also contains the tracking target, as a search image.

[0108] In step S1003, the inference unit 802 inputs the template image into the neural network and performs computational processing on the neural network to obtain the features of the target to be tracked in the template image.

[0109] In step S1006, the inference unit 802 inputs the search image into the neural network and performs computational processing on the neural network to acquire features in the search image.

[0110] In step S1007, the inference unit 802 calculates the cross-correlation between the features acquired in step S1003 and the features acquired in step S1006, and identifies (infers) the location and size of regions in the search image that are similar to (highly correlated with) the features of the target being tracked. The inference unit 203 then generates an inference location map and an inference size map, similar to those in the first embodiment.

[0111] In step S304, the integration unit 803 performs processing using the inference size map in the same manner as in the first embodiment to obtain a map which is the inference result of the size of the target to be tracked in the search image.

[0112] In step S1301, the post-processing unit 804 uses the inferred map of the tracked object's position and the size map obtained in step S304 to perform the same processing as in steps S904 and S905 above to obtain the object's attributes.

[0113] Thus, in this embodiment, as shown in Figure 12(c), learning the ensemble integrated weights results in higher ensemble integrated weight values ​​around the tracked target, while lower ensemble integrated weight values ​​around the untracked target. This is because the object size obtained around the untracked target differs from that of the tracked target, and if the object size of the untracked target is included in the ensemble, the error in the inferred size of the tracked target will increase. As a result, the ensemble weight around the untracked target becomes lower. Therefore, even when there are untracked targets around the tracked target, or when part of the tracked target is hidden by untracked targets, stable ensemble building becomes easier, improving the accuracy of size estimation for the tracked target.

[0114] In the above embodiments and modifications, the operation of the information processing device 1 and the information processing device 2 has been explained with specific examples, but the operation of these devices is not limited to the above examples. For example, the operation of the information processing device 1 described above is an example of operation in which the attributes of an object in an input image and the weights related to those attributes in each region of the input image are inferred (first inference), and the attributes of the object are inferred (second inference) based on the attributes and weights inferred in the first inference.

[0115] For example, the operation of the information processing device 2 described above is an example of an operation in which, as the output of a neural network that has an input image as input, the attributes of objects in the input image and the weights related to those attributes in each region of the input image are obtained (first acquisition), and the parameters of the neural network are updated based on the attributes and weights obtained by the first acquisition and the training data.

[0116] Furthermore, the numerical values, processing timings, processing order, processing entity, data (information) destination / source / storage location, etc., used in each of the above embodiments and modifications are given as examples for the purpose of providing a concrete explanation, and are not intended to limit the scope to such examples.

[0117] Furthermore, some or all of the embodiments and modified examples described above may be used in appropriate combinations. Alternatively, some or all of the embodiments and modified examples described above may be used selectively.

[0118] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0119] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0120] 201: Image acquisition unit 202: GT acquisition unit 203: Inference unit 204: Integration unit 205: Loss calculation unit 206: Update unit 207: Storage unit

Claims

1. A first inference means for inferring the size of an object in an input image and the weights relating to the size of each region in the input image, Based on the size and weight of the object inferred by the first inference means, a second inference means infers the size of the object. Equipped with, The size of the object refers to the size of the rectangular region containing the object. The first inference means is an information processing device characterized by inferring the attributes of an object based on a map obtained by the element product of a map of the attributes of an object in an input image, which is obtained based on the cross-correlation between the features of an object in the input image and the features of an image different from the input image, and a map representing the weights related to the attributes of each region in the input image.

2. A first inference means that infers the attributes of an object in an input image and the weights of those attributes for each region in the input image, A second inference means infers the attributes of the object based on the attributes and weights inferred by the first inference means. Equipped with, The first inference means is an information processing device characterized by inferring the attributes of an object based on a map obtained by the element product of a map of attributes of an object in an input image, which is obtained based on the cross-correlation between the features of an object in the input image and the features of an object in an image different from the input image, and a map representing the weights related to the attributes of each region in the input image.

3. The information processing apparatus according to claim 2, characterized in that the attribute is one of the size of the object, the position of the object, or the category of the object.

4. The information processing apparatus according to claim 2 or 3, characterized in that the first inference means obtains the output of the neural network into which the input image is input as the attribute and the weight.

5. The information processing apparatus according to any one of claims 1 to 4, characterized in that the second inference means infers the attributes of an object based on a map obtained by the element product of a map of the attributes of an object in the input image and a map representing the weights related to the attributes of each region in the input image.

6. The information processing apparatus according to any one of claims 1 to 5, characterized in that the second inference means infers the attributes of an object based on a map obtained by the element product of a channel-by-channel map of the attributes of an object in the input image and a map representing the weights related to the attributes of each region in the input image.

7. The information processing apparatus according to any one of claims 1 to 5, characterized in that the second inference means infers the attributes of an object based on a map obtained by the element product of a map of the attributes of an object in the input image and a binarized map obtained by binarizing a map representing the weights related to the attributes of each region in the input image.

8. The information processing apparatus according to any one of claims 1 to 5, characterized in that the second inference means infers the attributes of an object based on a map obtained by the element product of a map of the attributes of an object in the input image and a map obtained by nonlinearly transforming a map representing the weights related to the attributes of each region in the input image.

9. The information processing apparatus according to any one of claims 1 to 8, characterized in that each region in the input image is an image region containing two or more pixels.

10. The information processing apparatus according to any one of claims 1 to 9, characterized in that the first inference means and the second inference means are learned based on the difference between the attribute inferred by the second inference means based on the attribute and the weight, and the true value of the attribute.

11. The information processing apparatus according to any one of claims 1 to 10, wherein the second inference means infers the attribute based on a map obtained by the element product of a plurality of maps having a common attribute and a map representing the weight related to the attribute for each region in the input image.

12. An information processing method performed by an information processing device, The first inference means of the information processing device performs a first inference step of inferring the size of an object in an input image and the weights relating to the size of each region in the input image. The second inference means of the information processing device performs a second inference step in which it infers the size of the object based on the size and weight of the object inferred in the first inference step. Equipped with, The size of the object refers to the size of the rectangular region containing the object. An information processing method characterized in that, in the first inference step, the attributes of an object are inferred based on a map obtained by the element product of a map of the attributes of an object in the input image, which is obtained based on the cross-correlation between the features of an object in the input image and the features of an image different from the input image, and a map representing the weights related to the attributes of each region in the input image.

13. An information processing method performed by an information processing device, The first inference means of the information processing device performs a first inference step of inferring the attributes of an object in an input image and the weights of each region in the input image relating to those attributes, The second inference means of the information processing device performs a second inference step in which it infers the attributes of the object based on the attributes and weights inferred in the first inference step. Equipped with, In the first inference step, the attributes of the object are inferred based on a map obtained by the element-wise product of a map of the attributes of the object in the input image, which is obtained based on the cross-correlation between the features of the object in the input image and the features of the object in an image different from the input image, and a map representing the weights related to the attributes of each region in the input image. An information processing method characterized by the following:

14. A computer program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 11.

Images