Training device, inference device, training method, and inference method

Abstract

To provide a technique for correctly associating specific parts of the same object with each other regardless of the presence of another object in the vicinity of that object.SOLUTION: A first model for detecting a specific part is used for acquiring a likelihood map of the specific part in an input image. A second model for detecting an object to be tracked is used for acquiring a region map indicating a region of the specific part of the object to be tracked in the input image. The second model is trained on the basis of a loss obtained based on an element product map obtained by element products between the likelihood map and the region map and correct answer data indicating the region of the specific part of object to be tracked in the input image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to tracking technology. [Background technology]

[0002] There are technologies for continuously detecting and tracking a specific subject from each frame of a video image, such as those that use brightness and color information, those that use template matching, those that use Convolutional Neural Network (CNN), etc. These technologies are used in surveillance cameras equipped with a function for tracking subjects, and cameras equipped with an autofocus function that automatically focuses on a subject.

[0003] There is also known a technique for detecting a specific part of a subject, which may be used, for example, to pinpoint the eye of a person or animal as a subject.

[0004] Such techniques include a top-down approach that detects the entire area of ​​the subject and detects specific parts within that area, and a bottom-up approach that detects multiple specific parts and determines the specific parts of the subject by matching them. In the bottom-up approach, when matching specific parts with each other, they may be mistakenly matched with specific parts of an object other than the subject. In Patent Document 1, the correspondence of each specific part is output by estimating an affinity map that indicates the correspondence between specific parts. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Special table 2022-510417 [Non-patent literature]

[0006] [Non-Patent Document 1] SingleShotMultiboxDetector(ECCV2016) [Non-Patent Document 2] Real-TimeMDNet (ECCV2018) Summary of the Invention [Problem to be solved by the invention]

[0007] When matching specific parts of the same object from among separately detected specific parts, they may be mistakenly matched with parts of another object that exists nearby. For example, when a person in the foreground and a person in the background are close to each other on an image, the eyes of the person in the foreground may be mistakenly matched with the eyes of the person in the background. The present invention provides a technology for correctly matching specific parts of the same object even when another object exists nearby. [Means for solving the problem]

[0008] One aspect of the present invention is characterized in that it comprises a first acquisition means for acquiring a likelihood map of a specific part in an input image using a first model for detecting the specific part, a second acquisition means for acquiring a region map representing the region of the specific part of the tracked target in the input image using a second model for detecting a tracked target, and a learning means for learning the second model based on losses obtained based on an element product map obtained by element multiplying the likelihood map and the region map, and ground truth data indicating the region of the specific part of the tracked target in the input image. Effect of the Invention

[0009] According to the present invention, even if another object exists nearby, specific parts of the same object can be correctly associated with each other. [Brief description of the drawings]

[0010] [Figure 1] FIG. 2 is a block diagram showing an example of the functional configuration of the learning device 100. [Diagram 2]4 is a flowchart of a model learning process performed by the learning device 100. [Diagram 3] FIG. 4 is a schematic diagram of the processes in steps S101 to S103. [Figure 4] FIG. 11 is a diagram showing an example of a process for determining an average vector. [Diagram 5] FIG. 2 is a block diagram showing an example of the functional configuration of the inference device 200. [Figure 6] 4 is a flowchart of processing performed by the inference device 200. [Figure 7] FIG. 7 is a block diagram showing an example of the functional configuration of a learning device 700. [Figure 8] 4 is a flowchart of a learning process performed by the learning device 700. [Figure 9] FIG. 1 is a block diagram showing an example of the hardware configuration of a computer apparatus. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] [First embodiment] In this embodiment, a learning device that learns a model for detecting a specific part of a tracking target in an input image is described. More specifically, in this embodiment, a first model for detecting a specific part is used to obtain a likelihood map of the specific part in an input image, a second model for detecting a tracking target is used to obtain a region map representing the region of the specific part of the tracking target in the input image, and an example of a learning device that learns the second model based on a loss obtained based on an element product map obtained by an element product of the likelihood map and the region map and ground truth data indicating the region of the specific part of the tracking target in the input image is described.

[0013] An example of the functional configuration of a learning device 100 according to this embodiment is shown in the block diagram of Fig. 1. The model learning process performed by the learning device 100 according to this embodiment will be described with reference to the flowchart of Fig. 2.

[0014] The learning device 100 according to this embodiment holds a learning dataset used in the learning process. The learning dataset includes a set of consecutive images (a consecutive image set) and correct answer data corresponding to the images.

[0015] Each image in the set of consecutive images contains one or more objects including a tracking subject, which is a subject to be tracked. The set of consecutive images is, for example, a video sequence, and in this case, each frame in the video sequence corresponds to the image.

[0016] The correct answer data includes first area information that defines an image area of ​​a tracking subject included in an image corresponding to the correct answer data, and second area information that defines an image area of ​​a specific portion of an object included in the image.

[0017] In this embodiment, a case will be described in which the first region information is information indicating the center coordinates of a rectangular region surrounding the image region of the tracking subject, and the width and height of the rectangular region. Also, in this embodiment, a case will be described in which the second region information is information indicating the center coordinates of a rectangular region surrounding the image region of the specific part, and the width and height of the rectangular region. Also, in this embodiment, a case will be described in which the tracking subject is a person, and the specific part is a head. Note that the specific parts may be any part of an object, and the number of the specific parts is not limited to a specific number.

[0018] In step S101, the acquisition unit 110 acquires an unselected image and the correct answer data corresponding to the image from the learning dataset. For example, when the set of consecutive images is a video, the acquisition unit 110 acquires the images in order from the first frame, and acquires the correct answer data corresponding to the image.

[0019] In step S102, the detection unit 120 inputs the image acquired in step S101 into a specific part detection model and performs a calculation process of the specific part detection model. Here, the specific part detection model is "a model that has been trained to output a map (specific part likelihood map) indicating the likelihood corresponding to the pixel position of the input image (the likelihood that the pixel position is included in a specific part)." A learning model such as CNN or Transformer can be applied to the specific part detection model, and for example, the learning model disclosed in Non-Patent Document 1 can be applied. Through such calculation process, the detection unit 120 acquires the "specific part likelihood map corresponding to the image acquired in step S101," which is the output of the specific part detection model.

[0020] In addition, the detection unit 120 may use a specific part detection model or other trained models to obtain a "size map indicating the size of a specific part included in the image acquired in step S101" or an "offset map representing correction information for correcting the position of a specific part included in the image acquired in step S101".

[0021] In step S103, the detection unit 130 inputs the image acquired in step S101 into the tracking subject detection model and performs a calculation process of the tracking subject detection model. Here, the tracking subject detection model is a "model that has been trained to output a tracking subject likelihood map and a specific part area map." A learning model such as CNN or Transformer can be applied to the tracking subject detection model, and for example, the learning model disclosed in Non-Patent Document 2 can be applied. Here, the tracking subject likelihood map is a "map indicating the likelihood corresponding to the pixel position of the input image (the likelihood that the pixel position is included in the tracking subject)," and the specific part area map is a map indicating an area in the image where a specific part of the tracking subject may exist. Through such a calculation process, the detection unit 130 acquires "the tracking subject likelihood map and the specific part area map corresponding to the image acquired in step S101," which are the output of the tracking subject detection model.

[0022] Note that the detection unit 130 may use the tracked subject detection model or other trained models to acquire a "size map indicating the size of the tracked subject included in the image acquired in step S101" or an "offset map indicating correction information for correcting the position of the tracked subject included in the image acquired in step S101", etc. The vertical and horizontal sizes of these maps are the same as the vertical and horizontal sizes of the specific part likelihood map.

[0023] In step S104, the post-processing unit 140 generates a map obtained by calculating an element product of the specific part likelihood map acquired in step S102 and the specific part region map acquired in step S103, as an element product map.

[0024] A schematic diagram of the processing of steps S101 to S103 is shown in Fig. 3. The image acquired by acquisition unit 110 is input to a specific part detection model and a tracking subject detection model. A specific part likelihood map is output from the specific part detection model through arithmetic processing of the specific part detection model by detection unit 120. On the other hand, a tracking subject likelihood map and a specific part region map are output from the tracking subject detection model through arithmetic processing of the tracking subject detection model by detection unit 130. Then, a map obtained by calculating the element product of the specific part likelihood map and the specific part region map is generated as an element product map by post-processing unit 140.

[0025] In step S105, the calculation unit 150 generates a specific part correct map based on the correct answer data acquired in step S101, and calculates a loss for the specific part likelihood using the specific part correct map and the element product map generated in step S104. Here, the process performed by the calculation unit 150 to generate a specific part correct map based on the correct answer data will be described.

[0026] The specific part correct map is a map in which the second region information contained in the correct data is replaced with a two-dimensional likelihood distribution, and is the correct map for the specific part likelihood map. Here, the two-dimensional likelihood distribution is the two-dimensional standard normal distribution expressed by the following formula (1).

[0027]

number

[0028]

number

[0029] is a vector representing the two-dimensional coordinates (x, y) in the specific part correct map,

[0030]

number

[0031] represents the likelihood in two-dimensional coordinates (x, y).

[0032]

number

[0033] is the average vector in the specific part correct map, and is a vector with μx, the average value in the x-axis direction, as the x-component, and μy, the average value in the y-axis direction, as the y-component. The calculation unit 150 calculates this average vector using the correct data. The process for calculating the average vector will be described with reference to FIG. 4 as an example.

[0034] 4 shows the image acquired in step S101 superimposed on a map of the same size as the specific part likelihood map. In this example, the size of the specific part correct map is 24 pixels (elements) horizontal by 16 pixels (elements) vertical, and the size of the image is 720 pixels horizontal by 480 pixels vertical. The coordinate system is expressed with the pixel position of the upper left corner as the origin (0,0), the x-axis positive in the horizontal right direction, and the y-axis positive in the vertical downward direction. Furthermore, a rectangle 1001 indicates a rectangular area represented by the second area information included in the correct answer data, and a point 1002 indicates the center coordinates of the rectangular area, which are assumed to be (325,40) here.

[0035] At this time, the calculation unit 150 divides the center coordinates (325, 40) by the size ratio between the image and the specific part correct map = 720 / 24 (480 / 16) = 30, as shown below, and converts the result of this division into an integer value by rounding it up or down, etc., and uses the result as the average vector.

[0036] μx:325 / 30=10.83 ... →(rounded off)→11 μy: 40 / 30= 1.33 ... →(rounded)→ 1 Average vector μ=(μx, μy)=(11,1) Next, a process performed by the calculation unit 150 to calculate a loss for a specific part likelihood using the specific part correct map and the element product map will be described. As a loss function, a cross entropy loss shown in the following formula (2) is used.

[0037]

number

[0038] Here, M inf is the element product map, M gt is the specific part correct map, and the sum (Σ) is performed for all elements (number of elements N) of the element product map / specific part correct map.

[0039] In step S106, calculation unit 150 calculates a loss for the position and size of the tracked object based on the map acquired in step S103 and the ground truth data acquired in step S101. The "map acquired in step S103" is any one or all of the tracked object likelihood map, size map, and offset map. How to calculate the loss is publicly known from Non-Patent Document 2 and the like, and therefore will not be described here.

[0040] In step S107, the update unit 160 performs a learning process of the tracking object detection model based on the loss calculated in step S105 and the loss calculated in step S106. For example, the update unit 160 performs the learning process of the tracking object detection model by updating the weight parameters of the tracking object detection model so that the value of the linear sum of the loss calculated in step S105 and the loss calculated in step S106 becomes smaller. The learning process can be performed using, for example, an error backpropagation method.

[0041] The processes of steps S101 to S107 are repeated until a learning process end condition is met. The learning process end condition may be, for example, that the number of times the processes of steps S101 to S107 are repeated is equal to or greater than a threshold, that the value of the linear sum is equal to or less than a threshold, that the difference between the previous value and the current value of the linear sum is equal to or less than a threshold, etc. When the learning process end condition is met, the process according to the flowchart in FIG. 2 ends.

[0042] In this embodiment, one piece of learning data is acquired in one learning process, but multiple pieces of learning data may be acquired at once to perform the learning process. In this case, the processes of steps S102 to S106 are performed for each piece of learning data, and the loss calculated for each piece of learning data is used to perform the process of step S107.

[0043] In this way, in this embodiment, learning is performed so that the element product map obtained by the element product of the specific part likelihood map and the specific part area map approaches the specific part correct map. In other words, it can be said that the process of filtering only the specific part of the tracking subject from the specific parts of each object is learned using the tracking subject detection model. At this time, by using the tracking subject model that has been trained to detect the same tracking subject from images of different frames, it is possible to perform filtering processing that takes into account time-series information that the position of the tracking subject changes continuously between frames. This increases the discriminability between objects compared to when matching specific parts using a model trained with independent images that do not have a time series, prevents erroneous matching with another object, and improves matching accuracy.

[0044] The learning data set may be held by an external device instead of by the learning device 100. In this case, the learning device 100 can access the external device to acquire various data included in the learning data set.

[0045] [Second embodiment] In this embodiment, an inference device is described that detects (infers) the position of a specific part of a tracked subject in an input image using a trained tracked subject detection model generated by the learning device 100 according to the first embodiment.

[0046] An example of the functional configuration of the inference device 200 according to this embodiment is shown in the block diagram of Fig. 5. The process performed by the inference device 200 to detect a specific part of a tracked subject in an input image using the trained tracked subject detection model generated by the learning device 100 will be described with reference to the flowchart of Fig. 6. The process according to the flowchart of Fig. 6 shows the process performed for an image of one frame, and when processing images of multiple frames, the process according to the flowchart of Fig. 6 is performed for each of the images of the multiple frames.

[0047] In step S201, the acquisition unit 210 acquires one frame of an image. The method of acquiring the image is not limited to a specific method, and for example, the image may be acquired from an external device connected to the inference device 200 via a wired / wireless network such as a LAN or the Internet, or an image captured by an imaging device may be acquired.

[0048] In step S202, the detection unit 220 inputs the image acquired in step S201 to a specific part detection model (a model similar to the specific part detection model used by the learning device 100) and performs arithmetic processing of the specific part detection model. Through such arithmetic processing, the detection unit 220 acquires a "specific part likelihood map corresponding to the image acquired in step S201", which is the output of the specific part detection model.

[0049] In addition, the detection unit 220 may use a specific part detection model or other trained models to obtain a "size map indicating the size of a specific part included in the image acquired in step S201" or an "offset map representing correction information for correcting the position of a specific part included in the image acquired in step S201".

[0050] In step S203, detection unit 230 inputs the image acquired in step S201 to a tracking subject detection model (a tracking subject detection model trained in learning device 100) and performs arithmetic processing of the tracking subject detection model. Through such arithmetic processing, detection unit 230 acquires a "specific part area map corresponding to the image acquired in step S201", which is the output of the tracking subject detection model.

[0051] In step S204, the post-processing unit 240 generates a map obtained by calculating an element product of the specific part likelihood map acquired in step S202 and the specific part region map acquired in step S203, as an element product map.

[0052] In step S205, the post-processing unit 240 detects the position of the specific part of the tracking subject based on the element product map generated in step S204. The post-processing unit 240 identifies the coordinates (peak coordinates) (xp, yp) of the element with the maximum element value in the element product map, multiplies the peak coordinates (xp, yp) by the above-mentioned size ratio to convert them into coordinates on the image, and sets the converted coordinates as the position of the specific part of the tracking subject on the image. The size ratio is calculated by the post-processing unit 240 using the method described in the first embodiment.

[0053] Furthermore, when the specific portion detection model outputs a size map and an offset map, the post-processing unit 240 determines the element value of the element at the peak coordinate in each map as the size offset of the specific portion.

[0054] In this embodiment, the tracked subject is matched with a specific part (head), but it is also possible to match specific parts of the tracked subject with each other by outputting multiple specific part likelihood maps and specific part area maps and matching each specific part with the tracked subject.

[0055] [Third embodiment] In this embodiment, a new loss is added to the loss calculated in calculation unit 150. An example of the functional configuration of a learning device 700 according to this embodiment is shown in the block diagram of Fig. 7. In Fig. 7, functional units similar to those shown in Fig. 1 are given the same reference numerals, and descriptions of these functional units will be omitted or will be briefly described.

[0056] The learning process by the learning device 700 according to this embodiment will be described with reference to the flowchart in Fig. 8. In Fig. 8, the same process steps as those shown in Fig. 2 are given the same step numbers, and the description of these process steps will be omitted or will be briefly described.

[0057] In step S108, the detection unit 130 stores the specific part region map in the storage unit 170. In step S109, the calculation unit 150 judges whether the number of times that the process according to the flowchart in Fig. 8 has been executed is n or more (n or more iterations have passed). If the result of this judgment is that the number of times that the process according to the flowchart in Fig. 8 has been executed is n or more, the process proceeds to step S110, and if the number of times that the process according to the flowchart in Fig. 8 has been executed is less than n, the process proceeds to step S106.

[0058] In step S110, the calculation unit 150 calculates the loss for the specific part area based on the specific part area map acquired in the current step S103 and the “specific part area map acquired in the previous step S103” stored in the storage unit 170.

[0059] The calculation unit 150 first selects a plurality of specific part region maps as selected specific part region maps from the specific part region maps stored in the storage unit 170. For example, the calculation unit 150 selects as the selected specific part region maps the specific part region map stored in the storage unit 170 in step S108 n times ago (n is a natural number of 2 or more) to the specific part region map stored in the storage unit 170 in the previous step S108. The calculation unit 150 then generates a map obtained by averaging (moving average) these selected n selected specific part region maps as the correct specific part region map.

[0060] The calculation unit 150 calculates the loss using the cross entropy loss shown in the above formula (2) as the loss function. inf is the specific part region map obtained in step S103, M gt is the correct specific part region map.

[0061] In step S107 according to this embodiment, when the loss is calculated in step S110, the update unit 160 performs a learning process of the tracking subject detection model based on the loss calculated in step S105, the loss calculated in step S106, and the loss calculated in step S110. On the other hand, when step S110 is skipped, the update unit 160 performs a learning process of the tracking subject detection model based on the loss calculated in step S105 and the loss calculated in step S106, as in the first embodiment.

[0062] In this embodiment, learning is performed so that the difference between the current specific part area map and the past specific part area map is small. In other words, learning is performed so that the output of the specific part area map has consistency in time series. This suppresses the specific part existence area from changing significantly between frames due to a false reaction to an object other than the tracking subject, and prevents erroneous association with another object. In addition, by using a map calculated from a plurality of past specific part area maps as a correct answer value, when the tracking subject moves significantly in the continuous still images, the distribution of the calculated correct specific part area map has a broad base. This allows learning to be performed so that the distribution of the output specific part area map is allowed to spread, and the area where the specific part may exist can be appropriately estimated even for objects with large movements such as athletes.

[0063] [Fourth embodiment] 1, 5, and 7 may be implemented in hardware, and the other functional units except for storage unit 170 may be implemented in software (computer program). In the latter case, a computer device capable of executing the computer program is applicable to learning device 100, inference device 200, and learning device 700. An example of the hardware configuration of a computer device applicable to learning device 100, inference device 200, and learning device 700 will be described using the block diagram in FIG.

[0064] CPU 901 executes various processes using computer programs and data stored in RAM 902. As a result, CPU 901 controls the operation of the entire computer device, and executes or controls the various processes described as processes performed by devices to which the computer device is applied (learning device 100, inference device 200, learning device 700).

[0065] The RAM 902 has an area for storing computer programs and data loaded from the ROM 903 or storage device 906, and an area for storing data received from the outside via the I / F 907. The RAM 902 also has a work area used when the CPU 901 executes various processes. In this way, the RAM 902 can provide various areas as needed.

[0066] The ROM 903 stores setting data for the computer device, computer programs and data related to the startup of the computer device, computer programs and data related to the basic operation of the computer device, and the like.

[0067] The operation unit 904 is a user interface such as a keyboard, a mouse, and a touch panel screen, and the user can input various instructions to the CPU 901 by operating it.

[0068] The display unit 905 has a liquid crystal screen or a touch panel screen, and can display the results of processing by the CPU 901 as images, characters, etc. The display unit 905 may be a projection device such as a projector that projects images and characters.

[0069] Storage device 906 is a large-capacity information storage device such as a hard disk drive. Storage device 906 stores computer programs and data for causing CPU 901 to execute or control the various processes described as processes performed by OS (operating system), learning device 100, inference device 200, and learning device 700. The computer programs and data stored in storage device 906 are loaded into RAM 902 as appropriate under the control of CPU 901, and become targets for processing by CPU 901. Note that the above-mentioned storage unit 170 can be implemented using, for example, RAM 902 or storage device 906.

[0070] The I / F 907 is a communication interface for performing data communication with an external device via a wired / wireless network such as a LAN, the Internet, etc. The CPU 901, the RAM 902, the ROM 903, the operation unit 904, the display unit 905, the storage device 906, and the I / F 907 are all connected to a system bus 908.

[0071] Note that the learning device and the inference device may each be a computer device having the same configuration, or the learning device and the inference device may each be a computer device having a different configuration.

[0072] In addition, the numerical values, processing timing, processing order, processing subject, data (information) acquisition method / destination / source / storage location, etc. used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0073] In addition, a part or all of the embodiments described above may be used in appropriate combination. In addition, a part or all of the embodiments described above may be used selectively.

[0074] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0075] The invention of this specification includes the following learning device, inference device, learning method, inference method, and computer program. (Item 1) a first acquisition means for acquiring a likelihood map of a specific portion in an input image using a first model for detecting the specific portion; A second acquisition means for acquiring an area map representing an area of ​​a specific part of the tracking target in the input image by using a second model for detecting the tracking target; a learning means for learning the second model based on a loss obtained based on an element product map obtained by an element product of the likelihood map and the area map, and ground truth data indicating an area of ​​a specific part of the tracking target in the input image; A learning device comprising: (Item 2) The learning device described in item 1, characterized in that the learning means obtains, as the correct answer map, a two-dimensional likelihood distribution having a mean vector that is a result of dividing the center coordinates of the area represented by the correct answer data by a size ratio between the correct answer map and the input image, and learns the second model based on a loss obtained based on the correct answer map and the element product map. (Item 3) The learning device according to item 1, characterized in that the learning means learns the second model based on the loss and a loss obtained based on the area map and an average of multiple area maps obtained in the past. (Item 4) The learning device according to any one of items 1 to 3, wherein the second acquisition means acquires a likelihood map of the tracking target in the input image by using the second model. (Item 5) a first acquisition means for acquiring a likelihood map of a specific portion in an input image using a first model for detecting the specific portion; A second acquisition means for acquiring an area map representing an area of ​​a specific part of the tracking target in the input image by using a second model for detecting the tracking target; a detection means for detecting a position of the specific portion in the input image based on an element product map obtained by an element product of the likelihood map and the region map; An inference device comprising: (Item 6) 6. The inference device according to item 5, wherein the detection means detects the position of the specific part in the input image based on the coordinates of the element having the maximum element value in the element product map. (Item 7) A learning method performed by a learning device, comprising: a first acquisition step in which a first acquisition means of the learning device acquires a likelihood map of the specific part in an input image using a first model for detecting the specific part; A second acquisition step in which a second acquisition means of the learning device acquires a region map representing a region of a specific part of the tracking target in the input image using a second model for detecting the tracking target; a learning step in which a learning means of the learning device learns the second model based on a loss obtained based on an element product map obtained by an element product of the likelihood map and the area map and ground truth data indicating an area of ​​a specific part of the tracking target in the input image; A learning method comprising: (Item 8) An inference method performed by an inference device, comprising: a first acquisition step in which a first acquisition means of the inference device acquires a likelihood map of the specific portion in an input image using a first model for detecting the specific portion; a second acquisition step in which a second acquisition means of the inference device acquires an area map representing an area of ​​a specific portion of the tracking target in the input image using a second model for detecting the tracking target; a detection step in which a detection means of the inference device detects a position of the specific portion in the input image based on an element product map obtained by an element product of the likelihood map and the region map; An inference method comprising: (Item 9) A computer program for causing a computer to function as each of the means of the learning device described in any one of items 1 to 4. (Item 10) A computer program for causing a computer to function as each of the means of the inference device according to item 5 or 6.

[0076] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0077] 110: Acquisition unit 120: Detection unit 130: Detection unit 140: Post-processing 150: Calculation unit 160: Update unit

Claims

1. a first acquisition means for acquiring a likelihood map of a specific portion in an input image using a first model for detecting the specific portion; a second acquisition means for acquiring an area map representing an area of ​​a specific part of the tracking target in the input image using a second model for detecting the tracking target; a learning means for learning the second model based on a loss obtained based on an element product map obtained by element-product of the likelihood map and the region map, and ground truth data indicating a region of a specific part of the tracking target in the input image; and A learning device comprising:

2. The learning device according to claim 1, characterized in that the learning means calculates, as the correct answer map, a two-dimensional likelihood distribution whose mean vector is the result of dividing the center coordinates of the area represented by the correct answer data by the size ratio between the correct answer map and the input image, and learns the second model based on a loss obtained based on the correct answer map and the element product map.

3. The learning device according to claim 1, characterized in that the learning means learns the second model based on the loss and a loss obtained based on the region map and an average of multiple region maps obtained in the past.

4. The learning device according to claim 1 , wherein the second acquisition means acquires a likelihood map of the tracking target in the input image using the second model.

5. a first acquisition means for acquiring a likelihood map of a specific portion in an input image using a first model for detecting the specific portion; a second acquisition means for acquiring an area map representing an area of ​​a specific part of the tracking target in the input image using a second model for detecting the tracking target; a detection means for detecting the position of the specific portion in the input image based on an element product map obtained by element-product of the likelihood map and the region map; An inference device comprising:

6. 6. The inference device according to claim 5, wherein said detecting means detects the position of said specific part in said input image based on the coordinates of the element having the largest element value in said element product map.

7. An inference device as described in Claim 5, characterized in that the first acquisition means and the second acquisition means are models trained so that the element product map approaches the GT of the specific part.

8. The inference device described in Claim 5, characterized in that the second acquisition means is a model trained so that the area map is closer to the area map in a past image than to the input image.

9. A learning method performed by a learning device, a first acquisition step in which a first acquisition means of the learning device acquires a likelihood map of a specific part in an input image using a first model for detecting the specific part; a second acquisition step in which a second acquisition means of the learning device acquires a region map representing a region of a specific part of the tracking target in the input image using a second model for detecting the tracking target; a learning step in which a learning means of the learning device learns the second model based on a loss obtained based on an element product map obtained by element multiplying the likelihood map and the area map, and ground truth data indicating an area of ​​a specific part of the tracking target in the input image; A learning method comprising:

10. An inference method performed by an inference device, a first acquisition step in which a first acquisition means of the inference device acquires a likelihood map of a specific part in an input image using a first model for detecting the specific part; a second acquisition step in which a second acquisition means of the inference device acquires a region map representing a region of a specific part of the tracking target in the input image using a second model for detecting the tracking target; a detection step in which a detection means of the inference device detects the position of the specific part in the input image based on an element product map obtained by element product of the likelihood map and the region map; An inference method comprising:

11. A computer program for causing a computer to function as each of the means of the learning device according to any one of claims 1 to 4.

12. A computer program for causing a computer to function as each of the means of the inference device according to any one of claims 5 to 8.

Images