Information processing method, information processing program, and information processing apparatus

Abstract

To provide an information processing method, an information processing program, and an information processing apparatus configured to suppress over-detection.SOLUTION: An information processing apparatus includes a training unit. The training unit trains a learning model 300 which is configured to receive, as input, a first image and a first sample image and output information on an estimated value of the number of target objects included in the first image, so as to minimize total error of a first error and a second error, the first error being an error between a first density map related to the number of target objects included in the first image, and a first ground-truth density map related to the correct number of the target objects included in the first image, the first density map being derived based on correlation between a first image feature quantity of the first image including the target objects and a first sample image feature quantity of the first sample image representing a unit of the target objects, and the second error being represented by a loss function related to a second image not including the target objects.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] An embodiment of the present invention relates to an information processing method, an information processing program, and an information processing device. [Background technology]

[0002] There are known techniques for estimating the number of objects contained in an image. For example, a system has been disclosed that uses positive example data, which are pairs of local images of an object and correct information for the local images, to train an estimation model for detecting objects contained in an image. Another technique has also been disclosed that derives a density map of the objects contained in the image as an output from the model by inputting an image containing the object and a sample image of the object into the model.

[0003] However, in the conventional technology, there are cases where overdetection occurs, where objects other than the objects to be counted are also counted. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6633462 [Non-patent literature]

[0005] [Non-Patent Document 1] Chang Liu, Yujie Zhong, Andrew Zisserman, Weidi Xie, CounTR: Transformer-based Generalised Visual Counting, BMVC2022. Summary of the Invention [Problem to be solved by the invention]

[0006] An object of the present invention is to provide an information processing method, an information processing program, and an information processing device that can suppress overdetection. [Means for solving the problem]

[0007] An information processing method according to an embodiment is an information processing method executed by a computer, and includes a learning step of: training a learning model that receives inputs of the first image and the first sample image and outputs information related to an estimated value of the number of the objects to be counted included in the first image, the learning model being derived based on a correlation between a first image feature of the first image in which the objects to be counted appear and a first sample image feature of the first sample image representing one unit of the objects to be counted, and a first correct density map related to the number of correct objects to be counted included in the first image; and a second error represented by a loss function related to a second image in which the objects to be counted do not appear. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an information processing apparatus according to an embodiment. [Figure 2A] Schematic diagram of the first image. [Figure 2B] Schematic diagram of the first sample image. [Figure 3] FIG. 10 is an explanatory diagram of calculation of a first error. [Figure 4] Schematic diagram of the second image. [Figure 5] FIG. 10 is an explanatory diagram of calculation of a second error. [Figure 6] Schematic diagram of a display screen. [Figure 7] 1 is a flowchart showing the flow of information processing executed by an information processing device. [Figure 8] FIG. 1 is a schematic diagram of an information processing device. [Figure 9] Schematic diagram of the second sample image. [Figure 10] FIG. 10 is an explanatory diagram of calculation of a first error. [Figure 11] 1 is a flowchart showing the flow of information processing executed by an information processing device. [Figure 12] FIG. 1 is a schematic diagram of an information processing apparatus according to an embodiment. [Figure 13] 1 is a flowchart showing the flow of information processing executed by an information processing device. [Figure 14] FIG. 1 is a schematic diagram of an information processing apparatus according to an embodiment. [Figure 15] FIG. 10 is an explanatory diagram of calculation of a fourth error. [Figure 16] 1 is a flowchart showing the flow of information processing executed by an information processing device. [Figure 17] Hardware configuration diagram. DETAILED DESCRIPTION OF THE INVENTION

[0009] The information processing method, the information processing program, and the information processing device according to the present embodiment will be described in detail below with reference to the accompanying drawings.

[0010] (First embodiment) 1 is a schematic diagram of an example of an information processing device 10A according to the present embodiment. The information processing device 10A is an example of the information processing device 10.

[0011] The information processing device 10A includes a storage unit 12, a UI (user interface) unit 14, a communication unit 16, and a control unit 20. The storage unit 12, the UI unit 14, the communication unit 16, and the control unit 20 are communicatively connected via a bus 18 or the like.

[0012] The storage unit 12 stores various types of data. The storage unit 12 may be provided outside the information processing device 10A. Alternatively, the storage unit 12 and at least one of one or more functional units included in the control unit 20 (described later) may be mounted on an external information processing device communicatively connected to the information processing device 10 via a network or the like.

[0013] The UI unit 14 has a display function for displaying various information and an input function for receiving user input. The display function is, for example, a display, a projection device, etc. The input function is, for example, a pointing device such as a mouse or a touchpad, a keyboard, etc. The UI unit 14 may be a touch panel that integrates the display function and the input function.

[0014] The UI unit 14 may be configured to be communicably connected to the control unit 20 via a wired or wireless connection. The UI unit 14 may be configured to be provided outside the information processing device 10A, and the UI unit 14 and the control unit 20 may be connected via a network or the like.

[0015] The communication unit 16 is a communication interface for communicating with an external information processing device or the like of the information processing device 10A.

[0016] The control unit 20 executes information processing in the information processing device 10A. The control unit 20 includes a first error calculation unit 30A, a second error calculation unit 30B, a learning unit 30C, and an output control unit 30D.

[0017] The first error calculation unit 30A, the second error calculation unit 30B, the learning unit 30C, and the output control unit 30D are realized, for example, by one or more processors. For example, each of the above units may be realized by having a processor such as a CPU (Central Processing Unit) execute a program, i.e., by software. Each of the above units may be realized by a processor such as a dedicated IC or circuit, i.e., by hardware. Each of the above units may be realized by a combination of software and hardware. When multiple processors are used, each processor may realize one of the units, or may realize two or more of the units.

[0018] The first error calculation unit 30A calculates the first error.

[0019] The first error is the error between the first density map and the first correct density map, and is represented by the solution of a loss function for calculating the error.

[0020] The first density map is information regarding the number of counting objects contained in the first image.

[0021] FIG. 2A is a schematic diagram of an example of a first image 40A. The first image 40A is an image in which the counting object T is captured. FIG. 2A and the figures described below show an example in which the counting object T is a mark shaped like the sun. The first image 40A may be either image data obtained by photography or image data created by an image creation application or the like. When the first image 40A is image data created by an image creation application or the like, the first image 40A may be an image that includes the counting object T.

[0022] The counting objects T are objects whose number is to be counted. First image 40A needs to include at least the counting objects T, and may also include non-counting objects other than the counting objects T. The crescent moon-shaped mark in FIG. 2A and the figures described below is an example of a non-counting object other than the counting objects.

[0023] The first density map may be information regarding the number of counting objects T included in the first image 40A. For example, the first density map is information representing the arrangement of one or more counting objects T included in the first image 40A by defining the position, area, and range of each of the counting objects T included in the first image 40A. When the first density map represents the arrangement of the counting objects T in the first image 40A, an estimated number of counting objects T included in the first image 40A is derived by performing image analysis of the arrangement. The first density map may also be information representing the estimated number of counting objects T included in the first image 40A.

[0024] The first correct answer density map is information regarding the number of correct answers for the counting objects T included in the first image 40A. The first correct answer density map may be information regarding the number of correct answers for the counting objects T included in the first image 40A. For example, the first correct answer density map is information representing the arrangement of correct answers for the counting objects included in the first image 40A by defining the position, area, and range of each of the multiple counting objects T included in the first image 40A. When the first correct answer density map represents the arrangement of correct answers for the counting objects T in the first image 40A, the number of correct answers for the counting objects T included in the first image 40A is derived by image analysis of the arrangement of the correct answers. The first correct answer density map may also be information representing the number of correct answers for the counting objects T included in the first image 40A.

[0025] The first correct density map may be prepared in advance for each first image 40A.

[0026] The first density map is derived based on the correlation between the first image feature amount of the first image 40A and the first sample image feature amount of the first sample image.

[0027] The first image feature amount represents the feature amount of the first image 40A. The first image feature amount is represented, for example, by a vector or the like representing the feature of the first image 40A. The first sample image feature amount represents the feature amount of the first sample image. The first sample image feature amount is represented, for example, by a vector or the like representing the feature of the first sample image.

[0028] FIG. 2B is a schematic diagram of an example of a first sample image 40B. The first sample image 40B is an image representing one unit of the counting object T included in the first image 40A. In other words, the first sample image 40B includes an area representing one unit of the counting object T. The first sample image 40B may be either an extracted image obtained by extracting an area representing one unit of the counting object T from the first image 40A, or a separately prepared image. FIG. 2B shows an example of a first sample image 40B generated by extracting an area representing one unit of the counting object T from the first image 40A.

[0029] FIG. 3 is a diagram illustrating an example of calculation of the first error L1 by the first error calculation section 30A.

[0030] The first error calculation unit 30A inputs the first image 40A to the first encoder 50, thereby obtaining a first image feature of the first image 40A as an output from the first encoder 50. The first encoder 50 is a model that receives an image as an input and outputs the feature of the image. The model is, for example, a deep neural network (DNN) model obtained by deep learning.

[0031] Furthermore, the first error calculation unit 30A inputs the first sample image 40B to the second encoder 52, thereby obtaining first sample image feature amounts of the first sample image 40B as output from the second encoder 52. The second encoder 52 is a model that receives an image as input and outputs feature amounts of the image. The model is, for example, a deep neural network (DNN) model obtained by deep learning.

[0032] The first error calculation unit 30A inputs the first image feature and the first sample image feature to a third decoder 54, thereby obtaining a first density map of the first image 40A as an output from the third decoder 54. The third decoder 54 is a model that receives two types of image feature as input and outputs a density map based on the correlation between these image feature. The model is, for example, a deep neural network (DNN) model obtained by deep learning.

[0033] The first error calculation unit 30A calculates the loss function of the obtained first density map and the first correct density map to calculate the first error L1.

[0034] The loss function for calculating the first error L1 is expressed by the least square error shown in the following formula (1).

[0035]

number

[0036] In equation (1), L1 represents the first error, N represents the number of training data, and i represents an integer between 1 and N. i represents the first image 40A. i is the first image 40A, x i b represents the first correct density map of i represents the first sample image 40B. f(x i ,b i ) represents the first density map.

[0037] In addition, the first error calculation unit 30A may input a first image feature derived from one first image 40A and multiple first sample image feature values ​​derived from each of multiple first sample images 40B to the third decoder 54, calculate a first density map based on the correlation between them, and use it to calculate the first error L1.

[0038] Returning to Figure 1, we continue the explanation.

[0039] The second error calculation unit 30B calculates the second error.

[0040] The second error is an error represented by a loss function for the second image that does not include the counting target T. For example, the second error represents the error between the density map of the counting target T or the non-counting target derived from the second image and the density map regarding the correct number of the counting target T or the non-counting target.

[0041] In this embodiment, a case will be described in which the second error is an error calculated by a loss function between the second density map and a second correct density map indicating that the counting target T is not included in the second image.

[0042] The second density map is information regarding the number of counting objects contained in the second image.

[0043] 4 is a schematic diagram of an example of the second image 42A. The second image 42A is an image in which the counting object T is not captured. The second image 42A may be either image data obtained by photography or image data created by an image creation application or the like. When the second image 42A is image data created by an image creation application or the like, the second image 42A may be an image that does not include the counting object T.

[0044] Furthermore, the second image 42A may be an image that does not include a counting target T, and may include a non-counting target NT. A non-counting target NT is an object other than the counting target T, and is an object different from the counting target T. FIG. 4 shows an example in which the second image 42A is an image that includes a non-counting target NT.

[0045] The second density map may be information regarding the number of counting objects T included in the second image 42A. For example, the second density map is information representing the arrangement of the counting objects T included in the second image 42A by defining the position, area, and range of each of one or more counting objects T included in the second image 42A. When the second density map represents the arrangement of the counting objects T in the second image 42A, the number of counting objects T included in the second image 42A is derived by performing image analysis of the arrangement. The second density map may also be information representing an estimated number of counting objects T included in the second image 42A.

[0046] The second correct answer density map is information regarding the number of correct answers for the counting objects T included in the second image 42A. The second correct answer density map may be information regarding the number of correct answers for the counting objects T included in the second image 42A. For example, the second correct answer density map is information representing the arrangement state of the correct answers for the counting objects T included in the second image 42A by defining the position, area, and range of each of the multiple counting objects T included in the second image 42A. When the second correct answer density map represents the arrangement state of the correct answers for the counting objects T in the second image 42A, the number of correct answers for the counting objects T included in the second image 42A is derived by image analysis of the arrangement state of the correct answers. Furthermore, the second correct answer density map may be information representing the number of correct answers for the counting objects T included in the second image 42A.

[0047] As described above, the second image 42A is an image that does not include the counting objects T. Therefore, the second correct density map only needs to be information indicating that the number of counting objects T included in the second image 42A is zero (that is, the counting objects T are not included).

[0048] The second density map is derived based on the correlation between the second image feature amount of the second image 42A and the first sample image feature amount of the first sample image 40B.

[0049] The second image feature amount represents a feature amount of the second image 42 A. The second image feature amount is represented by, for example, a vector representing the feature of the second image 42 A.

[0050] FIG. 5 is a diagram illustrating an example of calculation of the second error L2 by the second error calculation section 30B.

[0051] The second error calculation unit 30B inputs the second image 42A to the first encoder 50, and obtains the second image feature amount of the second image 42A as an output from the first encoder 50.

[0052] Furthermore, the second error calculation unit 30B inputs the first sample image 40B to the second encoder 52, and obtains the first sample image feature amount of the first sample image 40B as an output from the second encoder 52.

[0053] The second error calculation unit 30B inputs the second image features and the first sample image features to the third decoder 54, and obtains, as output from the third decoder 54, a second density map regarding the number of counting objects T contained in the second image 42A.

[0054] The second error calculation unit 30B calculates the second error L2 by calculating a loss function of the obtained second density map and the second correct density map.

[0055] The loss function for calculating the second error L2 is expressed by the least square error shown in the following formula (2).

[0056]

number

[0057] In equation (2), L2 represents the second error, N represents the number of training data, and i represents an integer between 1 and N. i represents the second image 42A. i represents the first sample image 40B. f(x' i ,b i ) represents the second density map. Note that since the number of correct answers for the counting target T included in the second image 42A, which is represented by the second correct density map, is zero, the loss function of the second density map and the second correct density map is expressed by Equation (2).

[0058] In addition, the second error calculation unit 30B may input the second image feature derived from one second image 42A and multiple first sample image feature derived from each of the multiple first sample images 40B to the third decoder 54, calculate a second density map based on the correlation between them, and use it to calculate the second error L2.

[0059] Returning to Figure 1, we continue the explanation.

[0060] The learning unit 30C learns the learning model 300 so as to minimize the total error of the first error L1 and the second error L2.

[0061] The learning model 300 is a learning model that receives the first image 40A and the first sample image 40B as input and outputs information related to an estimated value of the number of counting objects T included in the first image 40A. The learning model 300 is, for example, a deep neural network (DNN) model obtained by deep learning.

[0062] The learning unit 30C learns the learning model 300 so as to minimize the total error expressed by equation (A).

[0063] L=L1+L2 Equation (A)

[0064] In formula (A), L represents the total error, L1 represents the first error, and L2 represents the second error.

[0065] For example, the learning unit 30C updates the parameters of the first encoder 50, the second encoder 52, and the third decoder 54 so as to minimize the total error L, and applies the latest updated parameters to the learning model 300, thereby learning the learning model 300.

[0066] Returning to Figure 1, we continue the explanation.

[0067] The output control unit 30D outputs information regarding an estimated value of the number of counting objects T included in the first image 40A, which is derived by inputting the first image 40A and the first sample image 40B into the learning model 300. "Output" means displaying, transmitting, storing, etc. That is, the output control unit 30D displays on the UI unit 14, transmits to an external information processing device via the communication unit 16, or stores in the memory unit 12 at least one of the learning model 300 and information regarding an estimated value of the number of counting objects T included in the first image 40A, which is derived from the learning model 300 by inputting the first image 40A and the first sample image 40B into the learning model 300.

[0068] Specifically, for example, the output control unit 30D outputs information relating to the estimated value to the UI unit 14.

[0069] FIG. 6 is a schematic diagram of an example of a display screen 60 displayed on the UI unit 14. As shown in FIG.

[0070] Display screen 60 includes, for example, first image 40A, first example image 40B, first density map M1, and estimate 60D.

[0071] For example, when the first image 40A is selected by a user operating the UI unit 14, the output control unit 30D displays the display screen 60 including the selected first image 40A on the UI unit 14. Then, when the output control unit 30D receives a selection of an area of ​​the user's desired counting target T included in the first image 40A by a user operating the UI unit 14, the output control unit 30D extracts the received area as the first sample image 40B.

[0072] 6 shows an example of a scene in which a unit area of ​​the counting object T included in the first image 40A is selected by frame F in response to a user's operation instruction on the UI unit 14. In this case, the output control unit 30D extracts the area within frame F in the first image 40A as a first sample image 40B of the counting object T.

[0073] Then, the output control unit 30D displays the extracted first sample image 40B on the display screen 60 as the counting object T.

[0074] Furthermore, for example, the output control unit 30D inputs the first image 40A and the first sample image 40B into the learning model 300, thereby obtaining a first density map M1 as an output from the learning model 300. The output control unit 30D also performs image analysis of the first density map M1 using a known method to derive an estimated value 60D of the number of counting objects T included in the first image 40A, and displays it on the display screen 60. FIG. 6 shows an example of a scene in which "6" is derived and displayed as the estimated value 60D.

[0075] The output control unit 30D may further display a superimposed image in which the first density map M1 is superimposed on the first image 40A.

[0076] Next, an example of the flow of information processing executed by the information processing device 10A of this embodiment will be described.

[0077] FIG. 7 is a flowchart showing an example of the flow of information processing executed by the information processing device 10A of this embodiment.

[0078] The first error calculation unit 30A calculates a first error L1 between the first density map derived based on the correlation between the first image feature of the first image 40A and the first sample image feature of the first sample image 40B and the first correct density map (step S100).

[0079] The second error calculation unit 30B calculates a second error L2 between the second density map derived based on the correlation between the second image feature amount of the second image 42A and the first sample image feature amount and the second correct density map (step S102).

[0080] The learning unit 30C learns the learning model 300 so as to minimize the total error of the first error L1 and the second error L2 (step S104).

[0081] The output control unit 30D outputs information regarding the estimated value of the number of counting objects T included in the first image 40A, which is derived by inputting the first image 40A and the first sample image 40B to the learning model 300 trained in step S104 (step S106). Then, this routine ends.

[0082] As described above, in the information processing device 10 of this embodiment, the learning unit 30C learns the learning model 300, which inputs the first image 40A and the first sample image 40B and outputs information related to the estimated number of counting objects T included in the first image 40A, so as to minimize the total error of the first error L1 and the second error L2. The first error L1 is the error between a first density map related to the number of counting objects T included in the first image 40A, which is derived based on the correlation between the first image feature of the first image 40A in which the counting objects T appear and the first sample image feature of the first sample image 40B representing one unit of the counting objects T, and a first correct density map related to the number of correct counting objects T included in the first image 40A. The second error L2 is an error represented by a loss function related to the second image 42A in which the counting objects T do not appear.

[0083] In this way, in the information processing device 10A of this embodiment, the learning model 300 is trained to minimize the total error of the second error L2 and the first error L1 using the second error L2 represented by a loss function of information derived from the second image 42A in which the counting object T does not appear and correct information indicating that the counting object T does not appear.

[0084] That is, in the information processing device 10A of this embodiment, in addition to the first image 40A in which the counting objects T appear, the second image 42A in which the counting objects T do not appear is also used to train the learning model 300. Therefore, the information processing device 10A of this embodiment can prevent non-counting objects NT other than the counting objects T included in the first image 40A from being counted as counting objects T, and can train the learning model 300 in which overdetection is suppressed. Overdetection means that objects other than the counting objects T are also counted. Therefore, in the information processing device 10 of this embodiment, by using the learning model 300 trained according to this embodiment, it is possible to suppress overdetection of the number of counting objects T included in the first image 40A and to count the number of counting objects T with high accuracy.

[0085] Therefore, the information processing device 10A of this embodiment can suppress overdetection.

[0086] (Second embodiment) In this embodiment, a form is described in which the second error L2 is a third error represented by a loss function between a third density map regarding the number of non-counting targets NT included in the first image 40A and a third correct density map regarding the correct number of non-counting targets NT included in the first image 40A.

[0087] In this embodiment, the same parts as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0088] 8 is a schematic diagram of an example of an information processing device 10B according to the present embodiment. The information processing device 10B is an example of the information processing device 10.

[0089] The information processing device 10B includes a storage unit 12, a UI unit 14, a communication unit 16, and a control unit 22. The storage unit 12, the UI unit 14, the communication unit 16, and the control unit 22 are communicatively connected via a bus 18 or the like.

[0090] The information processing device 10B has the same configuration as the information processing device 10A, except that it includes a control unit 22 instead of the control unit 20.

[0091] The control unit 22 executes information processing in the information processing device 10B. The control unit 22 includes a first error calculation unit 30A, a third error calculation unit 32B, a learning unit 32C, and an output control unit 30D. The control unit 22 is similar to the control unit 20 of the information processing device 10A of the above embodiment, except that the control unit 22 includes a third error calculation unit 32B and a learning unit 32C instead of the second error calculation unit 30B and the learning unit 30C.

[0092] The third error calculation unit 32B calculates the third error as the second error.

[0093] The third error is the error between the third density map and the third correct density map, and is represented by the solution of a loss function for calculating the error.

[0094] The third density map is information regarding the number of non-counting targets NT included in the first image 40A.

[0095] The third density map may be information regarding the number of non-count objects NT included in the first image 40A. For example, the third density map is information representing the arrangement of the non-count objects NT included in the first image 40A by defining the positions, areas, and ranges of the non-count objects NT included in the first image 40A. When the third density map represents the arrangement of the non-count objects NT in the first image 40A, an estimated number of non-count objects NT included in the first image 40A is derived by performing image analysis on the arrangement. Furthermore, the third density map may be information representing the estimated number of non-count objects NT included in the first image 40A.

[0096] The third correct answer density map is information regarding the number of correct answers for non-counting objects NT included in the first image 40A. The third correct answer density map may be information regarding the number of correct answers for non-counting objects NT included in the first image 40A. For example, the third correct answer density map is information representing the arrangement of correct answers for non-counting objects NT included in the first image 40A by defining the position, area, and range of each non-counting object NT included in the first image 40A. When the third correct answer density map represents the arrangement of correct answers for non-counting objects NT in the first image 40A, the number of correct answers for non-counting objects NT included in the first image 40A is derived by image analysis of the arrangement of the correct answers. Furthermore, the third correct answer density map may be information representing the number of correct answers for non-counting objects NT included in the first image 40A.

[0097] The third correct density map may be prepared in advance for each first image 40A.

[0098] The third density map is derived based on the correlation between the first image feature amount of the first image 40A and the second sample image feature amount of the second sample image.

[0099] 9 is a schematic diagram of an example of the second sample image 42B. The second sample image 42B is an image representing one unit of the non-counting objects NT. In other words, the second sample image 42B includes an area representing one unit of the non-counting objects NT. The second sample image 42B may be either an extracted image obtained by extracting an area representing one unit of the non-counting objects NT from the first image 40A, or a separately prepared image.

[0100] The second sample image feature amount represents the feature amount of the second sample image 42 B. The second sample image feature amount is represented by, for example, a vector representing the feature of the second sample image 42 B.

[0101] FIG. 10 is a diagram illustrating an example of calculation of the third error L3 by the third error calculation unit 32B.

[0102] The third error calculation unit 32B inputs the first image 40A to the first encoder 50, and obtains the first image feature amount of the first image 40A as an output from the first encoder 50.

[0103] Furthermore, the third error calculation unit 32B inputs the second sample image 42B to the second encoder 52, thereby obtaining the second sample image feature amount of the second sample image 42B as an output from the second encoder 52.

[0104] The third error calculation unit 32B inputs the first image feature and the second sample image feature to the third decoder 54, and obtains, as output from the third decoder 54, a third density map regarding the number of non-counting objects NT contained in the first image 40A.

[0105] The third error calculation unit 32B calculates a loss function of the obtained third density map and the third correct density map to calculate the third error L3.

[0106] The loss function for calculating the third error L3 is expressed, for example, by the least square error shown in the following equation (3).

[0107]

number

[0108] In equation (3), L3 represents the third error, N represents the number of training data, and i represents an integer between 1 and N. i represents the first image 40A. i represents the second sample image 42B. f(x i ,b' i ) represents the third density map. Note that equation (3) shows an example in which the number of correct answers for non-counting targets NT included in the first image 40A, as represented by the third correct answer density map, is zero. In this case, the loss function of the third density map and the third correct answer density map is expressed by equation (3).

[0109] The third error calculation unit 32B may input the first image feature derived from one first image 40A and multiple second sample image feature derived from each of multiple second sample images 42B to the third decoder 54, calculate a third density map based on the correlation between them, and use it to calculate the third error L3.

[0110] Returning to FIG. 8, the explanation will be continued.

[0111] The learning unit 32C learns the learning model 300 so as to minimize the total error of the first error L1 and the third error L3.

[0112] The learning unit 30C learns the learning model 300 so as to minimize the total error expressed by equation (B).

[0113] L=L1+L3...Formula (B)

[0114] In formula (B), L represents the total error, L1 represents the first error, and L3 represents the third error.

[0115] For example, the learning unit 30C updates the parameters of the first encoder 50, the second encoder 52, and the third decoder 54 so as to minimize the total error L, and applies the latest updated parameters to the learning model 300, thereby learning the learning model 300.

[0116] Next, an example of the flow of information processing executed by the information processing device 10B of this embodiment will be described.

[0117] FIG. 11 is a flowchart showing an example of the flow of information processing executed by the information processing device 10B of this embodiment.

[0118] The first error calculation unit 30A calculates a first error L1 between the first density map derived based on the correlation between the first image feature of the first image 40A and the first sample image feature of the first sample image 40B and the first correct density map (step S200).

[0119] The third error calculation unit 32B calculates a third error L3 represented by a loss function between a third density map derived based on the correlation between the second sample image feature of the second sample image 42B representing one unit of the non-counting object NT and the first image feature, and a third correct answer density map regarding the number of correct answers for the non-counting object NT included in the first image 40A (step S202).

[0120] The learning unit 32C learns the learning model 300 so as to minimize the total error of the first error L1 and the third error L3 (step S204).

[0121] The output control unit 30D outputs information about the estimated value of the number of counting objects T included in the first image 40A, which is derived by inputting the first image 40A and the first sample image 40B to the learning model 300 trained in step S204 (step S206). Then, this routine ends.

[0122] As described above, in the information processing device 10 of this embodiment, the learning unit 30C uses the third error L3 as the second error L2. The learning unit 30C then trains the learning model 300 so as to minimize the total error of the first error L1 and the third error L3. The third error L3 is an error represented by a loss function of a third density map related to the number of non-counting targets NT included in the first image 40A, which is derived based on the correlation between the second sample image feature amount and the first image feature amount of the second sample image 42B, which is a second image representing one unit of non-counting targets NT, and a third correct answer density map related to the number of correct answers for the non-counting targets NT included in the first image 40A.

[0123] In this way, in the information processing device 10B of this embodiment, the learning model 300 is trained to minimize the total error of the third error L3 and the first error L1 using the third error L3 represented by a loss function of information derived from the second sample image 42B in which the counting object T is not reflected and information regarding the number of correct answers for the non-counting object NT that is different from the counting object T.

[0124] That is, in the information processing device 10B of this embodiment, in addition to the first image 40A in which the counting objects T appear, a second sample image 42B in which the counting objects T do not appear and which represents one unit of the non-counting objects NT is further used to train the learning model 300. Therefore, the information processing device 10B of this embodiment can prevent non-counting objects NT other than the counting objects T included in the first image 40A from being counted as the counting objects T, and can train the learning model 300 in which overdetection is suppressed. Therefore, in the information processing device 10 of this embodiment, by using the learning model 300 trained according to this embodiment, it is possible to suppress overdetection of the number of counting objects T included in the first image 40A and to count the number of counting objects T with high accuracy.

[0125] Therefore, the information processing device 10B of this embodiment can suppress overdetection.

[0126] (Third embodiment) In this embodiment, a form will be described in which the second error L2 in the first embodiment and the third error L3 in the second embodiment are used as the second error.

[0127] In this embodiment, the same parts as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0128] 12 is a schematic diagram of an example of an information processing device 10C according to this embodiment. The information processing device 10C is an example of the information processing device 10.

[0129] The information processing device 10C includes a storage unit 12, a UI unit 14, a communication unit 16, and a control unit 23. The storage unit 12, the UI unit 14, the communication unit 16, and the control unit 23 are communicatively connected via a bus 18 or the like.

[0130] The information processing device 10C has the same configuration as the information processing device 10A, except that it includes a control unit 23 instead of the control unit 20.

[0131] The control unit 23 executes information processing in the information processing device 10C. The control unit 23 includes a first error calculation unit 30A, a second error calculation unit 30B, a third error calculation unit 32B, a learning unit 33C, and an output control unit 30D. The control unit 23 is similar to the control unit 20 of the information processing device 10A of the above embodiment, except that it further includes a third error calculation unit 32B and includes a learning unit 33C instead of the learning unit 30C. The third error calculation unit 32B is similar to the third error calculation unit 32B of the above embodiment.

[0132] The learning unit 33C learns the learning model 300 so as to minimize the total error of the first error L1 calculated by the first error calculation unit 30A, the second error L2 calculated by the second error calculation unit 30B, and the third error L3 calculated by the third error calculation unit 32B.

[0133] The learning unit 33C learns the learning model 300 so as to minimize the total error expressed by equation (C).

[0134] L=L1+L2+L3 Equation (C)

[0135] In formula (C), L represents the total error, L1 represents the first error, L2 represents the second error, and L3 represents the third error.

[0136] For example, the learning unit 33C updates the parameters of the first encoder 50, the second encoder 52, and the third decoder 54 so as to minimize the total error L, and applies the latest updated parameters to the learning model 300, thereby learning the learning model 300.

[0137] Next, an example of the flow of information processing executed by the information processing device 10B of this embodiment will be described.

[0138] FIG. 13 is a flowchart showing an example of the flow of information processing executed by the information processing device 10C of this embodiment.

[0139] The first error calculation unit 30A calculates a first error L1 between the first density map derived based on the correlation between the first image feature of the first image 40A and the first sample image feature of the first sample image 40B and the first correct density map (step S300).

[0140] The second error calculation unit 30B calculates a second error L2 between the second density map derived based on the correlation between the second image feature amount of the second image 42A and the first sample image feature amount and the second correct density map (step S302).

[0141] The third error calculation unit 32B calculates a third error L3 represented by a loss function between a third density map derived based on the correlation between the second sample image feature of the second sample image 42B representing one unit of the non-counting object NT and the first image feature, and a third correct answer density map regarding the number of correct answers for the non-counting object NT included in the first image 40A (step S304).

[0142] The learning unit 32C learns the learning model 300 so as to minimize the total error of the first error L1, the second error L2, and the third error L3 (step S306).

[0143] The output control unit 30D outputs information about the estimated value of the number of counting objects T included in the first image 40A, which is derived by inputting the first image 40A and the first sample image 40B to the learning model 300 trained in step S306 (step S308). Then, this routine ends.

[0144] As described above, in the information processing device 10C of this embodiment, the learning unit 33C learns the learning model 300 so as to minimize the total error of the first error L1, the second error L2, and the third error L3.

[0145] In this way, in the information processing device 10C of this embodiment, the learning model 300 is trained to minimize the total error of the first error L1, the second error L2, and the third error L3, which are represented by the loss functions of the information derived from each of the second image 42A and the second sample image 42B in which the counting object T does not appear, and the correct answer information indicating that the counting object T does not appear.

[0146] That is, in the information processing device 10C of this embodiment, in addition to the first image 40A in which the counting objects T appear, the learning model 300 is trained using the second image 42A and the second sample image 42B in which the counting objects T do not appear. Therefore, the information processing device 10C of this embodiment can prevent non-counting objects NT other than the counting objects T included in the first image 40A from being counted as the counting objects T, and can train the learning model 300 in which overdetection is suppressed. Therefore, by using the learning model 300 trained according to this embodiment, the information processing device 10C of this embodiment can prevent overdetection of the number of counting objects T included in the first image 40A and can count the number of counting objects T with high accuracy.

[0147] Therefore, the information processing device 10C of this embodiment can suppress overdetection.

[0148] (Fourth embodiment) In this embodiment, a form will be described in which the learning model 300 is trained so as to minimize the sum of the first error L1, the second error L2, and the third error L3 and the fourth error.

[0149] In this embodiment, the same parts as those in the above embodiment are given the same reference numerals, and detailed description thereof will be omitted.

[0150] 14 is a schematic diagram of an example of an information processing device 10D according to this embodiment. The information processing device 10D is an example of the information processing device 10.

[0151] The information processing device 10D includes a storage unit 12, a UI unit 14, a communication unit 16, and a control unit 24. The storage unit 12, the UI unit 14, the communication unit 16, and the control unit 24 are communicatively connected via a bus 18 or the like.

[0152] The information processing device 10D has the same configuration as the information processing device 10A, except that it includes a control unit 24 instead of the control unit 20.

[0153] The control unit 24 executes information processing in the information processing device 10D. The control unit 24 includes a first error calculation unit 30A, a second error calculation unit 30B, a third error calculation unit 32B, a fourth error calculation unit 34B, a learning unit 34C, and an output control unit 30D. The control unit 22 is similar to the control unit 20 of the information processing device 10A of the above embodiment, except that it further includes a third error calculation unit 32B and a fourth error calculation unit 34B, and includes a learning unit 34C instead of the learning unit 30C. The third error calculation unit 32B is similar to that of the above embodiment.

[0154] The fourth error calculation unit 34B calculates a fourth error, which is the error between the estimated label of the counting object T that appears in the first image 40A and the correct label of the counting object T.

[0155] FIG. 15 is a diagram illustrating an example of calculation of the fourth error L4.

[0156] The fourth error calculation unit 34B derives an estimated label of the counting object T from the correlation between the first sample image feature amount of the first sample image 40B and the second sample image feature amount of the second sample image 42B.

[0157] For example, the fourth error calculation unit 34B inputs the first sample image feature amount and the second sample image feature amount to an encoder 56 that receives the first sample image feature amount and the second sample image feature amount and outputs an estimated label representing the counting object T included in the first sample image 40B. The encoder 56 is, for example, a model that receives the first sample image feature amount and the second sample image feature amount and outputs an estimated label of the counting object T included in the first sample image 40B. The model is, for example, a deep neural network (DNN) model obtained by deep learning.

[0158] Then, the fourth error calculation unit 34B calculates the fourth error L4 by calculating a loss function of the estimated label and the correct label of the counting object T. The correct label of the counting object T may be stored in advance in the storage unit 12 or the like. For example, the correct label of the counting object T may be input in advance by a user's operation instruction on the UI unit 14 or the like and stored in advance in the storage unit 12. Furthermore, for example, the fourth error calculation unit 34B may derive the correct label in advance by clustering or the like based on features extracted from the first image 40A by a known feature extractor for image recognition, and store the correct label in advance in the storage unit 12.

[0159] The loss function for calculating the fourth error L4 is expressed by, for example, the following equation (4).

[0160]

number

[0161] In equation (4), L4 represents the fourth error. N represents the number of training data, i represents an integer between 1 and N, C represents the number of object categories, and j represents an integer between 1 and C. ij represents the correct label of the object category of the counting target T. ij is the first sample image and the second sample image bi The fourth error L4 is the cross-entropy.

[0162] Returning to FIG. 14, the explanation will be continued.

[0163] The learning unit 34C learns the learning model 300 so as to minimize the sum of the first error L1, the second error L2, and the third error L3 plus the fourth error L4.

[0164] The learning unit 30C learns the learning model 300 so as to minimize the addition result expressed by equation (D).

[0165] L'=L1+L2+L3+L4 Equation (D)

[0166] In equation (D), L' represents the addition result, L1 represents the first error, L2 represents the second error, L3 represents the third error, and L4 represents the fourth error.

[0167] For example, the learning unit 30C updates the parameters of the first encoder 50, the second encoder 52, the third decoder 54, and the encoder 56 so as to minimize the addition result L', and applies the latest updated parameters to the learning model 300, thereby learning the learning model 300.

[0168] The learning unit 30C may train the learning model 300 so as to minimize the sum of the error L, which is the sum of the first error L1 and the second error L2, and the fourth error L4, described in the first embodiment. The learning unit 30C may train the learning model 300 so as to minimize the sum of the error L, which is the sum of the first error L1 and the third error L3, and the fourth error L4, described in the second embodiment.

[0169] Next, an example of the flow of information processing executed by the information processing device 10D of this embodiment will be described.

[0170] FIG. 16 is a flowchart showing an example of the flow of information processing executed by the information processing device 10D of this embodiment.

[0171] The first error calculation unit 30A calculates a first error L1 between the first density map derived based on the correlation between the first image feature of the first image 40A and the first sample image feature of the first sample image 40B and the first correct density map (step S400).

[0172] The second error calculation unit 30B calculates a second error L2 between the second density map derived based on the correlation between the second image feature amount of the second image 42A and the first sample image feature amount and the second correct density map (step S402).

[0173] The third error calculation unit 32B calculates a third error L3 represented by a loss function between a third density map derived based on the correlation between the second sample image feature of the second sample image 42B representing one unit of the non-counting object NT and the first image feature, and a third correct answer density map regarding the number of correct answers for the non-counting object NT included in the first image 40A (step S404).

[0174] The fourth error calculation unit 34B calculates a fourth error L4 between the estimated label of the counting object T appearing in the first image 40A, derived from the correlation between the first sample image feature of the first sample image 40B and the second sample image feature of the second sample image 42B representing one unit of the non-counting object NT different from the counting object T, and the correct label of the counting object T (step S406).

[0175] The learning unit 32C learns the learning model 300 so as to minimize the sum of the first error L1, the second error L2, and the third error L3 and the fourth error L4 (step S408).

[0176] The output control unit 30D outputs information regarding the estimated value of the number of counting objects T included in the first image 40A, which is derived by inputting the first image 40A and the first sample image 40B to the learning model 300 trained in step S408 (step S410). Then, this routine ends.

[0177] As described above, in the information processing device 10D of this embodiment, the learning unit 30C learns the learning model 300 so as to minimize the sum of the first error L1 and at least one of the second error L2 and the third error L3 and the fourth error L4.

[0178] In this way, the information processing device 10D of this embodiment trains the learning model 300 by further using the estimated label of the counting object T.

[0179] Therefore, in the information processing device 10D of this embodiment, compared to the above embodiment, the learning model 300 can be further trained to be able to identify the counting objects T included in the first image 40A. Therefore, by using the learning model 300 of this embodiment, it is possible to count the counting objects T included in the first image 40A with higher accuracy, and it is possible to further suppress overdetection.

[0180] Next, an example of the hardware configuration of the information processing device 10 (information processing device 10A to information processing device 10D) of the above embodiment will be described.

[0181] FIG. 17 is a hardware configuration diagram of an example of the information processing device 10 of the above embodiment.

[0182] The information processing device 10 of the above embodiment includes a control device such as a CPU (Central Processing Unit) 90D, a storage device such as a ROM (Read Only Memory) 90E, a RAM (Random Access Memory) 90F, and a HDD (Hard Disk Drive) 90G, an I / F unit 90B that interfaces with various devices, an output unit 90A that outputs various information, an input unit 90C that accepts user operations, and a bus 90H that connects each unit, and has a hardware configuration that utilizes a normal computer.

[0183] In the information processing device 10 of the above embodiment, the CPU 90D reads out a program from the ROM 90E onto the RAM 90F and executes it, thereby realizing each of the above units on the computer.

[0184] The programs for executing the above processes executed by the information processing device 10 of the embodiment may be stored in the HDD 90G. Also, the programs for executing the above processes executed by the information processing device 10 of the embodiment may be provided by being pre-installed in the ROM 90E.

[0185] Furthermore, the program for executing the above-described processes executed by the information processing device 10 of the above-described embodiment may be stored in an installable or executable file format on a computer-readable storage medium such as a CD-ROM, CD-R, memory card, DVD (Digital Versatile Disc), or flexible disk (FD) and provided as a computer program product. Furthermore, the program for executing the above-described processes executed by the information processing device 10 of the above-described embodiment may be stored on a computer connected to a network such as the Internet and provided by being downloaded via the network. Furthermore, the program for executing the above-described processes executed by the information processing device 10 of the above-described embodiment may be provided or distributed via a network such as the Internet.

[0186] Although the present embodiment has been described above, the above embodiment is presented as an example and is not intended to limit the scope of the invention. This novel embodiment can be embodied in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. This embodiment and its modifications are included within the scope and spirit of the invention, and are also included in the inventions and their equivalents as defined in the claims. [Explanation of symbols]

[0187] 10, 10A, 10B, 10C, 10D Information processing device 30A 1st error calculation section 30B 2nd error calculation section 32B 3rd error calculation section 34B 4th error calculation section 30C, 32C, 33C, 34C Learning Department 30D Output control section

Claims

1. 1. A computer-implemented information processing method, comprising: a first error between a first density map relating to the number of counting objects included in the first image, which is derived based on a correlation between a first image feature amount of the first image in which the counting objects appear and a first sample image feature amount of a first sample image representing one unit of the counting objects, and a first correct answer density map relating to the number of correct answers of the counting objects included in the first image; a second error represented by a loss function for a second image that does not include the target to be counted; a learning step of learning a learning model that receives the first image and the first sample image as input and outputs information about an estimated value of the number of objects to be counted contained in the first image so as to minimize a total error of An information processing method including:

2. a second error calculation step of calculating the second error calculated by the loss function of a second density map relating to the number of the objects to be counted included in the second image, which is derived based on a correlation between a second image feature amount of the second image in which the objects to be counted are not included and the first sample image feature amount, and a second correct density map indicating that the objects to be counted are not included in the second image; The learning step training the learning model so as to minimize the total error between the first error and the second error; The information processing method according to claim 1 .

3. a third error calculation step of calculating a third error, which is the second error represented by the loss function of a third density map relating to the number of the non-count objects included in the first image, which is derived based on a correlation between a second sample image feature amount of a second sample image, which is the second image representing one unit of a non-count object different from the count object, and the first image feature amount, and a third correct answer density map relating to the number of correct answers of the non-count objects included in the first image; The learning step training the learning model so as to minimize the total error of the first error and the third error; The information processing method according to claim 1 .

4. a second error calculation step of calculating the second error calculated by the loss function of a second density map relating to the number of the objects to be counted included in the second image, which is derived based on a correlation between a second image feature amount of the second image in which the objects to be counted are not included and the first sample image feature amount, and a second correct answer density map indicating that the objects to be counted are not included in the second image; a third error calculation step of calculating a third error between a third density map relating to the number of non-countable objects included in the first image, which is derived based on a correlation between a second sample image feature amount of a second sample image representing one unit of a non-countable object different from the countable object, and the first image feature amount, and a third correct density map relating to the correct number of non-countable objects included in the first image; Including, The learning step training the learning model so as to minimize the total error of the first error, the second error, and the third error; The information processing method according to claim 1 .

5. a fourth error calculation step of calculating a fourth error between an estimated label of the counting object appearing in the first image, derived from a correlation between a first sample image feature amount of the first sample image and a second sample image feature amount of a second sample image representing one unit of a non-counting object different from the counting object, and a correct label of the counting object; The learning step training the learning model so as to minimize a sum of the total error and the fourth error; The information processing method according to any one of claims 1 to 4.

6. an output control step of outputting information about the estimated value derived by inputting the first image and the first sample image into the learning model; The information processing method according to claim 1 , comprising:

7. a first error between a first density map relating to the number of counting objects included in the first image, which is derived based on a correlation between a first image feature amount of the first image in which the counting objects appear and a first sample image feature amount of a first sample image representing one unit of the counting objects, and a first correct answer density map relating to the number of correct answers of the counting objects included in the first image; a second error represented by a loss function for a second image that does not include the target to be counted; a learning step of learning a learning model that receives the first image and the first sample image as input and outputs information about an estimated value of the number of objects to be counted contained in the first image so as to minimize a total error of An information processing program that causes a computer to execute the above.

8. a first error between a first density map relating to the number of counting objects included in the first image, which is derived based on a correlation between a first image feature amount of the first image in which the counting objects appear and a first sample image feature amount of a first sample image representing one unit of the counting objects, and a first correct answer density map relating to the number of correct answers of the counting objects included in the first image; a second error represented by a loss function for a second image that does not include the target to be counted; a learning unit that learns a learning model, the learning unit receiving the first image and the first sample image as input and outputting information about an estimated value of the number of the objects to be counted included in the first image, so as to minimize the total error of An information processing device comprising:

Images