Image processing apparatus, image processing method, and program

Abstract

Embodiments of this disclosure relate to an image processing apparatus, an image processing method, and a program. According to this disclosure, the image processing apparatus generates a feature map based on an image, determines whether the value of each cell in the feature map is equal to or greater than a lower threshold, calculates an activation value by inputting the value of each cell in the feature map that is equal to or greater than the lower threshold into an activation function, and generates an existence probability map indicating the distribution of the probability of the presence of objects in the image using the activation values.

Description

Cross-reference to related applications

[0001] The disclosure of Japanese Patent Application No. 2025-013383, filed on January 29, 2025, including the specification, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to image processing apparatus, image processing methods, and programs. Background Technology

[0003] The technologies listed below have been disclosed.

[0004] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2024-023030

[0005] A technique for obtaining information from images using neural networks has been developed. For example, Patent Document 1 discloses a semiconductor device capable of reducing the time required for image processing via neural networks. The semiconductor device of Patent Document 1 includes a memory for weight parameters to prevent the transfer of weight parameters from becoming a bottleneck in the neural network processing the image. Summary of the Invention

[0006] In Patent Document 1, it is assumed that a memory for the weighting parameters can be provided. In view of this, embodiments to be described later have been made, and other problems and novel features will be apparent from the description and drawings herein.

[0007] When calculating activation values ​​for a feature map obtained from an image, the image processing apparatus according to an embodiment calculates activation values ​​for cells having values ​​equal to or greater than a lower threshold. The image processing apparatus then uses the calculated activation values ​​to generate an existence probability map indicating the distribution of the probability of the presence of objects in the image.

[0008] According to an embodiment, a new technique for processing images by using neural networks is provided. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an overview of the operation of an image processing device.

[0010] Figure 2 This is a block diagram illustrating the functional configuration of an image processing device.

[0011] Figure 3 This is a block diagram illustrating the hardware configuration of a computer that implements an image processing device.

[0012] Figure 4 This is the first diagram illustrating the configuration of the neural network.

[0013] Figure 5 This is the second diagram illustrating the configuration of the neural network.

[0014] Figure 6 This is a flowchart illustrating the processing performed by an image processing device.

[0015] Figure 7 This is a diagram illustrating the configuration of an image processing device with an output unit. Detailed Implementation

[0016] In the following, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In the drawings, like or corresponding elements are indicated by the same reference numerals, and redundant descriptions are omitted as necessary for clarity of description. Furthermore, unless otherwise described, predetermined values ​​such as predetermined values ​​and thresholds are pre-stored in storage devices or the like that accessible from devices using those values. Furthermore, unless otherwise described, a storage unit includes any number of storage devices among one or more storage devices.

[0017] <Overview>

[0018] Figure 1 This is a diagram illustrating an overview of the operation of the image processing device 2000. Here, Figure 1 This diagram is intended to facilitate understanding of the overview of the image processing device 2000, and the operation of the image processing device 2000 is not limited to... Figure 1 The operation is illustrated in the figure.

[0019] Image processing device 2000 generates an existence probability map 80 from image 10 using neural network 60. Neural network 60 is configured to output the existence probability map 80 in response to input from image 10. The existence probability map 80 indicates the probability of the existence of an object in each of a plurality of partial regions in image 10. Here, the existence probability only needs to be a value indicating the degree of likelihood of existence, and does not need to be a value indicating probability. Therefore, the existence probability can take a value greater than 1.

[0020] The neural network 60 includes at least a feature extraction layer 50 and an activation function 70. An image 10 or a feature map generated from image 10 is input to the feature extraction layer 50. The feature map is a tensor indicating the amount of features in each unit. The feature extraction layer 50 generates a feature map 20 from the input image or feature map. The feature extraction layer 50 includes, for example, convolutional layers, pooling layers, etc.

[0021] Activation function 70 is an activation function applied to the units 22 constituting feature map 20. The activation function can be of any type. Image processing device 2000 generates activation map 40 from feature map 20 using activation function 70.

[0022] Here, the image processing device 2000 applies the activation function 70 only to units 22 that meet specific conditions. Specifically, the image processing device 2000 performs a process to determine whether the value of each unit in the feature map 20 is equal to or greater than a lower threshold (hereinafter referred to as threshold determination process). Through the threshold determination process, units with values ​​equal to or greater than the lower threshold are designated in the feature map 20. In the following, the units in the feature map 20 with values ​​equal to or greater than the lower threshold are also referred to as target units 30.

[0023] The image processing device 2000 inputs the value of each target unit 30 into the activation function 70 to obtain the activation value of each target unit 30, thereby generating an activation map 40. In the activation map 40, when the unit 22 at position (i, j) of the feature map 20 is a target unit 30, the unit at position (i, j) of the activation map 40 indicates the activation value calculated for the target unit 30. Furthermore, when the unit 22 at position (i, j) of the feature map 20 is not a target unit 30, the unit at position (i, j) of the activation map 40 indicates a predetermined value such as 0.

[0024] For example, in Figure 1 In the example, five target units 30 are specified from feature map 20. As a result, activation values ​​a1 to a5 are obtained from these five target units 30. Therefore, activation map 40 indicates the activation values ​​a1 to a5 for the five units corresponding to these five target units 30. Note that in Figure 1 In the activation diagram 40, the values ​​of cells other than those corresponding to the target cell 30 are omitted.

[0025] The image processing device 2000 generates an existence probability map 80 by using activation values ​​calculated for each target unit 30. For example, the image processing device 2000 directly uses the activation map 40 as the existence probability map 80. Alternatively, for example, the image processing device 2000 generates the existence probability map 80 by further processing the activation map 40.

[0026] <Example of operation effect>

[0027] According to the image processing apparatus 2000, activation values ​​are calculated for target units 30 that are units 22 constituting a feature map 20 obtained from the image 10 and have values ​​equal to or greater than a lower threshold. Then, an existence probability map 80 is generated using the calculated activation values. As described above, the image processing apparatus 2000 provides a novel technique for processing images using a neural network.

[0028] Here, calculating activation values ​​only for units 22 that have values ​​equal to or greater than the lower threshold, rather than calculating activation values ​​for all units 22, has the advantage of reducing the number of activation value calculations. This reduction in the number of activation value calculations then leads to a reduction in the time and computer resources required to calculate the activation values. As a result, according to the image processing device 2000, it is possible to reduce the time and computer resources required to calculate the spatial distribution of the object's existence probability.

[0029] Furthermore, by appropriately setting a lower threshold, it is possible to prevent a decrease in the accuracy of object detection. Therefore, according to the image processing device 2000, it is possible to accurately calculate the spatial distribution of the object's presence probability while reducing the time and computer resources required to calculate the spatial distribution of the object's presence probability. Note that the specific method for setting the lower threshold will be described later.

[0030] The image processing apparatus 2000 of this embodiment will be described in more detail below.

[0031] <Example of feature configuration>

[0032] Figure 2 This is a block diagram illustrating the functional configuration of an image processing device 2000. The image processing device 2000 includes a first generation unit 2020, a determination unit 2040, a calculation unit 2060, and a second generation unit 2080. The first generation unit 2020 acquires an image 10 and generates a feature map 20 based on the image 10. The first generation unit 2020 is implemented using a feature extraction layer 50. The determination unit 2040 determines whether the value of each cell 22 in the feature map 20 is equal to or greater than a lower threshold. The calculation unit 2060 applies the values ​​of target cells 30 (cells 22 with values ​​equal to or greater than the lower threshold) to an activation function 70 to calculate an activation value for each target cell 30. The second generation unit 2080 generates an existence probability map 80 using the activation values ​​calculated for each target cell 30.

[0033] <Example of hardware configuration>

[0034] For example, each functional configuration unit of the image processing device 2000 is implemented by hardware (e.g., hardwired electronic circuits, etc.) that implements each functional configuration unit. Furthermore, for example, each functional configuration unit of the image processing device 2000 is implemented by a combination of hardware and software (e.g., a combination of electronic circuits and a program for controlling the electronic circuits).

[0035] Figure 3This is a block diagram illustrating the hardware configuration of a computer 1000 that implements the image processing device 2000. The computer 1000 can be any computer. For example, the computer 1000 is a stationary computer such as a personal computer (PC) or a server machine. Alternatively, the computer 1000 can be a portable computer such as a smartphone or tablet terminal. Furthermore, the computer 1000 can be an integrated circuit such as a system-on-a-chip (SoC). The computer 1000 can be a dedicated computer designed to implement the image processing device 2000, or it can be a general-purpose computer.

[0036] For example, by installing a predetermined application in computer 1000, each function of image processing device 2000 is implemented in computer 1000. This application is configured by a program for configuring each function of image processing device 2000. Note that the program can be obtained using any method. For example, the program can be obtained from a storage medium (Digital Multifunction Disc (DVD), Universal Serial Bus (USB) memory, etc.) containing the program. Alternatively, for example, the program can be obtained by downloading it from a server device that manages the storage device containing the program.

[0037] Computer 1000 includes processor 1040, bus 1020, memory 1060, storage device 1080, input / output interface (I / F) 1100, and network interface 1120. Bus 1020 is a data transmission path through which processor 1040, memory 1060, storage device 1080, I / O interface 1100, and network interface 1120 send and receive data to each other. However, the method of interconnecting processor 1040 and the like is not limited to bus connection.

[0038] Processor 1040 can be any of various processors such as a microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), and field-programmable gate array (FPGA). Memory 1060 is a main storage device implemented using random access memory (RAM), etc. Storage device 1080 is an auxiliary storage device implemented using read-only memory (ROM), flash memory, memory cards, etc. Input / output interface 1100 is an interface for connecting computer 1000 to input / output devices. Network interface 1120 is an interface for connecting computer 1000 to a network.

[0039] Storage device 1080 stores programs for implementing each functional configuration unit of image processing device 2000 (programs for implementing the above applications). Processor 1040 loads the programs into memory 1060 and executes the programs to implement each functional configuration unit of image processing device 2000.

[0040] The image processing device 2000 can be implemented by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same and can be different from each other.

[0041] <Neural Network 60 Configuration>

[0042] As described above, the neural network 60 includes at least a feature extraction layer 50 and an activation function 70. Any configuration can be applied as a specific configuration of the neural network 60 described above. Here, some specific configurations of the neural network 60 will be illustrated with examples.

[0043] <<Example 1>>

[0044] Figure 4 This is the first diagram illustrating the configuration of neural network 60. Figure 4 In the neural network 60, there are feature extraction layers 50 and activation functions 70.

[0045] The first generation unit 2020 inputs image 10 into the feature extraction layer 50. The feature extraction layer 50 outputs feature map 20 in response to the input of image 10.

[0046] Determining unit 2040 determines whether the value of each unit 22 of the feature map 20 output from feature extraction layer 50 is equal to or greater than a lower threshold. Calculation unit 2060 calculates the activation value for each target unit 30 by inputting each target unit 30, which is a unit having a value equal to or greater than the lower threshold, into activation function 70.

[0047] The calculation unit 2060 generates an activation map 40 using the calculated activation values. The activation map 40 at least indicates the activation values ​​obtained from the target unit 30. Figure 4 In the diagram, the cells with dashed patterns in the activation diagram 40 are cells that indicate the activation values ​​obtained from the corresponding target cells 30.

[0048] For example, in feature map 20, it is assumed that cell 22 at position (i, j) is target cell 30. In this case, activating the cell at position (i, j) in feature map 40 indicates the activation value calculated from the target cell 30 at position (i, j). On the other hand, in feature map 20, it is assumed that cell 22 at position (i, j) is not target cell 30. In this case, activating the cell at position (i, j) in feature map 40 indicates a predetermined value (e.g., 0).

[0049] The activation map 40 is input to the second generation unit 2080. The second generation unit 2080 generates the existence probability map 80 by using the input activation values. For example, the second generation unit 2080 directly uses the activation map 40 as the existence probability map 80.

[0050] Here, the image processing device 2000 can generate an existence probability map 80 for each type of object among a variety of types (hereinafter also referred to as classes). For example, suppose image 10 is an image obtained from a camera mounted on a vehicle. In this case, the image processing device 2000 can be used to detect various types of objects such as people, vehicles, buildings, and guardrails.

[0051] As described above, in the case of generating an existence probability map 80 for each type of object among multiple types of objects, the neural network 60 includes a feature extraction layer 50 for each type of object. Specifically, the neural network 60 includes a feature extraction layer 50 for generating an existence probability map 80 indicating the existence probability of a person, a feature extraction layer 50 for generating an existence probability map 80 indicating the existence probability of a vehicle, and so on.

[0052] <<Example 2>>

[0053] Figure 5 This is the second diagram illustrating the configuration of neural network 60. Figure 5 In the example, neural network 60 is a neural network based on fully convolutional first-order object detection (FCOS).

[0054] Figure 5 The neural network 60 includes a feature extraction layer 90, a classification branch 110, and a centerness calculation branch 120.

[0055] Image processing device 2000 inputs image 10 to feature extraction layer 90. Feature extraction layer 90 outputs feature map 100 in response to the input of image 10. Feature map 100 is input to both classification branch 110 and centrality calculation branch 120. Therefore, feature map 100 can be said to represent the distribution of feature quantities obtained from image 10 that are useful to both the task performed by classification branch 110 and the task performed by centrality calculation branch 120.

[0056] In classification branch 110, a class score map 130 is generated from feature map 100. Class score map 130 indicates the spatial distribution of class scores. Class scores represent the probability that an object of a particular class exists in the image region corresponding to each cell of feature map 100 in image 10.

[0057] More specifically, in classification branch 110, when the first generation unit 2020 inputs feature map 100 into feature extraction layer 50-1, feature map 20-1 is output from feature extraction layer 50-1. Feature extraction layer 50-1 is one of multiple feature extraction layers 50 included in neural network 60. Feature map 20-1 is one of multiple feature maps 20 generated by neural network 60.

[0058] Determining unit 2040 designates target cell 30-1 from feature map 20-1 by performing a threshold determination process on feature map 20-1. Target cell 30-1 is the target cell 30 designated in feature map 20-1. More specifically, determining unit 2040 determines whether the value of each cell in feature map 20-1 is equal to or greater than a lower threshold. Determining unit 2040 designates cells in feature map 20-1 with values ​​equal to or greater than the lower threshold as target cells 30-1.

[0059] The computation unit 2060 inputs each target unit 30-1 into the activation function 70-1 to calculate the activation value for each target unit 30-1. The activation function 70-1 is one of a plurality of activation functions 70 included in the neural network 60.

[0060] The computation unit 2060 generates a class score map 130 using activation values ​​obtained from each target unit 30-1. The class score map 130 is an activation map generated using activation values ​​obtained from each target unit 30-1. For example, in the activation map 40, when the unit at position (i, j) in the feature map 20-1 is not a target unit 30-1, the unit at position (i, j) in the class score map 130 indicates a predetermined value (e.g., 0).

[0061] In the centrality calculation branch 120, a centrality map 140 is generated from the feature map 100. The centrality map 140 indicates the centrality value for each cell. The centrality map 140 indicates how close the image region of image 10 corresponding to each cell of feature map 100 is to the center of the object.

[0062] More specifically, in the centrality calculation branch 120, when the first generation unit 2020 inputs feature map 100 into feature extraction layer 50-2, feature map 20-2 is output from feature extraction layer 50-2. Feature extraction layer 50-2 is one of multiple feature extraction layers 50 included in neural network 60. Feature map 20-2 is one of multiple feature maps 20 generated by neural network 60.

[0063] Determining unit 2040 designates target unit 30-2 from feature map 20-2 by performing a threshold determination process on feature map 20-2. Target unit 30-2 is the target unit 30 designated in feature map 20-2. More specifically, determining unit 2040 determines whether the value of each unit in feature map 20-2 is equal to or greater than a lower threshold. Determining unit 2040 designates units in feature map 20-2 with values ​​equal to or greater than the lower threshold as target units 30-2.

[0064] The computation unit 2060 inputs each target unit 30-2 into the activation function 70-2 to calculate the activation value for each target unit 30-2. The activation function 70-2 is one of a plurality of activation functions 70 included in the neural network 60.

[0065] The computation unit 2060 generates a centrality map 140 using activation values ​​obtained from each target cell 30-2. The centrality map 140 is an activation map generated using activation values ​​obtained from each target cell 30-2. For example, in activation map 140, when the cell at position (i, j) in feature map 20-2 is not a target cell 30-2, the cell at position (i, j) in centrality map 140 indicates a predetermined value (e.g., 0).

[0066] The second generation unit 2080 generates the existence probability map 80 by using the class fraction map 130 and the centrality map 140. For example, the second generation unit 2080 generates the existence probability map 80 by calculating the cell-by-cell product between the class fraction map 130 and the centrality map 140.

[0067] The operation of calculating the cell-by-cell product between two graphs refers to the operation of calculating the product between cells at the same position. For example, in the existence probability graph 80, the value of the cell at position (i, j) indicates the product of the value of the cell at position (i, j) in the class fraction graph 130 and the value of the cell at position (i, j) in the centrality graph 140.

[0068] As described above, the image processing device 2000 can generate an existence probability map 80 for each of the multiple classes. In generating the existence probability map 80 for each of the multiple classes, the neural network 60 generates a class score map 130 for each class. For this purpose, a feature extraction layer 50-1 is prepared for each class. For each class, the second generation unit 2080 calculates a unit-by-unit product between the class score map 130 generated for that class and the centrality map 140 to generate the existence probability map 80 for that class.

[0069] <Processing Procedure>

[0070] Figure 6This is a flowchart illustrating the processing procedure performed by the image processing device 2000. First generation unit 2020 acquires image 10 (S102). First generation unit 2020 generates feature map 20 based on image 10 (S104). Determination unit 2040 determines whether the value of each cell in feature map 20 is equal to or greater than a lower threshold, to designate target cells 30 (each cell 22 having a value equal to or greater than the lower threshold) (S106). Calculation unit 2060 calculates activation values ​​for target cells 30 (S108). Second generation unit 2080 generates presence probability map 80 based on the activation values ​​calculated for each target cell 30 (S110).

[0071] <Image 10 Acquisition: S102>

[0072] The first generation unit 2020 acquires image 10 (S102). Various methods can be used to acquire the image to be processed. For example, image 10 is stored in a storage unit accessible from the image processing device 2000. The first generation unit 2020 acquires image 10 stored in the storage unit.

[0073] For example, the user of the image processing device 2000 specifies the image to be processed as image 10 by specifying a filename, etc. The first generation unit 2020 retrieves the image specified by the user as image 10 from the storage unit.

[0074] Alternatively, for example, image 10 is sent to image processing device 2000 from a device other than image processing device 2000. In this case, the first generation unit 2020 receives the image sent from the other device and processes the image into image 10.

[0075] Any device can send image 10 to image processing device 2000. For example, the device could be a camera that has already generated image 10. In this case, for example, image processing device 2000 detects objects from image 10 generated by the camera. For example, if the camera is set up in a vehicle, it captures the situation around the vehicle in image 10. Therefore, by generating an existence probability map 80 from image 10 using image processing device 2000, it is possible to determine the distribution of the probability of the existence of objects around the vehicle.

[0076] <Generating Feature Map 20: S104>

[0077] The first generation unit 2020 generates a feature map 20 based on the image 10 using the feature extraction layer 50 (S104). As described above, for example, the feature extraction layer 50 is configured to generate the feature map 20 in response to the input of the image (see [link]). Figure 4In this case, the first generation unit 2020 obtains the feature map 20 from the feature extraction layer 50 by inputting the image 10 into the feature extraction layer 50.

[0078] Furthermore, for example, feature extraction layer 50 is configured to generate feature map 20 in response to input of the feature map (see [link]). Figure 5 In this case, the first generation unit 2020 obtains the feature map 20 by inputting the feature map generated from the image 10 into the feature extraction layer 50.

[0079] <Threshold determination process: S106>

[0080] Determining unit 2040 determines whether the value of each cell 22 in feature map 20 is equal to or greater than a lower threshold, in order to designate target cell 30 (S106). When the value of cell 22 is equal to or greater than the lower threshold, determining unit 2040 designates cell 22 as target cell 30. On the other hand, when the value of cell 22 is not equal to or greater than the lower threshold, determining unit 2040 does not designate cell 22 as target cell 30.

[0081] <<Methods for Determining the Lower Threshold>>

[0082] The lower limit threshold can be determined in various ways. For example, the lower limit threshold can be manually determined by the administrator or user of the image processing device 2000.

[0083] Furthermore, for example, the lower threshold can be determined based on its relationship with the activation value as follows. Here, the lower threshold is preferably determined such that the activation value obtained from the target unit 30 becomes relatively large to some extent. That is, preferably, the lower threshold is determined such that the unit 22 with a relatively large activation value is designated as the target unit 30, while the unit 22 with a relatively small activation value is not designated as the target unit 30.

[0084] Therefore, firstly, for example, a threshold (hereinafter referred to as the second threshold) is determined for the activation values ​​obtained using activation function 70. The second threshold is, for example, a threshold used to determine whether an object exists (hereinafter referred to as the probability threshold). The lower limit threshold is calculated by inputting the second threshold into the inverse function of activation function 70. For example, in the case where activation function 70 is a sigmoid function, the inverse function is a logit function.

[0085] Specifically, the lower limit threshold Th is calculated using the second threshold T2 as follows.

[0086] [Mathematical Formula 1]

[0087]

[0088] Here, f^-1() denotes the inverse function of activation function 70. Therefore, f() denotes activation function 70.

[0089] For example, such as Figure 4 As illustrated in the diagram, it is assumed that activation map 40 is directly used as presence probability map 80. In this case, the aforementioned probability threshold can be used as the second threshold T2. For example, in presence probability map 80, it is assumed that image regions of image 10 corresponding to units with values ​​of 0.8 or greater are considered as image regions containing objects. In this case, the probability threshold is 0.8. Therefore, 0.8 is used as T2.

[0090] Here, a function that does not have an inverse function over its entire domain can also be used as the activation function 70. In this case, the inverse function f⁻¹() is defined by extracting the range that defines the inverse function from the activation function 70, and the second threshold is included in this range. Then, the lower limit threshold is calculated from the second threshold using the inverse function f⁻¹().

[0091] <Generation of probability graph 80: S110>

[0092] The second generation unit 2080 generates an existence probability map 80 (S108) by using the activation values ​​calculated for each target unit 30. For example, as in the reference Figure 4 In the described example, the second generation unit 2080 can directly use the activation graph 40 as the existence probability graph 80. Additionally, for example, as in the reference... Figure 5 In the example described, the second generation unit 2080 generates an existence probability map 80 from two activation maps (class score map 140 and centrality map 140).

[0093] <Output of Results>

[0094] The image processing device 2000 outputs information indicating the processing result (hereinafter referred to as output information). The functional configuration unit for outputting output information is called the output unit. Figure 7 This is a diagram illustrating the configuration of an image processing device 2000 with an output unit.

[0095] The output information can include any kind of information. For example, output unit 2100 generates output information including an existence probability map 80. Here, when an existence probability map 80 is generated for each of the multiple classes, the output information can include an existence probability map 80 for each class.

[0096] Note that by using the existence probability map 80 obtained for each class, data (e.g., a vector) indicating the existence probability of objects of each class is obtained for each region of the image 10 corresponding to each cell of the existence probability map 80. For example, suppose an existence probability map 80 is generated for each of the three types of classes C1, C2, and C3. In this case, the image processing device 2000 can obtain a vector (L1[i,j], L2[i,j], L3[i,j]), where the existence probability L1[i,j] of objects of class C1, the existence probability L2[i,j] of objects of class C2, and the existence probability L3[i,j] of objects of class C3 are listed for each position (i,j).

[0097] Any mode can be applied to the output mode of the output information. For example, output unit 2100 can store the output information in any storage unit. Furthermore, for example, output unit 2100 can output the output information to any display device so that the display device displays the output information. Additionally, for example, output unit 2100 can send the output information to another device.

[0098] Training a Neural Network 60

[0099] The neural network 60 is pre-trained. Training of the neural network 60 is performed using multiple training samples. The training samples are, for example, a pair of training images and a ground reality map indicating the spatial distribution of the probability of the presence of objects in the training images. The ground reality map indicates the ideal probability map 80 to be output from the neural network 60 in response to the input of the training images.

[0100] The device that performs training on the neural network 60 (hereinafter referred to as the training device) calculates the loss using an existence probability map 80 obtained by inputting training images into the neural network 60 and a ground truth map. Furthermore, the training device updates the trainable parameters (weights, biases, etc.) included in the neural network 60 using the calculated loss. The training device trains the neural network 60 by repeating the parameter updates using multiple training samples.

[0101] In the foregoing description, this disclosure has been described with reference to embodiments, but this disclosure is not limited to the embodiments described above. Various modifications that can be understood by those skilled in the art can be made to the configurations and details of this disclosure within the scope of this disclosure. In addition, each embodiment can be appropriately combined with other embodiments.

[0102] Each figure is merely an illustration provided to depict one or more embodiments. Each figure is not necessarily associated with only one specific embodiment, but may be associated with one or more other embodiments. As will be understood by those skilled in the art, various features or steps described in any of the figures may be combined with features or steps illustrated in one or more other figures, for example, to produce embodiments not explicitly illustrated or described. Not all features or steps illustrated in any of the figures used to describe embodiments are necessarily essential, and some features or steps may be omitted. The order of steps described in any figure may be appropriately changed.

[0103] In this disclosure, a program includes a set of instructions (or software code) for causing the computer to perform one or more functions described in the embodiments when the program is loaded into a computer. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example, and not limitation, the computer-readable medium or tangible storage medium may include random access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs) or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray discs or other optical disc storage devices, cassette tapes, magnetic tapes, disk storage devices, or other magnetic storage devices. The program may be transmitted on a transient computer-readable medium or a communication medium. By way of example, and not limitation, the transient computer-readable medium or communication medium may include electrical, optical, acoustic, or other forms of propagation signals.

[0104] <Supplementary Notes>

[0105] (Supplementary Note 1)

[0106] An image processing device, comprising:

[0107] A first generation unit is configured to generate feature maps based on an image;

[0108] A determining unit is configured to determine whether the value of each cell in the feature map is equal to or greater than a lower threshold.

[0109] A calculation unit, configured to calculate an activation value by inputting a value equal to or greater than the lower threshold into an activation function; and

[0110] A second generation unit is configured to generate an existence probability map by using the activation value, the existence probability map indicating the distribution of the existence probability of objects in the image.

[0111] (Supplementary Note 2)

[0112] According to the image processing device described in Supplementary Note 1

[0113] The existence probability diagram indicates that:

[0114] The activation value of the cell corresponding to the cell in the feature map that has a value equal to or greater than the lower threshold; and

[0115] A predetermined value for the cell corresponding to the cell in the feature map that has a value less than the lower threshold.

[0116] (Supplementary Note 3)

[0117] According to the image processing device described in Supplementary Note 1

[0118] The first generation unit generates a first feature map and a second feature map in the feature map.

[0119] The determining unit is configured as follows:

[0120] Determine whether the value of each cell in the first feature map of the feature map is equal to or greater than the first lower limit threshold in the lower limit threshold; and

[0121] Determine whether the value of each cell in the second feature map of the feature map is equal to or greater than the second lower limit threshold in the lower limit threshold, and

[0122] The second generation unit is configured as follows:

[0123] A class graph is generated by using the activation values ​​calculated for the first feature graph in the feature graph, which have values ​​equal to or greater than the first lower threshold in the lower threshold. The class graph indicates the distribution of the probability of the existence of objects of a particular class.

[0124] A centrality map, indicating the distribution of proximity to the center of an object, is generated by using activation values ​​calculated for the units of the second feature map in the feature map that have a value equal to or greater than the second lower threshold.

[0125] The existence probability graph of the object for the specific class is generated by using the class graph and the centrality graph.

[0126] (Supplementary Note 4)

[0127] The image processing device described in Supplementary Note 3

[0128] The class diagram therein indicates:

[0129] The activation value of the cell corresponding to the cell in the first feature map of the feature map that has a value equal to or greater than the first lower limit threshold; and

[0130] A first predetermined value for a cell corresponding to a cell in the first feature map of the feature map that has a value less than the first lower threshold value in the lower threshold, and

[0131] The centrality map indicates that:

[0132] The activation value of the cell corresponding to the cell in the first feature map of the feature map that has a value equal to or greater than the second lower limit threshold in the lower limit threshold; and

[0133] A second predetermined value for a cell corresponding to a cell in the second feature map of the feature map that has a value less than the second lower limit threshold in the lower limit threshold.

[0134] (Supplementary Note 5)

[0135] According to the image processing device described in Supplementary Note 1

[0136] The lower limit threshold is obtained by inputting a probability threshold used to determine whether an object exists into the inverse function of the activation function.

[0137] (Supplementary Note 6)

[0138] An image processing method, executed by a computer, the image processing method comprising:

[0139] The first generation step based on image feature map generation;

[0140] The step of determining whether the value of each cell in the feature map is equal to or greater than a lower threshold.

[0141] The calculation step involves inputting the value of the unit, which is equal to or greater than the lower threshold, into the activation function to calculate the activation value; and

[0142] A second generation step involves generating an existence probability map using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

[0143] (Supplementary Note 7)

[0144] A program for causing a computer to perform a method, the method comprising:

[0145] The first generation step based on image feature map generation;

[0146] The step of determining whether the value of each cell in the feature map is equal to or greater than a lower threshold.

[0147] The calculation step involves inputting the value of the unit, which is equal to or greater than the lower threshold, into the activation function to calculate the activation value; and

[0148] A second generation step involves generating an existence probability map using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

[0149] (Supplementary Note 8)

[0150] An image processing device, comprising:

[0151] One or more storage elements, in which instructions are stored; and

[0152] One or more processors,

[0153] The one or more processors thereon execute the instructions to:

[0154] Feature maps are generated from images;

[0155] Determine whether the value of each cell in the feature map is equal to or greater than the lower threshold.

[0156] The activation value is calculated by inputting the value of the unit that is equal to or greater than the lower threshold into the activation function; and

[0157] An existence probability map is generated by using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

[0158] (Supplementary Note 9)

[0159] A non-transitory computer-readable storage medium storing a program for causing a computer to perform a method, the method comprising:

[0160] The first generation step based on image feature map generation;

[0161] The step of determining whether the value of each cell in the feature map is equal to or greater than a lower threshold.

[0162] The calculation step involves inputting the value of the unit, which is equal to or greater than the lower threshold, into the activation function to calculate the activation value; and

[0163] A second generation step involves generating an existence probability map using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

[0164] Some or all of the elements described in Supplementary Notes 2 to 5 that depend on Supplementary Note 1 (e.g., configuration and functionality) may also depend on each of Supplementary Notes 6 to 9 through the same dependencies as those in Supplementary Notes 2 to 5. Some or all of the elements described in any of the supplementary notes may be applied to various hardware, software, recording units for recording software, systems, and methods.

Claims

1. An image processing apparatus, comprising: A first generation unit is configured to generate feature maps based on an image; A determining unit is configured to determine whether the value of each cell in the feature map is equal to or greater than a lower threshold. A calculation unit configured to calculate an activation value by inputting a value of the unit that is equal to or greater than the lower threshold into an activation function; as well as A second generation unit is configured to generate an existence probability map by using the activation value, the existence probability map indicating the distribution of the existence probability of objects in the image.

2. The image processing apparatus according to claim 1, The existence probability diagram indicates that: The activation value of the cell corresponding to the cell in the feature map that has a value equal to or greater than the lower threshold; and A predetermined value for the cell corresponding to the cell in the feature map that has a value less than the lower threshold.

3. The image processing apparatus according to claim 1, The first generation unit generates a first feature map and a second feature map in the feature map. The determining unit is configured as follows: Determine whether the value of each cell in the first feature map of the feature map is equal to or greater than the first lower limit threshold in the lower limit threshold; and Determine whether the value of each cell in the second feature map of the feature map is equal to or greater than the second lower limit threshold in the lower limit threshold, and The second generation unit is configured as follows: A class graph is generated by using the activation values ​​calculated for the first feature graph in the feature graph, which have values ​​equal to or greater than the first lower threshold in the lower threshold. The class graph indicates the distribution of the probability of the existence of objects of a particular class. A centrality map, indicating the distribution of proximity to the center of an object, is generated by using activation values ​​calculated for the units of the second feature map in the feature map that have a value equal to or greater than the second lower threshold. The existence probability graph of the object for the specific class is generated by using the class graph and the centrality graph.

4. The image processing apparatus according to claim 3, The class diagram therein indicates: The activation value for the cell corresponding to the cell in the first feature map of the feature map that has a value equal to or greater than the first lower limit threshold in the lower limit threshold; as well as A first predetermined value for a cell corresponding to a cell in the first feature map of the feature map that has a value less than the first lower threshold value in the lower threshold, and The centrality map indicates that: The activation value for the cell corresponding to the cell in the first feature map of the feature map that has a value equal to or greater than the second lower limit threshold in the lower limit threshold; as well as A second predetermined value for a cell corresponding to a cell in the second feature map of the feature map that has a value less than the second lower limit threshold in the lower limit threshold.

5. The image processing apparatus according to claim 1, The lower limit threshold is obtained by inputting a probability threshold used to determine whether an object exists into the inverse function of the activation function.

6. An image processing method, executed by a computer, the image processing method comprising: The first generation step based on image feature map generation; The step of determining whether the value of each cell in the feature map is equal to or greater than a lower threshold. The calculation step involves inputting the value of the unit, which is equal to or greater than the lower threshold, into the activation function to calculate the activation value; as well as A second generation step involves generating an existence probability map using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

7. A computer-readable storage medium storing a program for causing a computer to perform a method, the method comprising: The first generation step based on image feature map generation; The step of determining whether the value of each cell in the feature map is equal to or greater than a lower threshold. The calculation step involves inputting the value of the unit, which is equal to or greater than the lower threshold, into the activation function to calculate the activation value; as well as A second generation step involves generating an existence probability map using the activation values, the existence probability map indicating the distribution of the probability of the presence of objects in the image.

Images