Image processing equipment, image processing methods and programs

Abstract

This disclosure relates to image processing apparatus, image processing methods, and programs. The image processing apparatus according to this disclosure generates a second image by performing quantization processing on a first image, generates a feature map based on the second image, generates an existence probability map indicating the probability distribution of the presence of objects in the first image using the feature map, performs a matching determination process, which determines whether the value of each cell matches a fixed value in the feature map or the existence probability map, and performs dequantization processing on the feature map or the existence probability map based on the result of the matching determination process.

Description

Cross-references to related applications

[0001] The disclosure of Japanese Patent Application No. 2025-013384, filed on January 29, 2025, including the specification, drawings and abstract, is incorporated herein by reference in its entirety. Technical Field

[0002] This disclosure relates to an image processing apparatus, an image processing method, and a program. Background Technology

[0003] The technologies listed below have been disclosed.

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2024-023030

[0005] A technique for obtaining information from images using neural networks has been developed. For example, Patent Document 1 discloses a semiconductor device capable of reducing the time required for neural networks to process images. The semiconductor device of Patent Document 1 includes a memory for weight parameters to prevent the transmission of weight parameters from becoming a bottleneck in the neural network processing images. Summary of the Invention

[0006] In Patent Document 1, it is assumed that a memory for the weighting parameters can be provided. The embodiments described later are made in view of this situation, and other issues and novel features will be apparent from the description and drawings in this specification.

[0007] The image processing apparatus according to an embodiment quantizes an image and dequantizes a feature map obtained from the quantized image or an existence probability map generated using the feature map. It determines whether the value of a unit matches a fixed value and performs dequantization based on the determination result.

[0008] According to an embodiment, a new technique for processing images by using a neural network is provided. Attached Figure Description

[0009] Figure 1 This is a diagram illustrating an overview of the operation of an image processing device.

[0010] Figure 2 It is a block diagram illustrating the functional configuration of an image processing device.

[0011] Figure 3 This is a block diagram illustrating the hardware configuration of a computer that implements an image processing device.

[0012] Figure 4 The first diagram illustrates the configuration of the neural network.

[0013] Figure 5This is the second diagram illustrating the configuration of the neural network.

[0014] Figure 6 This is a first flowchart illustrating the process performed by an image processing device.

[0015] Figure 7 This is a second flowchart illustrating the process performed by an image processing device.

[0016] Figure 8 The diagram illustrates the dequantization function determined based on the dequantization threshold and the observed maximum value.

[0017] Figure 9 The illustration shows the case where the feature map is treated as the target map. Figure 4 A more specific configuration of the neural network illustrated.

[0018] Figure 10 The illustration shows the case where the feature map is regarded as the target map. Figure 5 A more specific configuration of the neural network illustrated.

[0019] Figure 11 This is a diagram illustrating the configuration of an image processing device with an output unit. Detailed Implementation

[0020] In the following description, embodiments of this disclosure will be described in detail with reference to the accompanying drawings. In the drawings, identical or corresponding elements are indicated by the same reference numerals, and redundant descriptions are omitted where necessary for clarity. Furthermore, unless otherwise described, predetermined values ​​(such as predetermined values ​​and thresholds) are pre-stored in storage devices or the like that accessible from devices using these values. Additionally, unless otherwise described, storage units include any number of one or more storage devices.

[0021] <Overview>

[0022] Figure 1 This is a diagram illustrating an overview of the operation of the image processing device 2000. Here, Figure 1 This diagram is provided to facilitate understanding of the image processing device 2000, and the operation of the image processing device 2000 is not limited to... Figure 1 The operation is illustrated in the diagram.

[0023] Image processing device 2000 generates an existence probability map 80 from image 10 using neural network 60. Neural network 60 is configured to output the existence probability map 80 in response to input image. The existence probability map 80 indicates the probability of an object's presence in each of multiple partial regions of the input image. Here, the existence probability only needs to be a value indicating the degree of likelihood of presence, not necessarily a value indicating probability. Therefore, the existence probability can take a value greater than 1.

[0024] The quantized image 12, obtained by performing quantization processing on image 10, is input into neural network 60. Therefore, image processing device 2000 performs quantization processing on image 10 to generate quantized image 12. In the quantization processing, the value of each pixel in image 10 is quantized. Therefore, quantized image 12 is an image in which the value of each pixel in image 10 has been quantized.

[0025] Here, the range of pixel values ​​in quantized image 12 may differ from the range of pixel values ​​in image 10. This is because the pixel values ​​in quantized image 12 and image 10 may be represented by different data types. For example, the pixel values ​​in image 10 may be represented by 32-bit floating-point numbers, while the pixel values ​​in quantized image 12 may be represented by 8-bit integer numbers. Therefore, the pixel values ​​in image 10 are quantized to match the range of pixel values ​​in quantized image 12. In the following text, the data type of the pixel values ​​in image 10 is referred to as the first data type, and the data type of the pixel values ​​in quantized image 12 is referred to as the second data type.

[0026] For example, the second data type is the same as the data type of the weights or biases (hereinafter referred to as weights, etc.) of the neural network 60. In this case, the quantization processing performed on the image 10 is also performed on the weights, etc. of the neural network 60 beforehand. In this way, the data type of the pixel values ​​of the quantized image 12 input to the neural network 60 and the data type of the weights, etc. of the neural network 60 both become the second data type.

[0027] Note that image 10 can be an image generated by a camera (hereinafter referred to as the source image), or it can be an image generated by performing various preprocessing operations on the source image. For example, in the latter case, the source image is an image in which each pixel is represented by an 8-bit integer from each of the R, G, and B channels. By performing preprocessing on the source image, image 10 is generated in which each pixel is represented by a 32-bit floating-point number.

[0028] The neural network 60 includes at least a feature extraction layer 50. A quantized image 12 or a feature map generated from the quantized image 12 is input to the feature extraction layer 50. The feature map is a tensor indicating the amount of features in each unit. The feature extraction layer 50 generates a feature map 20 from the input quantized image 12 or feature map. The feature extraction layer 50 includes, for example, convolutional layers, pooling layers, etc.

[0029] The image processing device 2000 generates an existence probability map 80 from the feature map 20. The specific method for generating the existence probability map 80 from the feature map 20 will be described later.

[0030] Here, the image processing device 2000 performs dequantization processing on each cell of the specific image (hereinafter referred to as the target image). The target image is, for example, the feature image 20 or the existence probability image 80.

[0031] The data type of the dequantized data is called the third data type. The third data type is at least different from the second data type. The third data type can be the same as or different from the first data type. When the third data type is the same as the first data type, dequantization is performed so that the data type of the cell in probability graph 80 matches the numeric type of the cell in graph 10. In the following text, the characters x, y, and z are used for the sign of the value of the first data type, the sign of the value of the second data type, and the sign of the value of the third data type, respectively.

[0032] Here, as a dequantization method, there exists a way to map the range of values ​​that a data type before dequantization can take to the range of values ​​that a data type after dequantization can take. For example, suppose the range of values ​​for a second data type before dequantization is Y_min or greater and Y_max or less. Also suppose the range of values ​​for a third data type after dequantization is Z_min or greater and Z_max or less. In this case, a linear function, etc., that maps the range of values ​​for Y_min or greater and Y_max or less to the range of values ​​for Z_min or greater and Z_max or less is defined as a dequantization function.

[0033] On the other hand, in the image processing device 2000, the conversion between z, the value of the dequantized unit, and y, the value of the unit before dequantization, is defined as a quantization process expressed by the following formula (1). The dequantization process of the image processing device 2000 (i.e., the conversion from y, the value before dequantization, to z, the value after dequantization) is also processed using the correspondence of formula (1).

[0034] [Mathematics 1]

[0035] Here, dq^-1(z) is the quantization function. Th is a predetermined value called the dequantization threshold.

[0036] In formula (1), C is the first fixed value. For example, the first fixed value is zero or the minimum value Y_min in the second data type. For example, the minimum value of the 8-bit integer type is -128. The first fixed value is preferably determined as the value obtained by inputting the dequantized threshold Th into the quantization function dq^-1, that is, it is determined as C=dq^-1(Th).

[0037] For example, dequantization can be performed using the entire range of values ​​that the second data type can take (Y_min or greater and Y_max or less, as mentioned above). In this case, Y_min, which is the minimum value of the second data type, is used as the fixed value C in formula (1).

[0038] Specifically, in the quantization represented by formula (1), the value z of the unit in the third data type that is greater than the dequantized threshold Th and equal to or less than Z_max is mapped to the range that is greater than Y_min and equal to or less than Y_max in the second data type through the quantization function. Therefore, in dequantization, y that is greater than Y_min and equal to or less than Y_max in the second data type is mapped to the range that is greater than Th and equal to or less than Z_max in the third data type. That is, the range of [y_min <= y <= Y_max] is mapped to the range of [Th < z <= Z_max].

[0039] Moreover, in quantization, the value of the unit in the third data type that is equal to or greater than Z_min and equal to or less than Th is mapped to the minimum value Y_min in the second data type. Therefore, when dequantizing y = y_min, z = Th is obtained.

[0040] Here, as will be described later, it is preferable to determine the dequantized threshold Th as the boundary value between important values and unimportant values in the post-dequantization processing. When Th is set in this way, the range of the third data type that is unimportant after dequantization [z <= Th] is mapped to a fixed value C (such as Y_min) in the second data type. Therefore, through the inverse function dq of the quantization function, a wide range in the second data type (such as the range greater than or equal to Y_min and less than or equal to Y_max, which is the entire possible numerical range of the second data type) can be dequantized into the third data type.

[0041] Note that in Figure 1 the example of, the presence probability map 80 is the target map. Therefore, dequantization is performed on the presence probability map 80. Hereinafter, the presence probability map 80 in which the value of each unit is dequantized is also referred to as the dequantized presence probability map 82. [[ID=This is a block diagram illustrating the functional configuration of an image processing device 2000. The image processing device 2000 includes a quantization unit 2020, a first generation unit 2040, a second generation unit 2060, a determination unit 2080, and a dequantization unit 2100. The quantization unit 2020 quantizes the image 10 to generate a quantized image 12. The first generation unit 2040 generates a feature map 20 based on the quantized image 12. The second generation unit 2060 generates an existence probability map 80 using the feature map 20. The determination unit 2080 determines whether the value of each cell in the target image, which is the target of dequantization processing, matches a first fixed value. The dequantization unit 2100 performs dequantization processing on each cell of the target image based on the determination result of the determination unit 2080.

[0046] <Example of hardware configuration>

[0047] For example, each functional configuration unit of the image processing device 2000 is implemented by hardware (e.g., hardwired electronic circuits). Alternatively, each functional configuration unit of the image processing device 2000 is implemented by a combination of hardware and software (e.g., a combination of electronic circuits and a program for controlling the electronic circuits).

[0048] Figure 3 This is a block diagram illustrating the hardware configuration of a computer 1000 that implements the image processing device 2000. The computer 1000 can be any computer. For example, the computer 1000 is an integrated circuit such as a system-on-a-chip (SoC). Alternatively, the computer 1000 can be a fixed computer such as a personal computer (PC) or a server machine. Furthermore, the computer 1000 can be a portable computer such as a smartphone or tablet terminal. The computer 1000 can be a dedicated computer designed for implementing the image processing device 2000, or it can be a general-purpose computer.

[0049] For example, each function of the image processing device 2000 is implemented on the computer 1000 by installing a predetermined application. This application is configured by a program for configuring each function of the image processing device 2000. Note that the program can be obtained using any method. For example, the program can be obtained from a storage medium (Digital Universal Disc (DVD), Universal Serial Bus (USB) memory, etc.). Alternatively, the program can be obtained, for example, by downloading it from a server device that manages the storage device.

[0050] Computer 1000 includes processor 1040, bus 1020, memory 1060, storage device 1080, input / output interface (I / F) 1100, and network interface 1120. Bus 1020 is the data transmission path for processor 1040, memory 1060, storage device 1080, I / O interface 1100, and network interface 1120 to send and receive data with each other. However, the method of connecting processor 1040 and the like to each other is not limited to bus connection.

[0051] Processor 1040 can be any of various processors, such as a microprocessor unit (MPU), central processing unit (CPU), graphics processing unit (GPU), digital signal processor (DSP), and field-programmable gate array (FPGA). Memory 1060 is a main storage device implemented using random access memory (RAM), etc. Storage device 1080 is an auxiliary storage device implemented using read-only memory (ROM), flash memory, memory cards, etc. Input / output interface 1100 is an interface for connecting computer 1000 to input / output devices. Network interface 1120 is an interface for connecting computer 1000 to a network.

[0052] Storage device 1080 stores programs for implementing each functional configuration unit of image processing device 2000 (programs for implementing the above applications). Processor 1040 loads the programs stored in storage device 1080 into memory 1060 and executes the programs to implement each functional configuration unit of image processing device 2000.

[0053] The image processing device 2000 can be implemented by one computer 1000 or by multiple computers 1000. In the latter case, the configurations of the computers 1000 do not need to be the same and can be different from each other.

[0054] <Neural Network 60 Configuration>

[0055] Here, some specific configurations of the neural network 60 will be illustrated. Note that the neural network 60 described here does not include the matching determination process and the dequantization process. The neural network 60 including the matching determination process and the dequantization process will be described later.

[0056] <<Example 1>>

[0057] Figure 4 This is the first diagram illustrating the configuration of neural network 60. Figure 4 In this neural network 60, a feature extraction layer 50 and an activation function 70 are included. The activation function 70 is an activation function applied to each unit constituting the feature map 20. The activation function can be of any type.

[0058] The first generation unit 2040 inputs the quantized image 12 into the feature extraction layer 50. The feature extraction layer 50 outputs a feature map 20 in response to the input of the quantized image 12.

[0059] The second generation unit 2060 generates an existence probability map 80 from the feature map 20. For this purpose, for example, the second generation unit 2060 generates an activation map 40 from the feature map 20 using an activation function 70. Each cell in the activation map 40 indicates the activation value of a cell in the feature map 20 corresponding to that cell. The activation values ​​of the cells in the feature map 20 are obtained from the activation function 70 by inputting the cell values ​​into the activation function 70.

[0060] Furthermore, the second generation unit 2060 generates an existence probability map 80 based on the activation map 40. For example, the second generation unit 2060 directly uses the activation map 40 as the existence probability map 80.

[0061] Here, the image processing device 2000 can generate an existence probability map 80 for each type of object among a variety of types (hereinafter also referred to as categories). For example, suppose image 10 is an image obtained from a camera mounted on a vehicle. In this case, the image processing device 2000 can be used to detect objects of various categories, such as people, vehicles, buildings, and guardrails.

[0062] For example, in generating an existence probability map 80 for each of multiple object types, the neural network 60 includes a feature extraction layer 50 for each object type. Specifically, the neural network 60 includes a feature extraction layer 50 for generating an existence probability map 80 indicating the existence probability of a person, a feature extraction layer 50 for generating an existence probability map 80 indicating the existence probability of a vehicle, and so on.

[0063] <<Example 2>>

[0064] Figure 5 This is the second diagram illustrating the configuration of neural network 60. Figure 5 In the example, neural network 60 is a neural network based on fully convolutional single-level object detection (FCOS).

[0065] Figure 5 The neural network 60 includes a feature extraction layer 90, a classification branch 110, and a centrality calculation branch 120.

[0066] The quantized image 12 output from the quantization unit 2020 is input to the feature extraction layer 90. The feature extraction layer 90 outputs a feature map 100 in response to the input of the quantized image 12. The feature map 100 is input to the classification branch 110 and the centrality calculation branch 120. Therefore, it can be said that the feature map 100 represents the distribution of features obtained from the quantized image 12, which are useful for the tasks performed by the classification branch 110 and the centrality calculation branch 120.

[0067] In classification branch 110, in response to the input of feature map 100, a category score map 130 is output. Category score map 130 indicates the spatial distribution of category scores. Category scores represent the probability that an object of a particular category exists in the image region of image 10 corresponding to each cell of feature map 100.

[0068] More specifically, in classification branch 110, when the first generation unit 2040 inputs feature map 100 to feature extraction layer 50-1, feature map 20-1 is output from feature extraction layer 50-1. Feature extraction layer 50-1 is one of multiple feature extraction layers 50 included in neural network 60. Feature map 20-1 is one of multiple feature maps 20 generated by neural network 60.

[0069] The second generation unit 2060 generates a category score map 130 from the feature map 20-1. The category score map 130 is an activation map obtained from the feature map 20-1. Then, the second generation unit 2060 generates the category score map 130 by calculating the activation value of each unit in the feature map 20-1 using an activation function 70-1. The activation function 70-1 is one of a plurality of activation functions 70 included in the neural network 60.

[0070] In the centrality calculation branch 120, in response to the input of feature map 100, a centrality map 140 is output. Centrality map 140 indicates the centrality value of each cell. Centrality map 140 indicates how close the image region of image 10 corresponding to each cell of feature map 10 is to the center of the object.

[0071] More specifically, in the centrality calculation branch 120, when the first generation unit 2040 inputs feature map 100 to feature extraction layer 50-2, feature map 20-2 is output from feature extraction layer 50-2. Feature extraction layer 50-2 is one of multiple feature extraction layers 50 included in neural network 60. Feature map 20-2 is one of multiple feature maps 20 generated by neural network 60.

[0072] The second generation unit 2060 generates a centrality map 140 from the feature map 20-2. The centrality map 140 is an activation map obtained from the feature map 20-2. Then, the second generation unit 2060 generates the centrality map 140 by calculating the activation value of each unit in the feature map 20-2 using an activation function 70-2. The activation function 70-2 is one of a plurality of activation functions 70 included in the neural network 60.

[0073] Furthermore, the second generation unit 2060 generates the existence probability map 80 by using the category score map 130 and the centrality map 140. For example, the second generation unit 2060 generates the existence probability map 80 by calculating the cell-by-cell product between the category score map 130 and the centrality map 140.

[0074] The operation of calculating the cell-by-cell product between two graphs means calculating the product between cells at the same location. For example, in the existence probability graph 80, the value of the cell at position (i, j) indicates the product of the value of the cell at position (i, j) in the category score graph 130 and the value of the cell at position (i, j) in the centrality graph 140.

[0075] As described above, the image processing device 2000 can generate an existence probability map 80 for each of the multiple categories. In generating the existence probability map 80 for each of the multiple categories, the neural network 60 generates a category score map 130 for each category. For this purpose, a feature extraction layer 50-1 is prepared for each category. For each category, the second generation unit 2060 generates the existence probability map 80 for that category by calculating the unit-by-unit product between the category score map 130 and the centrality map 140 generated for that category.

[0076] <Processing Procedure>

[0077] Figure 6 This is a first flowchart illustrating the processing procedure performed by the image processing device 2000. Figure 6 In the example, probability map 80 is considered the target map.

[0078] Quantization unit 2020 acquires image 10 (S102). Quantization unit 2020 performs quantization processing on image 10 to generate quantized image 12 (S104). First generation unit 2040 generates feature map 20 based on quantized image 12 (S106). Second generation unit 2060 generates existence probability map 80 from feature map 20 (S108). Determination unit 2080 determines whether the value of each cell in existence probability map 80 matches a first fixed value (S110). Dequantization unit 2100 performs dequantization processing on each cell in existence probability map 80 based on the determination result in S110 (S112).

[0079] Figure 7 This is a second flowchart illustrating the processing procedure performed by the image processing device 2000. Figure 7 In the example, feature map 20 is considered as the target map. Note that steps S102 to S106 are... Figure 6 and Figure 7 This is common to all. Therefore, the following description will begin from the process after step S106. Figure 7 The flowchart.

[0080] Determining unit 2080 determines whether the value of each cell in feature map 20 matches a first fixed value (S208). Dequantizing unit 2100 performs dequantization processing on each cell in feature map 20 based on the determination result in S208 (S210). Second generation unit 2060 generates existence probability map 80 by using dequantized feature map 20 (S212).

[0081] <Image 10 Acquisition: S102>

[0082] The quantization unit 2020 acquires image 10 (S102). Various methods can be used to acquire the image to be processed. For example, image 10 is stored in a storage unit accessible from the image processing device 2000. The quantization unit 2020 acquires image 10 stored in the storage unit.

[0083] For example, the user of the image processing device 2000 specifies the image to be treated as image 10 by specifying a filename, etc. The quantization unit 2020 retrieves the user-specified image from the storage unit as image 10.

[0084] Alternatively, image 10 may be sent to image processing device 2000 from a device other than image processing device 2000. In this case, quantization unit 2020 receives the image sent from another device and processes that image as image 10.

[0085] Any device can send image 10 to image processing device 2000. For example, the device could be a camera that generated image 10. In this case, for example, image processing device 2000 generates an existence probability map 80 for image 10 generated by the camera. For example, if a camera is provided in a vehicle, the situation around the vehicle is captured in image 10. Therefore, by using image processing device 2000 to generate existence probability map 80 from image 10, the existence probability distribution of objects around the vehicle can be determined.

[0086] <Quantization Processing: S104>

[0087] The quantization unit 2020 performs quantization processing on the image 10 to generate a quantized image 12 (S104). Various specific quantization methods exist. For example, the quantization unit 2020 quantizes the value of each pixel of the image 10 according to the following formula (2).

[0088] [Mathematics 2]

[0089] Here, q(x) represents the quantization function. s and u both represent predetermined parameters used for quantization. Ya represents the minimum value of y during quantization (i.e., the minimum pixel value that can be taken in the quantized image 12). Furthermore, Yb represents the maximum value of y during quantization (i.e., the maximum pixel value that can be taken in the quantized image 12).

[0090] Alternatively, the quantization process described above maps the pixel values ​​of image 10 to a range of values, where Ya is greater than or equal to Yb is less than or equal to Yb. For example, Ya and Yb are set to the minimum value Y_min and the maximum value Y_max that the second data type can take, respectively.

[0091] <Generating Feature Map 20: S106>

[0092] The first generation unit 2040 generates a feature map 20 based on the quantized image 12 using the feature extraction layer 50 (S106). As described above, for example, the feature extraction layer 50 is configured to generate the feature map 20 in response to the input of the image (see [link]). Figure 4 In this case, the first generation unit 2040 obtains the feature map 20 from the feature extraction layer 50 by inputting the quantized image 12 into the feature extraction layer 50.

[0093] Additionally, for example, feature extraction layer 50 is configured to generate feature map 20 in response to input of a feature map other than feature map 20 (see [reference]). Figure 5 In this case, the first generation unit 2040 obtains the feature map 20 by inputting the feature map generated from the quantized image 12 into the feature extraction layer 50.

[0094] like Figure 6 and Figure 7 As illustrated, the processing performed after generating feature map 20 differs between the case where the existence probability map 80 is considered the target map and the case where feature map 20 is considered the target map. First, the case where the existence probability map 80 is considered the target map will be described. Then, the case where feature map 20 is considered the target map will be described.

[0095] <The case where probability graph 80 is considered as the target graph>

[0096] <<Generation of Probability Graph 80: S108>>

[0097] The second generation unit 2060 generates an existence probability map 80 from the feature map 20 (S108). For example, as referenced... Figure 4 As described in the example, the second generation unit 2060 generates an activation map 40 from the feature map 20, and generates an existence probability map 80 from the activation map 40. Additionally, for example, as referenced... Figure 5 As described in the example, the second generation unit 2060 generates multiple activation maps 40 from the feature map 20, and generates an existence probability map 80 based on the multiple generated activation maps 40.

[0098] <<Process confirmed: S110>>

[0099] Determining unit 2080 determines whether the value of each cell in the existence probability map 80 matches a first fixed value (S110). For example, determining unit 2080 sequentially compares the value of each cell in the existence probability map 80 with a first fixed value C (e.g., Y_min) in a second data type before dequantization. Alternatively, determining unit 2080 may perform comparisons of multiple cell values ​​with the first fixed value C (e.g., Y_min) in parallel. For example, for parallel comparison processing, a Single Instruction Multiple Data (SIMD) instruction may be used.

[0100] For example, integers and floating-point numbers are used as the second and third data types, respectively. Comparing values ​​in the second data type (integers) before dequantization has the advantage of improving processing speed and power efficiency compared to comparing values ​​in the third data type (floating-point) after dequantization.

[0101] <<Dequantization Process: S110>>

[0102] Dequantization unit 2100 performs dequantization processing (S112) on each cell of the existence probability map 80 based on the result of the determination (i.e., matching determination process) performed in S110. Specifically, dequantization unit 2100 performs dequantization processing by using a dequantization function z=dq(y), which is the inverse function of the quantization function y=dq^-1(z) of formula (1).

[0103] For example, the image processing device 2000 performs a matching determination process on all cells of the existence probability map 80, and then performs a dequantization process. Alternatively, for example, the image processing device 2000 sequentially performs the process of "performing a matching determination process and performing a dequantization process based on the determination result" on each cell of the existence probability map 80.

[0104] Alternatively, for example, the image processing device 2000 may divide the units of the existence probability map 80 into groups with a predetermined number of units, and perform processing on each group. That is, the image processing device 2000 sequentially performs the process of "performing a matching determination process on each unit included in the group and performing dequantization processing based on the determination result" on multiple groups.

[0105] Preferably, the group size is set such that all units included in a group can be processed in parallel. For example, in the case of implementing Single Instruction Multiple Data (SIMD) instructions in processor 1040, the group size is set to the number of data items that a SIMD instruction can process at one time.

[0106] For example, dequantization unit 2100 uses a dequantization function to exclude units from the dequantization target whose value in the probability map 80 is the first fixed value. In other words, dequantization unit 2100 only performs dequantization on units whose value is not the first fixed value.

[0107] Here, the dequantization existence probability map 82 is generated based on the result of the dequantization process of the dequantization unit 2100. For example, an initial value such as the dequantization threshold Th is set as the initial value for each cell of the dequantization existence probability map 82. After generating the dequantization existence probability map 82 in which the initial values ​​are set, the dequantization unit 2100 sets the value calculated using the dequantization function (i.e., z=dq(y)) to the corresponding cell of the dequantization existence probability map 82. In this way, for the cells of the dequantization existence probability map 82 corresponding to the cells of the existence probability map 80 excluded from the dequantization target using the dequantization function, an initial value such as the dequantization threshold Th is still set. As a result, each cell of the existence probability map 80 with a value of the first fixed value is dequantized to a common fixed value as the initial value.

[0108] For example, suppose we set an initial value Th for each cell in the dequantized existence probability graph 82. Further, in the existence probability graph 80, suppose the value y[i1, j1] of the cell at position (i1, j1) is not a first fixed value. In this case, we set dq(y[i1, j1]) for the cell at position (i1, j1) in the dequantized existence probability graph 82. On the other hand, in the existence probability graph 80, suppose the value y[i2, j2] of the cell at position (i2, j2) is a first fixed value. In this case, the cell at position (i2, j2) in the dequantized existence probability graph 82 remains at the initial value Th.

[0109] The dequantization unit 2100 does not necessarily generate the dequantized presence probability map 82. For example, the dequantization unit 2100 generates a tuple (i, j, z[i, j]) for each cell where the value of the presence probability map 80 is not the first fixed value, and the tuple represents a combination of the position (i, j) of the cell and the value z[i, j] obtained through the value y[i, j] of the dequantization unit.

[0110] Here, how to perform determination in the dequantization process will be described in more detail. For example, the quantization function dq in the quantization formula (1) is determined such that the numerical range of "equal to or greater than the dequantized threshold Th and equal to or less than the observed maximum value of the cells in the presence probability map 80" after dequantization is mapped to the numerical range of "equal to or greater than Y_min and equal to or less than Y_max" that the value before dequantization (i.e., the value of the second data type) can take. The observed maximum value of the cells in the presence probability map 80 is the maximum value of the values actually indicated in the cells predicted to be in the presence probability map 80%. The dequantization process is determined to be the inverse process of the quantization process determined in this way.

[0111] The observed maximum value of the cells in the presence probability map 80 is determined by, for example, previous calibration. In the previous calibration, the presence probability map 80 is generated for each of the multiple images 10. Then, the observed maximum value is determined using the set of cell values obtained from the multiple generated presence probability maps 80.

[0112] For example, the maximum value included in the set of cell values is used as the observed maximum value. Additionally, for example, the value obtained by adding a predetermined margin to the maximum value included in the set of cell values is used as the observed maximum value. Note that calibration is preferably performed such that the observed maximum value of the cells in the presence probability map 80 is Y_max which is the maximum value of the third data type.

[0113] Figure 8 is a graph showing the dequantization function determined based on the dequantized threshold and the observed maximum value. First, in the quantization process, the quantization function y = dq^-1(z) is determined such that the numerical range of z where [Th < z <= Z_max] is mapped to the numerical range of Y where [Y_min < y <= Y_max]. Further, in the quantization process, the numerical range of z where [Th < z] is determined so as to be mapped to y = Ymin. Thus, the dequantization function z = dq(y) is determined such that the numerical range of y where [Y_min <= y <= Y_max] is mapped to the numerical range of z where [Th <= z <= Z_max]. As a more specific example, the dequantization function is defined as a function representing a straight line passing through two points (Y_min, Th) and (Y_max, Z_max).

[0114] Here, quantization, which converts all values ​​equal to or less than Th to Y_min, also has the advantage of improving processing speed. For example, suppose we generate an existence probability map 80 for each of the multiple categories. In this case, for each of the multiple regions of image 10, we obtain a vector (hereinafter referred to as the existence probability vector) indicating the existence probability of objects in each category.

[0115] For example, suppose we generate an existence probability map 80 for each of the three types of categories C1, C2, and C3. In this case, we obtain an existence probability vector (L1[i,j], L2[i,j], L3[i,j]), where for each position (i,j), we list the existence probability L1[i,j] of objects in category C1, the existence probability L2[i,j] of objects in category C2, and the existence probability L3[i,j] of objects in category C3.

[0116] Relative to image 10, assume that object-related processing is performed using a dequantized existence probability map 82 obtained by dequantizing the existence probability map 80. In this processing, regions where no objects exist can be ignored. Therefore, it is preferable that regions where no objects exist can be easily specified.

[0117] Here, it is assumed that regions where no objects exist are specified based on the determination criterion that "when the value of a cell in the dequantized existence probability map 82 is equal to or less than a predetermined probability threshold, there is no object in the cell." In this case, the probability threshold is used as the post-dequantization threshold Th. By using the probability threshold as the post-dequantization threshold, all elements of the existence probability vector obtained for regions where no objects exist indicate Y_min. Therefore, by detecting existence probability vectors where all elements are Y_min, regions where no objects exist can be easily specified even before dequantization processing is performed on the existence probability map 80. Thus, the processing can be accelerated, for example, by performing dequantization only on cells indicating regions where objects exist.

[0118] For example, by using parallel comparison processing with SIMD instructions, it is easy and fast to detect that all elements are Y_min in the existence probability vector. For instance, SIMD instructions are executed on a vector where all elements are set to Y_min and the existence probability vector to "determine in parallel whether the values ​​at the same positions are consistent with each other." Through parallel comparison processing, it is easy and fast to determine whether all elements of the existence probability vector are Y_min.

[0119] Here, there exists a processor characterized in that the cost of a matching determination process, which determines whether two values ​​match, is lower than the cost of a size comparison process, which determines whether one value is greater than another. In this processor, the cost of a matching determination process that determines whether a cell's value matches Y_min is lower than the cost of a size comparison process that determines whether a cell's value is equal to or less than Y_min. Therefore, an image processing device 2000, designed with dequantization processing such that the values ​​of cells excluded from the processing target are mapped to a fixed value, can detect cells that can be excluded from the processing target at low cost.

[0120] <<Method for determining the threshold after dequantization>>

[0121] When the dequantization function is determined to be the inverse function of the quantization function as defined above, the value of the cell with the minimum value Y_min in the probability graph 80 is dequantized to the dequantized threshold Th. Therefore, values ​​smaller than Th will not appear in the dequantized values. Thus, the dequantized threshold Th is preferably determined as the boundary value between "relatively small values ​​for which differences between them do not need to be considered" and "relatively large values ​​for which differences between them need to be considered." In other words, the dequantized threshold is preferably defined as the boundary value between relatively unimportant values ​​and relatively important values. In this way, all relatively unimportant values ​​after dequantization are aggregated into a single value of Y_min before dequantization. Therefore, the relatively unimportant values ​​appearing after dequantization are only the dequantized threshold Th.

[0122] The dequantization threshold Th can be determined in various ways. For example, the dequantization threshold can be manually determined by the administrator or user of the image processing device 2000.

[0123] Alternatively, the dequantization threshold can be determined, for example, under the assumption that there is a use case for the probability map 80. For example, as described above, the probability threshold used as a standard for determining whether an object exists in the region is used as the dequantization threshold.

[0124] For example, suppose activation map 40 is directly used as presence probability map 80. And suppose the probability threshold is set to 0.8. In this case, in presence probability map 80, image regions of image 10 corresponding to units with values ​​of 0.8 or greater are considered image regions of present objects. Therefore, the same value as the probability threshold, 0.8, is used as the dequantized threshold.

[0125] When the dequantized presence probability map 82 is used to detect objects, the importance level of cells with values ​​greater than the probability threshold is higher than that of cells in the dequantized presence probability map 82 with values ​​equal to or less than the probability threshold. Since the values ​​of cells greater than the probability threshold have a relatively high importance level, it is preferable to use a dequantization function for calculation. On the other hand, since the values ​​of cells equal to or less than the probability threshold have a relatively low importance level, it is assumed that there is no problem in detecting objects, even when converting this value to a fixed value Th. Furthermore, as mentioned above, cells equal to or less than the probability threshold can be excluded from the processing targets. Therefore, the probability threshold can be used as the dequantized threshold Th.

[0126] <Case where feature map 20 is considered as the target map>

[0127] When feature map 20 is considered as a target map, dequantization is performed on feature map 20. Here, when dequantization is performed on feature map 20, matching determination and dequantization are performed in neural network 60. Therefore, the configuration of neural network 60 when feature map 20 is a target map will be described more specifically first.

[0128] Figure 9 The illustration shows the case where feature map 20 is regarded as the target map. Figure 4 A more specific configuration of the neural network 60 illustrated. In Figure 9 In the neural network 60, the determining unit 2080 performs a matching determination process on the feature map 20 output from the feature extraction layer 50. Furthermore, based on the result of the matching determination process, the dequantization unit 2100 performs a dequantization process on the feature map 20. As a result, a dequantized feature map 22 is generated. Each unit of the dequantized feature map 22 is obtained by dequantizing the corresponding unit of the feature map 20 by the dequantization unit 2100.

[0129] Figure 10 The illustration shows the case where feature map 20 is regarded as the target map. Figure 5 A more specific configuration of the neural network 60 illustrated. Note that in Figure 10 In this paper, components other than classification branch 110 and centrality calculation branch 120 are omitted.

[0130] exist Figure 10In the neural network 60, the determining unit 2080 performs a matching determination process on the feature map 20-1 output from the feature extraction layer 50-1. Furthermore, based on the result of the matching determination process, the dequantization unit 2100 performs a dequantization process on the feature map 20-1. As a result, a dequantized feature map 22-1 is generated. Each unit of the dequantized feature map 22-1 is obtained by dequantizing the corresponding unit of the feature map 20-1 by the dequantization unit 2100. However, a dequantization function can be used to exclude units of the feature map 20-1 with values ​​of a first fixed value from the dequantization target.

[0131] Similarly, the determining unit 2080 performs a matching determination process on the feature map 20-2 output from the feature extraction layer 50-2. Furthermore, based on the result of the matching determination process, the dequantization unit 2100 performs a dequantization process on the feature map 20-2. As a result, a dequantized feature map 22-2 is generated. Each cell of the dequantized feature map 22-2 is obtained by dequantizing the corresponding cell of the feature map 20-2 by the dequantization unit 2100. However, a dequantization function can be used to exclude cells of the feature map 20-2 with values ​​of a first fixed value from the dequantization target.

[0132] <<Match Determination Process: S208>>

[0133] The determining unit 2080 determines whether the value of each cell in the feature map 20 matches a first fixed value (S208). As described above, the determining unit 2080 can sequentially compare the value of each cell in the feature map 20 of the second data type before dequantization with the first fixed value C (e.g., Y_min), or it can perform the comparison of the values ​​of multiple cells with the first fixed value C (e.g., Y_min) in parallel. Compared to comparing values ​​in the third data type, which is a floating-point type, after dequantization, comparing values ​​in the second data type, which is an integer type, before dequantization has the advantage of improving processing speed and power efficiency.

[0134] <<<Example of methods for determining the dequantized threshold Th>>>

[0135] The dequantization threshold Th can be determined in various ways. For example, the dequantization threshold Th can be manually determined by the administrator or user of the image processing device 2000. Alternatively, as described below, the dequantization threshold Th can be determined by considering the use case of the dequantized feature map 20 (dequantized feature map 22).

[0136] With feature map 20 considered as the target map, dequantized feature map 22 is generated from feature map 20. Then, activation map 40 is generated by calculating the activation value of each cell in dequantized feature map 22.

[0137] Therefore, for example, a dequantization threshold Th is determined such that the activation values ​​obtained from the dequantized feature map 22 become relatively large to some extent. Specifically, firstly, an activation threshold Ta is determined for the activation values ​​obtained using activation function 70. For example, the aforementioned probability threshold is used as the activation threshold.

[0138] A dequantized threshold Th is determined such that the value obtained by inputting the dequantized threshold Th into activation function 70 matches the activated threshold. Therefore, the dequantized threshold Th is computed by inputting the activated threshold into the inverse function of activation function 70. For example, if activation function 70 is a sigmoid function, the inverse function is a logit function.

[0139] Specifically, the dequantized threshold Th is calculated using the activation threshold Ta as follows.

[0140] [Mathematics 3]

[0141] Here, f^-1() denotes the inverse function of activation function 70. Therefore, f() denotes activation function 70.

[0142] Note that the first fixed value C can be determined using the dequantized threshold Th determined in this way. Specifically, the first fixed value C is determined as C=dq^-1(Th) by using y=dq^-1(z), where y=dq^-1(z) is the inverse function of the dequantization function z=dq(y) used by the dequantization unit 2100.

[0143] Here, a function without an inverse function can also be used as the activation function 70. In this case, the inverse function f-1() is defined by extracting the range from which the inverse function can be defined from the activation function 70, and the activation threshold is included in this range. Then, the dequantized threshold is calculated from the activation threshold using the inverse function f-1().

[0144] Note that, as Figure 10 As illustrated, it is assumed that category score map 130 and centrality map 140 are generated as activation map 40. That is, it is assumed that feature map 20-1 for calculating category scores and feature map 20-2 for calculating centrality are generated as feature map 20. In this case, the fixed value for dequantization of feature map 20-1 and the fixed value for dequantization of feature map 20-2 are preferably set to the same value, and the same value is more preferably the minimum value X_min of the second data type.

[0145] <<Dequantization Process: S210>>

[0146] Dequantization unit 2100 performs dequantization processing on each cell of feature map 20 based on the determination result in S208 (S210). For example, the method of performing dequantization on feature map 20 in S210 is similar to the method of performing dequantization on existence probability map 80 in S112. That is, dequantization unit 2100 uses a dequantization function to exclude cells of feature map 20 whose value is a first fixed value from the dequantization target. Then, dequantization unit 2100 performs dequantization on each cell of feature map 20 whose value is not the first fixed value using a dequantization function.

[0147] For example, dequantization unit 2100 initializes dequantized feature map 22 in the same manner as dequantization existence probability map 82. That is, dequantization unit 2100 generates dequantized feature map 22, in which an initial value such as a dequantized threshold Th is set for each cell. Then, dequantization unit 2100 performs dequantization on each cell of feature map 20 whose value is not the first fixed value using a dequantization function, and sets the value obtained by dequantization to the corresponding cell of dequantized feature map 22. In this case, the initial value is still set in the cell of dequantized feature map 22 corresponding to the cell of feature map 20 whose value is the first fixed value.

[0148] <<Generation of the probability graph 80: S212>>

[0149] The second generation unit 2060 generates an existence probability map 80 using the dequantized feature map 22 (S212). For example, the second generation unit 2060 inputs the value of each cell of the dequantized feature map 22 into the activation function 70 to obtain the activation value of each cell of the dequantized feature map 22. Further, the second generation unit 2060 generates an activation map 40, which indicates the activation value obtained for each cell of the dequantized feature map 22. Then, the second generation unit 2060 generates the existence probability map 80 based on the activation map 40.

[0150] Note that when generating the existence probability map 80 from the dequantized feature map 22, the value indicated by each cell of the existence probability map 80 is a dequantized value (a value of the third data type). Therefore, the existence probability map 80 generated in S212 is the dequantized existence probability map 80 (i.e., the dequantized feature map 22).

[0151] Here, it is assumed that the first fixed value is determined with respect to the activation value. Specifically, it is assumed that the first fixed value is determined by “applying the post-activation threshold Ta to the inverse function of activation function 70 to obtain the post-dequantization threshold Th, and applying the post-dequantization threshold Th to the quantization function y=dq^-1(z) to obtain the first fixed value C=dq^-1(Th)”. In this case, it can be said that the activation values ​​obtained from the cells in the dequantized feature map 22 whose values ​​are set to the post-dequantization threshold Th have a lower importance level.

[0152] Therefore, for units in the dequantized feature map 22 whose values ​​are the dequantized threshold Th, the second generation unit 2060 does not necessarily use the activation function 70 to calculate the activation values. Specifically, the second generation unit 2060 uses a predetermined fixed value (hereinafter referred to as the second fixed value) as the activation value for units in the dequantized feature map 22 whose values ​​are the dequantized threshold. For example, the activation threshold Ta is used as the second fixed value. On the other hand, the second generation unit 2060 calculates the activation values ​​for units in the dequantized feature map 22 whose values ​​are not the dequantized threshold by inputting the values ​​of the units into the activation function 70.

[0153] Here, the unit in dequantized feature map 22 whose value is the dequantized threshold is the unit corresponding to the unit in feature map 20 whose value was determined to match the first fixed value during the matching determination process. Therefore, units in dequantized feature map 22 corresponding to units in feature map 20 excluded from the activation target using activation function 70 can be excluded. Note that during the dequantization process, it is possible to record which unit in dequantized feature map 22 has the dequantized threshold value.

[0154] This method generates the activation map 40 without performing the activation function 70 operation on cells whose values ​​are the dequantized threshold. Therefore, the computational cost required to generate the activation map 40 can be reduced.

[0155] <Output of Results>

[0156] The image processing device 2000 outputs information indicating the processing result (hereinafter referred to as output information). The functional configuration unit for outputting output information is called the output unit. Figure 11 This diagram illustrates the configuration of an image processing device 2000 with an output unit.

[0157] The output information can include any kind of information. For example, output unit 2120 generates output information including a dequantized presence probability map 82. When generating a presence probability map 80 for each of multiple categories, the output information can include a presence probability map 80 for each category. Note that the dequantized presence probability map 82 can be a presence probability map 80 generated from the dequantized feature map 22.

[0158] Note that by using the existence probability map 80 obtained for each category, data (e.g., a vector) indicating the existence probability of objects of each category is obtained for each region of the image 10 corresponding to each cell of the existence probability map 80. For example, suppose that the existence probability map 80 is generated for each of the three types of categories C1, C2, and C3. In this case, the image processing device 2000 can obtain an existence probability vector (L1[i,j], L2[i,j], L3[i,j]), where the existence probability L1[i,j] of objects of category C1, the existence probability L2[i,j] of objects of category C2, and the existence probability L3[i,j] of objects of category C3 are listed for each position (i,j).

[0159] Any mode can be applied to the output mode of the output information. For example, output unit 2120 can store the output information in any storage unit. Alternatively, output unit 2120 can output the output information to any display device so that the display device can display the output information. Alternatively, output unit 2120 can send the output information to another device.

[0160] Training a Neural Network 60

[0161] The neural network 60 is pre-trained. Training of the neural network 60 is performed using multiple training samples. The training samples are, for example, a pair of training images and a ground truth map, indicating the spatial distribution of the probability of the presence of objects in the training images. The ground truth map indicates the ideal probability map 80 output from the neural network 60 in response to the input of the training images.

[0162] The device that performs training on the neural network 60 (hereinafter referred to as the training device) calculates a loss using an existence probability map 80 and a ground truth map obtained by inputting training images into the neural network 60. Further, the training device updates the trainable parameters (biases, weights between nodes, etc.) included in the neural network 60 using the calculated loss. The training device trains the neural network 60 by repeating parameter updates using multiple training samples.

[0163] In the foregoing, this disclosure has been described with reference to embodiments, but this disclosure is not limited to the above embodiments. Within the scope of this disclosure, various modifications to the configuration and details of this disclosure can be understood by those skilled in the art. In addition, each embodiment can be appropriately combined with other embodiments.

[0164] Each figure is merely an illustration provided to describe one or more embodiments. Each figure is not necessarily associated with only one specific embodiment, but may be associated with one or more other embodiments. Those skilled in the art will understand that various features or steps described with reference to any of the figures may be combined with features or steps illustrated in one or more other figures, for example, to produce embodiments not explicitly illustrated or described. All features or steps illustrated in any of the figures used to describe embodiments are not necessarily essential, and some features or steps may be omitted. The order of steps described in any figure may be changed appropriately.

[0165] In this disclosure, when a program is loaded into a computer, the program includes a set of instructions (or software code) for causing the computer to perform one or more functions described in the embodiments. The program may be stored on a non-transient computer-readable medium or a tangible storage medium. By way of example, and not limitation, the computer-readable medium or tangible storage medium may include random access memory (RAM), read-only memory (ROM), flash memory, solid-state drives (SSDs) or other memory technologies, CD-ROMs, digital versatile discs (DVDs), Blu-ray discs or other optical disc storage devices, magnetic tape cassettes, magnetic tapes, disk storage devices, or other magnetic storage devices. The program may be transmitted on a transient computer-readable medium or a communication medium. By way of example, and not limitation, the transient computer-readable medium or communication medium may include electrical, optical, acoustic, or other forms of propagated signals.

[0166] <Supplementary Notes>

[0167] (Supplementary Note 1)

[0168] An image processing device, comprising:

[0169] A quantization unit is configured to generate a second image by performing quantization processing on a first image;

[0170] The first generation unit is configured to generate a feature map based on the second image;

[0171] The second generation unit is configured to generate an existence probability map that indicates the existence probability distribution of objects in the first image by using feature maps;

[0172] The unit is configured to perform a matching determination process, which determines whether the value of each unit matches a predetermined fixed value in a feature map or an existence probability map; and

[0173] The dequantization unit is configured to perform dequantization on the feature map or existence probability map based on the result of the matching determination process.

[0174] (Supplementary Note 2)

[0175] According to Supplementary Note 1, the image processing device

[0176] The dequantization unit is configured as follows:

[0177] When the value of a cell does not match a fixed value, the cell value is dequantized using a dequantization function; and

[0178] When the value of a cell matches a fixed value, the predetermined value is used as the value obtained by dequantizing the value of the cell.

[0179] (Supplementary Note 3)

[0180] According to Supplementary Note 2, the image processing device

[0181] The matching determination process and dequantization process are performed on the existing probability graph, and

[0182] The fixed value is obtained by inputting the probability threshold used to determine the existence of an object into the inverse function of the dequantization function.

[0183] (Supplementary Note 4)

[0184] According to Supplementary Note 3, the image processing device

[0185] The inverse function of the dequantization function is defined as the minimum possible value of the data type used in the feature map or existence probability map, such that the fixed value is set to the minimum value.

[0186] (Supplementary Note 5)

[0187] According to Supplementary Note 1, the image processing device

[0188] The matching and dequantization processes are performed on the feature maps, and

[0189] The second generation unit generates an existence probability map by using a dequantized feature map.

[0190] (Supplementary Note 6)

[0191] According to Supplementary Note 3, the image processing device

[0192] The first generation unit generates a first feature map and a second feature map in the feature map.

[0193] The determining unit performs a matching determination process on each of the first feature map and the second feature map in the feature map.

[0194] The dequantization unit performs dequantization processing on each of the first feature map and the second feature map in the feature map, and

[0195] The second generation unit is configured as follows:

[0196] By using the first feature map in the feature map, a category map indicating the probability distribution of the existence of objects of a specific category is generated;

[0197] By using a second feature map within the feature map, a centrality map indicating the distribution of proximity to the object's center is generated; and

[0198] By using category graphs and centrality graphs, an existence probability graph is generated for objects of a specific category.

[0199] (Supplementary Note 7)

[0200] According to Supplementary Note 5, the image processing device

[0201] The second generation unit generates an existence probability map from the dequantized feature map by using an activation function.

[0202] (Supplementary Note 8)

[0203] According to Supplementary Note 7, the image processing device

[0204] The fixed value is obtained by inputting the value into the inverse function of the dequantization function, which is obtained by inputting the probability threshold used to determine the existence of an object into the inverse function of the activation function.

[0205] (Supplementary Note 9)

[0206] According to Supplementary Note 8, the image processing device

[0207] The inverse function of the dequantization function is defined as the minimum possible value of the data type used in the first and second feature maps of the feature map, such that the fixed value is set to the minimum value.

[0208] (Supplementary Note 10)

[0209] According to Supplementary Note 7, the image processing device

[0210] The second generation unit is configured as follows:

[0211] By inputting the dequantized value of a cell into an activation function, the activation value of the cell that matches the fixed value is calculated; and

[0212] Use a predetermined value as the activation value for a cell whose value does not match the fixed value.

[0213] (Supplementary Note 11)

[0214] A computer-executed image processing method, the method comprising:

[0215] The quantization step generates the second image by performing quantization processing on the first image;

[0216] The first generation step involves generating a feature map based on the second image;

[0217] The second generation step involves generating an existence probability map that indicates the existence probability distribution of objects in the first image by using feature maps.

[0218] The steps include performing a threshold determination process, which determines whether the value of each cell is equal to or less than the lower bound threshold of the feature map or the existence probability map; and

[0219] The dequantization step performs dequantization processing on the feature map or existence probability map based on the result of the threshold determination process.

[0220] (Supplementary Note 12)

[0221] A program used to make a computer execute:

[0222] The quantization step generates the second image by performing quantization processing on the first image;

[0223] The first generation step involves generating a feature map based on the second image;

[0224] The second generation step involves generating an existence probability map that indicates the existence probability distribution of objects in the first image by using feature maps.

[0225] The steps include performing a threshold determination process, which determines whether the value of each cell is equal to or less than the lower bound threshold of the feature map or the existence probability map; and

[0226] The dequantization step performs dequantization processing on the feature map or existence probability map based on the result of the threshold determination process.

[0227] Some or all of the elements (e.g., configuration and functionality) described in Supplementary Notes 2 through 8, which depend on Supplementary Note 1, may also depend on each of Supplementary Notes 9 through 12 through the same dependencies as Supplementary Notes 2 through 8. Some or all of the elements described in any of the Supplementary Notes may be applied to a variety of hardware, software, recording units for recording software, systems, and methods.

Claims

1. An image processing apparatus, comprising: A quantization unit is configured to generate a second image by performing quantization processing on a first image; The first generation unit is configured to generate a feature map based on the second image; The second generation unit is configured to generate an existence probability map by using the feature map, the existence probability map indicating the distribution of the existence probability of objects in the first image; The determining unit is configured to perform a matching determination process, which is a process of determining whether the value of each unit matches a predetermined fixed value of the feature map or the existence probability map; as well as The dequantization unit is configured to perform dequantization processing on the feature map or the existence probability map based on the result of the matching determination process.

2. The image processing apparatus according to claim 1, The dequantization unit is configured as follows: When the value of the unit does not match the fixed value, the value of the unit is dequantized using a dequantization function; and When the value of the unit matches the fixed value, the predetermined value is used as the value obtained by dequantizing the value of the unit.

3. The image processing apparatus according to claim 2, The matching determination process and the dequantization process are performed on the existence probability graph, and The fixed value is obtained by inputting a probability threshold used to determine the existence of an object into the inverse function of the dequantization function.

4. The image processing apparatus according to claim 3, The inverse function of the dequantization function is defined as the minimum possible value of the data type used in the feature map or the existence probability map, such that the fixed value is set to the minimum value.

5. The image processing apparatus according to claim 1, The matching determination process and the dequantization process are performed on the feature map, and The second generation unit generates the existence probability map by using the dequantized feature map.

6. The image processing apparatus according to claim 3, The first generation unit generates a first feature map and a second feature map in the feature map. The determining unit performs the matching determination process on each of the first feature map and the second feature map in the feature map. The dequantization unit performs the dequantization process on each of the first feature map and the second feature map in the feature map, and The second generation unit is configured as follows: A category map is generated by using the first feature map in the feature map, the category map indicating the distribution of the probability of the existence of objects of a particular category; A centrality map is generated by using the second feature map in the feature map, the centrality map indicating the distribution of proximity to the center of the object; and The existence probability map is generated for the object of the specific category by using the category map and the centrality map.

7. The image processing apparatus according to claim 5, The second generation unit generates the existence probability map from the dequantized feature map by using an activation function.

8. The image processing apparatus according to claim 7, The fixed value is obtained by inputting a value obtained by inputting the inverse function of the dequantization function into the inverse function of the activation function by inputting a probability threshold used to determine the existence of an object into the inverse function of the activation function.

9. The image processing apparatus according to claim 8, The inverse function of the dequantization function is defined such that the fixed value is set to the minimum of the possible values ​​of the data types used in the first feature map and the second feature map of the feature map.

10. The image processing apparatus according to claim 7, The second generation unit is configured as follows: By inputting the dequantized value of the cell into the activation function, the activation value of the cell that matches the fixed value is calculated; and A predetermined value is used as the activation value for the unit whose value does not match the fixed value.

11. An image processing method, executed by a computer, the method comprising: The quantization step generates the second image by performing quantization processing on the first image; The first generation step involves generating a feature map based on the second image; The second generation step involves generating an existence probability map using the feature map, the existence probability map indicating the distribution of the existence probability of objects in the first image; The steps are defined, and a threshold determination process is performed, which is the process of determining whether the value of each unit is equal to or less than the lower limit threshold of the feature map or the existence probability map; as well as The dequantization step involves performing dequantization processing on the feature map or the existence probability map based on the result of the threshold determination process.

12. A computer-readable storage medium storing a program, the program being used to cause a computer to execute a process, the process comprising: The quantization step generates the second image by performing quantization processing on the first image; The first generation step involves generating a feature map based on the second image; The second generation step involves generating an existence probability map using the feature map, the existence probability map indicating the distribution of the existence probability of objects in the first image; The steps are defined, and a threshold determination process is performed, which is the process of determining whether the value of each unit is equal to or less than the lower limit threshold of the feature map or the existence probability map; as well as The dequantization step involves performing dequantization processing on the feature map or the existence probability map based on the result of the threshold determination process.

Images