Visualization device and visualization method

Abstract

This visualization device is configured to comprise: a function acquisition unit (1) which, when input data is given to an input layer, acquires a model function that is a function obtained by modeling the computation of a plurality of intermediate layers of a neural network that outputs an analysis result of the input data from an output layer; a contribution level calculation unit (2) that calculates the contribution level of each of the intermediate layers to the analysis result on the basis of the model function acquired by the function acquisition unit (1); and a map generation unit (3) that generates a visualization map of each of the intermediate layers on the basis of the contribution level calculated by the contribution level calculation unit (2). When the number of the plurality of intermediate layers is M (M is an integer of 2 or more), the contribution level calculation unit (2) repeatedly performs an update process for updating the value of the m-th (m=1, ..., M) intermediate layer from the input layer side on the basis of the model function, and calculates the contribution level of the m-th intermediate layer on the basis of the step interval between the step of the s-th (s is an integer of 1 or more) update process and the step of the (s+1)-th update process, and a gradient related to the value of the m-th intermediate layer after the s-th update process.

Description

Visualization device and visualization method

[0001] The present disclosure relates to a visualization device and a visualization method.

[0002] When input data is provided to an input layer, a visualization device calculates the contribution of each intermediate layer to the analysis result based on a model function, which is a function that models the operations of multiple intermediate layers in a neural network that outputs an analysis result of the input data from an output layer, and generates a visualization map of each intermediate layer based on the contribution of each intermediate layer. For example, Non-Patent Document 1 discloses an example of such a visualization device that generates a visualization map of each intermediate layer using a visualization technique called Grad-CAM (Gradient-weighted Class Activation Mapping). Grad-CAM calculates the gradient of a model function in one of multiple intermediate layers and linearly approximates the model function using the gradient of the model function in the one intermediate layer. Grad-CAM then calculates the contribution of each intermediate layer to the analysis result based on a linear function, which is the model function after the linear approximation, and generates a visualization map of each intermediate layer based on the contribution of each intermediate layer.

[0003] "Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization", Submitted on 7 October 2016 (v1), last revised on 3 December 2019 (this version, v4), [Retrieved April 25, 2024], Internet: https: / / arxiv.org / abs / 1610.02391

[0004] Although devices that use Grad-CAM to generate visualization maps for each intermediate layer linearly approximate the model function, there is a problem in that the difference between the linear function, which is the model function after linear approximation, and the model function can become large, which can result in a deterioration in the accuracy of generating visualization maps for each intermediate layer.

[0005] The present disclosure has been made to solve the above-mentioned problems, and aims to provide a visualization device that can suppress deterioration in the accuracy of generating a visualization map.

[0006] A visualization device according to the present disclosure includes a function acquisition unit that acquires a model function, which is a function that models the operations of multiple intermediate layers included in a neural network that outputs an analysis result of the input data from an output layer when input data is provided to an input layer, a contribution calculation unit that calculates the contribution of each intermediate layer to the analysis result based on the model function acquired by the function acquisition unit, and a map generation unit that generates a visualization map of each intermediate layer based on the contribution calculated by the contribution calculation unit. When the number of multiple intermediate layers is M (M is an integer equal to or greater than 2), the contribution calculation unit repeatedly performs an update process that updates the value of an m-th (m = 1, ..., M)-th intermediate layer from the input layer side based on the model function, and calculates the contribution of the m-th intermediate layer based on the step interval between the s-th (s is an integer equal to or greater than 1)-th update process step and the (s + 1)-th update process step and the gradient related to the value of the m-th intermediate layer after the s-th update process.

[0007] According to the present disclosure, it is possible to suppress deterioration in the accuracy of generating a visualization map.

[0008] 1 is a block diagram showing a visualization device according to a first embodiment; FIG. 2 is a hardware block diagram showing hardware of the visualization device according to the first embodiment; FIG. 3 is a hardware block diagram of a computer when the visualization device is realized by software, firmware, or the like; FIG. 4 is a flowchart showing a visualization method, which is a processing procedure of the visualization device; FIG. 5 is an explanatory diagram showing how image data I to be analyzed is provided to an input layer as input data, and the probability of a class to which an object shown in an image represented by the image data I belongs is output from an output layer; and FIG. 6 is an explanatory diagram showing x, y, and c of a 3D tensor H[x, y, c]. ｍ 1 is an explanatory diagram showing intermediate results of the model function f １・・・Ｍ FIG. 10 is an explanatory diagram showing an example of a model function f １・・・ＭFIG. 1 is an explanatory diagram showing the contribution calculation in multiple steps of (I) and the contribution calculation by the conventional method. ｍ Visualization map C ｐｏｉｎｔ FIG. 10 is an explanatory diagram showing a display example of (x, y). FIG. 11 is a configuration diagram showing a visualization device according to embodiment 2. FIG. 12 is a hardware configuration diagram showing hardware of the visualization device according to embodiment 2. FIG. 13 is a configuration diagram showing a visualization device according to embodiment 3. FIG. 14 is a hardware configuration diagram showing hardware of the visualization device according to embodiment 3. FIG. 15 is an explanatory diagram showing a display example of a visualization map C(c) of a plurality of channels c in an intermediate layer H.

[0009] In order to explain the present disclosure in more detail, embodiments of the present disclosure will be described below with reference to the accompanying drawings.

[0010] Embodiment 1. FIG. 1 is a configuration diagram showing a visualization device according to embodiment 1. FIG. 2 is a hardware configuration diagram showing hardware of the visualization device according to embodiment 1. The visualization device shown in FIG. 1 includes a function acquisition unit 1, a contribution calculation unit 2, a map generation unit 3, and a display processing unit 4. The neural network applied to the visualization device shown in FIG. 1 includes, for example, a convolutional neural network (CNN). When input data is provided to the input layer of a neural network, the neural network outputs an analysis result of the input data from the output layer. If the input data is image data, the analysis result of the input data is, for example, the probability of a class to which each of one or more objects shown in an image represented by the image data belongs.

[0011] The function acquisition unit 1 is realized by, for example, a function acquisition circuit 11 shown in Fig. 2. The function acquisition unit 1 acquires a model function, which is a function that models the operations of multiple intermediate layers of a neural network. The function acquisition unit 1 outputs the model function to the contribution calculation unit 2.

[0012] The contribution calculation unit 2 is realized, for example, by the contribution calculation circuit 12 shown in FIG. 2 . The contribution calculation unit 2 acquires a model function from the function acquisition unit 1. The contribution calculation unit 2 calculates the contribution of each intermediate layer to the analysis result of the input data based on the model function. Specifically, when the number of intermediate layers is M (M is an integer equal to or greater than 2), the contribution calculation unit 2 repeatedly performs an update process to update the value of the mth intermediate layer (m = 1, ..., M) from the input layer side based on the model function. The contribution calculation unit 2 calculates the contribution of the mth intermediate layer based on the step interval between the sth update process step (s is an integer equal to or greater than 1) and the (s + 1)th update process step and the gradient related to the value of the mth intermediate layer after the sth update process. The contribution calculation unit 2 outputs the contribution of each intermediate layer to the analysis result of the input data to the map generation unit 3.

[0013] The map generation unit 3 is realized by, for example, the map generation circuit 13 shown in FIG. 2 . The map generation unit 3 acquires the contribution of each intermediate layer from the contribution calculation unit 2. The map generation unit 3 generates a visualization map of each intermediate layer based on the contribution of each intermediate layer. The map generation unit 3 outputs the visualization map of each intermediate layer to the display processing unit 4.

[0014] The display processing unit 4 is realized by, for example, the display processing circuit 14 shown in Fig. 2. The display processing unit 4 acquires the visualization maps of the respective intermediate layers from the map generating unit 3. The display processing unit 4 displays the visualization maps of the respective intermediate layers on a display (not shown) or the like.

[0015] 1, it is assumed that each of the components of the visualization device, that is, a function acquisition unit 1, a contribution calculation unit 2, a map generation unit 3, and a display processing unit 4, is realized by dedicated hardware as shown in Fig. 2. That is, it is assumed that the visualization device is realized by a function acquisition circuit 11, a contribution calculation circuit 12, a map generation circuit 13, and a display processing circuit 14. Each of the function acquisition circuit 11, the contribution calculation circuit 12, the map generation circuit 13, and the display processing circuit 14 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a combination thereof.

[0016] The components of the visualization device are not limited to those realized by dedicated hardware, and the visualization device may be realized by software, firmware, or a combination of software and firmware. The software or firmware is stored in the memory of a computer as a program. The computer refers to hardware that executes a program, and includes, for example, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a central processing unit, a processing unit, an arithmetic unit, a microprocessor, a microcomputer, a processor, or a DSP (Digital Signal Processor).

[0017] 3 is a hardware configuration diagram of a computer when the visualization device is realized by software, firmware, etc. When the visualization device is realized by software, firmware, etc., a program for causing a computer to execute the respective processing procedures of the function acquisition unit 1, the contribution calculation unit 2, the map generation unit 3, and the display processing unit 4 is stored in a memory 21. Then, a processor 22 of the computer executes the program stored in the memory 21.

[0018] 2 shows an example in which each of the components of the visualization device is realized by dedicated hardware, while Fig. 3 shows an example in which the visualization device is realized by software, firmware, etc. However, this is merely an example, and some of the components in the visualization device may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0019] Next, the operation of the visualization device shown in Fig. 1 will be described. Fig. 4 is a flowchart showing a visualization method, which is a processing procedure of the visualization device. Fig. 5 is an explanatory diagram showing how image data I to be analyzed is provided as input data to an input layer, and the probability of a class to which an object shown in the image represented by the image data I belongs is output from the output layer. In the example of Fig. 5, when the object shown in the image represented by the image data I is a motorcycle, the class probabilities output from the output layer include the probability that the object is a motorcycle, the probability that the object is a dog, the probability that the object is a cat, the probability that the object is a bicycle, etc.

[0020] In FIG. 5, the input layer is omitted, and Y is the output layer that outputs the probability of each class. １ , f ２ , ..., f Ｍ is the hidden layer H in the neural network ｍ (m=1, ..., M), where M is an integer equal to or greater than 2. For example, if there are M hidden layers H １ ~H Ｍ A model function f is a function in which the operation of １・・・Ｍ (I) is expressed as the following formula (1): f １・・・Ｍ (I) = f Ｍ (...f ３ (f ２ (f １ (I)))) (1) f １ (I) is the intermediate layer H １ is a model function of the operation at f １ The value of (I) is the intermediate layer H １ This is the intermediate result at f２ (f １ (I)) is f ２ (H １ ) and f ２ (H １ ) is the value of the intermediate layer H ２ This is the intermediate result at f ３ (f ２ (f １ (I))) is f ３ (H ２ ) and f ３ (H ２ ) is the value of the intermediate layer H ３ This is the intermediate result at f ｍ (H ｍ－１ ) is the value of the mth hidden layer H ｍ is an intermediate result of m=1,...,M, and H ０ =I.

[0021] If the neural network is, for example, a CNN, the hidden layer H ｍ The intermediate result of the above is represented by a 3D tensor H[x, y, c] as shown in Fig. 6. Fig. 6 is an explanatory diagram showing x, y, and c of the 3D tensor H[x, y, c]. In Fig. 6, x is the horizontal axis of the image, y is the vertical axis of the image, and c is a channel corresponding to one or more objects appearing in the image. Each channel can be represented by a 2D array h[x, y, 1], h[x, y, 2], ..., h[x, y, n], where n is the number of objects appearing in the image. When the number of channels is n, the intermediate layer H ｍ The intermediate result in f ｍ (H ｍ－１ ) is calculated by the weight w １ , w ２ , ..., w ｎ The m-th hidden layer H ｍ FIG. 10 is an explanatory diagram showing an intermediate result in

[0022] The function acquisition unit 1 acquires the function from a plurality of intermediate layers H １ ~H Ｍ A model function f is a function in which the operation of１・・・Ｍ (I) (step ST1 in FIG. 4). The function acquisition unit 1 acquires the model function f １・・・Ｍ (I) is output to the contribution calculation unit 2. １・・・Ｍ 8 is an explanatory diagram showing an example of (I). In FIG. 8, the horizontal axis represents the intermediate layer H=(H １ , H ２ , H ３ , H ４ , H ５ , H ６ , ...), and the vertical axis represents the model function f １・・・Ｍ (I) shows the value f(H).

[0023] The contribution calculation unit 2 acquires the model function f(H) from the function acquisition unit 1. Based on the model function f(H), the contribution calculation unit 2 calculates the contribution DC of the m-th (m=1, ..., M)-th hidden layer H to the analysis result of the input data (step ST2 in FIG. 4). The contribution calculation unit 2 outputs the contribution DC of the m-th hidden layer H to the analysis result to the map generation unit 3.

[0024] The calculation process of the contribution degree DC by the contribution degree calculation unit 2 will be specifically described below. ｍ is given, the model function f Ｍ＋１・・・ＬＡ (H ｍ ) based on the intermediate layer H ｍ In the following, for the sake of simplicity, we will use the model function f Ｍ＋１・・・ＬＡ (H ｍ ) subscript Ｍ＋１・・・ＬＡ And, H ｍ The subscript m is omitted. ｍ is represented by H, and f Ｍ＋１・・・ＬＡ Let () be represented by f(). When image data I is given to the neural network, all hidden layers H in the neural network have some value. The initial value of the hidden layer H is, for example, h ０ In the visualization device shown in FIG. 1, in the algorithm for calculating the contribution of the hidden layer H, the initial value h ０ Is, h １ , h ２ , ... and so on.

[0025] The value of the hidden layer H is updated as shown in the following formulas (2) to (5), so that the contribution of the hidden layer H can be calculated. In the formulas, s is a variable that indicates the step of the update process, and means that it is the sth update process. ｓ From h ｓ＋１ To update to ｓ In this case, the model function f(H) is changed to f(H=h ｓ ) to the lower value f(H=h ｓ＋１ ) can be quickly reduced to ｓ The gradient f′(H) with respect to is expressed as in the following equation (2): The value of the hidden layer H is updated as in the following equation (3).

[0026] In formula (3), σ ｓ is the step interval, which represents how far H can move.

[0027] According to the following formulas (4) and (5), h ｓ Each element of f(H = h ｓ ) and f(H = h ｓ＋１ ) and the contribution to the difference between the two. ｓ This can be done based on the Taylor expansion of f(H) in the vicinity.

[0028]

[0029] The above algorithm for calculating the contribution of the hidden layer H calculates the contribution of any hidden layer H in the neural network. If the user wants to calculate the contribution of another hidden layer H, the user must specify the other hidden layer H and execute the above algorithm again. The above algorithm calculates the contribution of the hidden layer H by f(H = h ｓ ) is less than a user-specified threshold or s is greater than a user-specified threshold.

[0030] The contribution DC of the m-th hidden layer H is expressed by the following formula (6): As shown in formula (6), the contribution calculation unit 2 calculates the step interval σ between the s-th update processing step and the (s+1)-th update processing step. ｓ and the gradient f′ (H=h ｓ ) and based on the contribution DC ｓ Calculate.

[0031] In equation (6), i is a variable indicating an element of H=(H[1], H[2], ...). The element corresponds to an object appearing in the image.

[0032] FIG. 9 shows the model function f １・・・Ｍ 1 is an explanatory diagram showing the contribution calculation in multiple steps of (I) and the contribution calculation by the conventional method. １・・・Ｍ In the examples of FIGS. 8 and 9, the contribution calculation unit 2 calculates the contribution by linear approximation once in the first step h １ and the second step h ２ Distance σ １ and the gradient f' of the first hidden layer H (H = h １ ) and the first step h １ and the second step h ２ Contribution DC between １ The contribution calculation unit 2 calculates the second step h ２ and the third step h ３ Distance σ ２ and the gradient f' of the second hidden layer H (H = h ２ ) and the second step h ２ and the third step h ３ Contribution DC between ２ Similarly, the contribution calculation unit 2 calculates the third step h ３ and the fourth step h ４ Contribution DC between ３ Calculate the fourth step h ４ and the fifth step h ５ Contribution DC between ４ Calculate the fifth step h５ and the sixth step h ６ Contribution DC between ５ The contribution calculation unit 2 calculates the contribution DC １ , D.C. ２ , D.C. ３ ...and calculate the overall contribution DC. DC = DC １ +DC ２ +DC ３ + ... (7)

[0033] In addition, the intermediate layer H ｍ The intermediate result of is expressed as a 3D tensor H[x, y, c], so the contribution calculation unit 2 calculates the intermediate layer H ｍ The contribution DC[c] of the hidden layer H is calculated for each pixel (x, y). Hereinafter, the contribution of each pixel (x, y) in each channel c is represented by DC[x, y, c].

[0034] The map generation unit 3 acquires the contribution DC[x, y, c] of the intermediate layer H from the contribution calculation unit 2. The map generation unit 3 generates a visualized map C of the intermediate layer H based on the contribution DC[x, y, c] of the intermediate layer H. ｐｏｉｎｔ Specifically, the map generator 3 calculates the sum of contributions DC[x, y, c] of the same pixel (x, y) of multiple channels c in the intermediate layer H as shown in the following equation (8), thereby generating the visualized map C ｐｏｉｎｔ The map generator 3 generates a visualized map C of the intermediate layer H. ｐｏｉｎｔ (x, y) is output to the display processing unit 4.

[0035]

[0036] The display processing unit 4 receives the visualized map C of the intermediate layer H from the map generating unit 3. ｐｏｉｎｔ The display processing unit 4 obtains the visualized map C of the intermediate layer H as shown in FIG. ｐｏｉｎｔ (x, y) is displayed on a display (not shown) or the like (step ST4 in FIG. 4). ｍVisualization map C ｐｏｉｎｔ FIG. 10 is an explanatory diagram showing a display example of (x, y).

[0037] In the first embodiment described above, the visualization device is configured to include a function acquisition unit 1 that acquires a model function, which is a function that models the operations of multiple intermediate layers included in a neural network that outputs an analysis result of the input data from an output layer when input data is provided to an input layer; a contribution calculation unit 2 that calculates the contribution of each intermediate layer to the analysis result based on the model function acquired by the function acquisition unit 1; and a map generation unit 3 that generates a visualization map of each intermediate layer based on the contribution calculated by the contribution calculation unit 2. When the number of multiple intermediate layers is M (M is an integer equal to or greater than 2), the contribution calculation unit 2 repeatedly performs an update process that updates the value of the m-th (m = 1, ..., M)-th intermediate layer from the input layer side based on the model function, and calculates the contribution of the m-th intermediate layer based on the step interval between the s-th (s is an integer equal to or greater than 1)-th update process step and the (s + 1)-th update process step and the gradient related to the value of the m-th intermediate layer after the s-th update process. Therefore, the visualization device can suppress deterioration in the accuracy of generating the visualization map.

[0038] Second Embodiment In a second embodiment, a visualization device will be described in which a contribution calculation unit 5 calculates the step interval between the sth update processing step and the (s+1)th update processing step using the value of the mth hidden layer.

[0039] FIG. 11 is a configuration diagram showing a visualization device according to embodiment 2. In FIG. 11, the same reference numerals as in FIG. 1 indicate the same or equivalent parts, and detailed description thereof will be omitted. FIG. 12 is a hardware configuration diagram showing hardware of the visualization device according to embodiment 2. In FIG. 12, the same reference numerals as in FIG. 2 indicate the same or equivalent parts, and detailed description thereof will be omitted. The visualization device shown in FIG. 11 includes a function acquisition unit 1, a contribution calculation unit 5, a map generation unit 3, and a display processing unit 4.

[0040] The contribution calculation unit 5 is realized by, for example, a contribution calculation circuit 15 shown in FIG. 12. The contribution calculation unit 5 acquires the gradient f'(H) of the model function f(H) in the m-th hidden layer H from the function acquisition unit 1. The contribution calculation unit 5 calculates the gradient f'(H) of the model function f(H) in the m-th hidden layer H as the step interval between the s-th update processing step and the (s+1)-th update processing step, using the m-th hidden layer H and the gradient f'(H) of the model function f(H) in the m-th hidden layer H. ｓ and the (s+1)th step h ｓ＋１ Step interval σ ｓ The contribution calculation unit 5 calculates the calculated step interval σ ｓ and the gradient f′(H=hs) of the model function f(H) in the m-th hidden layer H, the contribution DC is calculated.

[0041] 11, it is assumed that each of the components of the visualization device, namely, the function acquisition unit 1, the contribution calculation unit 5, the map generation unit 3, and the display processing unit 4, is realized by dedicated hardware as shown in Fig. 12. That is, it is assumed that the visualization device is realized by a function acquisition circuit 11, a contribution calculation circuit 15, a map generation circuit 13, and a display processing circuit 14. Each of the function acquisition circuit 11, the contribution calculation circuit 15, the map generation circuit 13, and the display processing circuit 14 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0042] The components of the visualization device are not limited to those realized by dedicated hardware, and the visualization device may be realized by software, firmware, or a combination of software and firmware. When the visualization device is realized by software, firmware, or the like, programs for causing a computer to execute the respective processing procedures of the function acquisition unit 1, the contribution calculation unit 5, the map generation unit 3, and the display processing unit 4 are stored in the memory 21 shown in Fig. 3. Then, the processor 22 shown in Fig. 3 executes the programs stored in the memory 21.

[0043] 12 shows an example in which each of the components of the visualization device is realized by dedicated hardware, while Fig. 3 shows an example in which the visualization device is realized by software, firmware, etc. However, this is merely an example, and some of the components in the visualization device may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0044] Next, the operation of the visualization device shown in Fig. 11 will be described. However, except for the contribution calculation unit 5, the visualization device is the same as that shown in Fig. 1. Therefore, only the operation of the contribution calculation unit 5 will be described here. ｓ and the (s+1)th step h ｓ＋１ Step interval σ ｓ If is too wide, f(H=h ｓ ) and f(H = h ｓ＋１ ) becomes larger, and the sth step h ｓ and the (s+1)th step h ｓ＋１ Contribution DC between ｓ On the other hand, the calculation accuracy of the sth step h ｓ and the (s+1)th step h ｓ＋１ Step interval σ ｓ If is too narrow, f' (H = h ｓ ) to h ｓ Underflow can occur when adding to

[0045] The contribution calculation unit 5 receives from the function acquisition unit 1 the gradient f′(H=h) of the model function f(h) in the m-th hidden layer H. ｓ The contribution calculation unit 5 obtains the sth step h ｓ and the (s+1)th step h ｓ＋１ Step interval σ ｓ Calculate.

[0046] In equation (9), g is σ ｓ is a constant less than 1 to prevent from becoming too large.

[0047] The contribution calculation unit 5 calculates the step interval σｓ After calculating the contribution degree, the sth step h ｓ Contribution of DC ｓ The contribution calculation unit 5 calculates the contribution at the sth step h ｓ Contribution of DC ｓ to the map generating unit 3. As described above, the contribution calculating unit 5 calculates the step interval σ ｓ By calculating the step interval σ ｓ This prevents underflow due to too narrow step interval σ ｓ This prevents the accuracy of calculating the contribution DC from being degraded due to the range being too wide.

[0048] In the second embodiment described above, the visualization device is configured so that the contribution calculation unit 5 calculates the step interval between the sth update processing step and the (s+1)th update processing step using the value of the mth hidden layer. Therefore, the visualization device can prevent underflow, which is caused by treating the sth step and the (s+1)th step as the same hidden layer, while suppressing deterioration in the calculation accuracy of the contribution of the hidden layer.

[0049] Third Embodiment In a third embodiment, a visualization device will be described in which a map generation unit 6 generates a visualization map of a plurality of channels in each hidden layer based on the contribution calculated by the contribution calculation unit 2.

[0050] FIG. 13 is a configuration diagram showing a visualization device according to embodiment 3. In FIG. 13, the same reference numerals as those in FIGS. 1 and 11 indicate the same or corresponding parts, and detailed description thereof will be omitted. FIG. 14 is a hardware configuration diagram showing hardware of the visualization device according to embodiment 3. In FIG. 14, the same reference numerals as those in FIGS. 2 and 12 indicate the same or corresponding parts, and detailed description thereof will be omitted. The visualization device shown in FIG. 13 includes a function acquisition unit 1, a contribution calculation unit 2, a map generation unit 6, and a display processing unit 4.

[0051] The map generation unit 6 is realized by, for example, the map generation circuit 16 shown in FIG. 14 . The map generation unit 6 acquires the contribution of each hidden layer from the contribution calculation unit 2. Similar to the map generation unit 3 shown in FIG. 1 , the map generation unit 6 generates a visualization map of each hidden layer based on the contribution of each hidden layer. The map generation unit 6 generates a visualization map of a plurality of channels in each hidden layer based on the contribution of each hidden layer. The map generation unit 6 outputs the visualization map of each hidden layer and the visualization map of each channel to the display processing unit 4.

[0052] In the visualization device shown in Fig. 13, the map generation unit 6 is applied to the visualization device shown in Fig. 1. However, this is merely an example, and the map generation unit 6 may also be applied to the visualization device shown in Fig. 11.

[0053] 13, it is assumed that each of the components of the visualization device, namely, the function acquisition unit 1, the contribution calculation unit 2, the map generation unit 6, and the display processing unit 4, is realized by dedicated hardware as shown in Fig. 14. That is, it is assumed that the visualization device is realized by a function acquisition circuit 11, a contribution calculation circuit 12, a map generation circuit 16, and a display processing circuit 14. Each of the function acquisition circuit 11, the contribution calculation circuit 12, the map generation circuit 16, and the display processing circuit 14 corresponds to, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC, an FPGA, or a combination thereof.

[0054] The components of the visualization device are not limited to those realized by dedicated hardware, and the visualization device may be realized by software, firmware, or a combination of software and firmware. When the visualization device is realized by software, firmware, or the like, programs for causing a computer to execute the respective processing procedures of the function acquisition unit 1, the contribution calculation unit 2, the map generation unit 6, and the display processing unit 4 are stored in a memory 21 shown in Fig. 3. Then, a processor 22 shown in Fig. 3 executes the programs stored in the memory 21.

[0055] 14 shows an example in which each of the components of the visualization device is realized by dedicated hardware, while Fig. 3 shows an example in which the visualization device is realized by software, firmware, etc. However, this is merely an example, and some of the components in the visualization device may be realized by dedicated hardware, and the remaining components may be realized by software, firmware, etc.

[0056] Next, the operation of the visualization device shown in Fig. 13 will be described. However, apart from the map generation unit 6, the visualization device is the same as that shown in Fig. 1. Therefore, the operation of the map generation unit 6 will be mainly described here.

[0057] The map generator 6 acquires the contribution DC(x, y, c) of the m-th hidden layer H from the contribution calculator 2. Similar to the map generator 3 shown in FIG. 1 , the map generator 6 generates a visualized map C of the hidden layer H based on the contribution DC(x, y, c) of the m-th hidden layer H. ｐｏｉｎｔ Generate (x, y).

[0058] Furthermore, the map generating unit 6 generates a visualization map C(c) of a plurality of channels c in the m-th hidden layer H, as shown in the following formula (10), based on the contribution DC(x, y, c) of the m-th hidden layer H. The map generating unit 6 generates the visualization map C(c) of the hidden layer H ｐｏｉｎｔ (x, y) and a visualization map C(c) of multiple channels c in the m-th hidden layer H are output to the display processing unit 4.

[0059]

[0060] The display processing unit 4 receives the visualized map C of the m-th (m=1, . . . , M) intermediate layer H from the map generating unit 6. ｐｏｉｎｔ (x, y) and a visualization map C(c) of a plurality of channels c in the m-th hidden layer H. As shown in FIG. 10 , the display processing unit 4 obtains the visualization map C ｐｏｉｎｔ(x, y) on a display or the like (not shown). Furthermore, the display processing unit 4 displays a visualization map C(c) of a plurality of channels c in the m-th intermediate layer H on a display or the like (not shown), as shown in FIG. 15 . FIG. 15 is an explanatory diagram showing a display example of the visualization map C(c) of a plurality of channels c in the intermediate layer H. In the example of FIG. 15 , c=1, 2, 3. FIG. 15 shows an example in which an object shown in an image indicated by image data I is a car.

[0061] In the above-described third embodiment, the visualization device is configured so that the map generation unit 6 generates a visualization map of each intermediate layer, and also generates a visualization map of multiple channels in each intermediate layer based on the contribution calculated by the contribution calculation unit 2. Therefore, the visualization device can allow the user to check the visualization map of multiple channels in each intermediate layer.

[0062] In addition, the present disclosure allows for free combination of the respective embodiments, modification of any of the components of the respective embodiments, or omission of any of the components of the respective embodiments.

[0063] The visualization device of the present disclosure can be used as a device for generating a visualization map that visualizes the contribution of each hidden layer to the analysis results of a neural network having multiple hidden layers.

[0064] REFERENCE SIGNS LIST 1 Function acquisition unit, 2, 5 Contribution calculation unit, 3, 6 Map generation unit, 4 Display processing unit, 11 Function acquisition circuit, 12, 15 Contribution calculation circuit, 13, 16 Map generation circuit, 14 Display processing circuit, 21 Memory, 22 Processor.

Claims

1. A visualization device comprising: a function acquisition unit that acquires a model function, which is a function that models the operations of multiple intermediate layers of a neural network that outputs an analysis result of the input data from an output layer when input data is provided to an input layer; a contribution calculation unit that calculates the contribution of each intermediate layer to the analysis result based on the model function acquired by the function acquisition unit; and a map generation unit that generates a visualization map of each intermediate layer based on the contribution calculated by the contribution calculation unit, wherein when the number of multiple intermediate layers is M (M is an integer of 2 or more), the contribution calculation unit repeatedly performs an update process that updates the value of the m-th (m = 1, ..., M) intermediate layer from the input layer side based on the model function, and calculates the contribution of the m-th intermediate layer based on the step interval between the s-th (s is an integer of 1 or more) update process step and the (s + 1)-th update process step and the gradient related to the value of the m-th intermediate layer after the s-th update process.

2. The visualization device described in claim 1, characterized in that the mth intermediate layer has a plurality of elements, and the contribution calculation unit calculates the contribution of each element to the mth intermediate layer, and calculates the contribution of the mth intermediate layer from the contribution of each element.

3. The visualization device described in claim 1 or claim 2, characterized in that the contribution calculation unit calculates the step interval between the sth update processing step and the (s+1)th update processing step based on the value of the mth hidden layer.

4. A visualization device as claimed in any one of claims 1 to 3, characterized in that the map generation unit, in addition to generating a visualization map for each intermediate layer, generates a visualization map for multiple channels in each intermediate layer based on the contribution calculated by the contribution calculation unit.

5. The visualization device according to any one of claims 1 to 4, further comprising a display processing unit that displays the visualization map generated by the map generation unit.

6. A visualization device according to any one of claims 1 to 5, characterized in that the input data is image data, and the analysis result of the input data is the probability of a class to which an object shown in an image represented by the image data belongs.

7. A visualization method comprising: a function acquisition unit acquires a model function that models the operations of multiple intermediate layers of a neural network that, when input data is given to an input layer, outputs an analysis result of the input data from an output layer; a contribution calculation unit calculates the contribution of each intermediate layer to the analysis result based on the model function acquired by the function acquisition unit; and a map generation unit generates a visualization map of each intermediate layer based on the contribution calculated by the contribution calculation unit, wherein when the number of multiple intermediate layers is M (M is an integer of 2 or more), the contribution calculation unit repeatedly performs an update process that updates the value of the m-th (m = 1, ..., M) intermediate layer from the input layer side based on the model function, and calculates the contribution of the m-th intermediate layer based on the step interval between the s-th (s is an integer of 1 or more) update process step and the (s + 1)-th update process step and the gradient related to the value of the m-th intermediate layer after the s-th update process.

Images