Information processing apparatus, information processing method, computer-readable non-volatile storage medium, and terminal apparatus

Abstract

An information processing device is equipped with a control unit. The control unit evaluates equivalence between the first learning model before the occurrence of the weight reduction according to the weight reduction method and the second learning model after the occurrence of the weight reduction according to the weight reduction method. The control unit determines a second learning model based on the evaluation result.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, an information processing method, a computer-readable non-volatile storage medium, and a terminal device. Background Art

[0002] As artificial intelligence (AI) becomes more popular, providing different services using AI models such as convolutional neural network (CNN) models or deep neural network (DNN) models is expanding.

[0003] Generally, it is known that the operation amount of the AI ​​model is large and the load on the processor is high. Therefore, in order to reduce the load imposed on the processor, a technology for reducing the operation amount of the AI ​​model is known.

[0004] Reference List

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Publication No. 2022-14569 Summary of the invention

[0007] Technical issues

[0008] The recognition accuracy is mainly used as an indicator of the weight reduction of the AI ​​model. For example, the weight reduction of the AI ​​model is performed so that the recognition accuracy of the AI ​​model after the weight reduction is greater than a predetermined value.

[0009] However, in the conventional weight reduction process, although the performance of the AI ​​model after the weight reduction is evaluated, the change of the AI ​​model before and after the weight reduction is not evaluated. In other words, there is a possibility that the AI ​​model itself changes before and after the weight reduction.

[0010] Therefore, the present disclosure provides a mechanism that can further control changes in the AI ​​model before and after the weight reduction process.

[0011] Note that the above problem or object is only one of multiple problems or objects that can be solved or achieved by multiple embodiments disclosed in this specification.

[0012] Solution to the problem

[0013] The information processing device of the present disclosure includes a control unit. The control unit evaluates equivalence between a first learning model before weight reduction by a weight reduction method and a second learning model after weight reduction by the weight reduction method. The control unit determines the second learning model based on the evaluation result. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a view for describing an example of a weight reduction process according to the proposed technology of the present disclosure.

[0015] Figure 2 is a diagram for describing an example of weight reduction processing.

[0016] Figure 3 It is a view for describing the feature quantity extracted by the neurons of the pre-compression model.

[0017] Figure 4 It is a view for describing the feature quantity extracted by the neurons of the post-compression model.

[0018] Figure 5 is a diagram for describing another example of the weight reduction process.

[0019] Figure 6 is a block diagram showing a configuration example of an information processing device according to an embodiment of the present disclosure.

[0020] Figure 7 is a diagram showing an example of a first XAI process according to an embodiment of the present disclosure.

[0021] Figure 8 : is a view showing an example of a first equivalence evaluation process according to an embodiment of the present disclosure.

[0022] Fig. 9 is a diagram illustrating another example of the first equivalence evaluation process according to an embodiment of the present disclosure.

[0023] Fig.10 is a view showing an example of a second XAI process according to an embodiment of the present disclosure.

[0024] Fig.11 : is a view showing an example of a second equivalence evaluation process according to an embodiment of the present disclosure.

[0025] Fig.12 is a view showing an example of a third XAI process according to an embodiment of the present disclosure.

[0026] Fig.13 : is a view showing an example of a third equivalence evaluation process according to an embodiment of the present disclosure.

[0027] Fig.14 : is a view showing an example of a third XAI process and a third equivalence evaluation process according to an embodiment of the present disclosure.

[0028] Fig.15 is a diagram illustrating another example of the third XAI process and the third equivalence evaluation process according to an embodiment of the present disclosure.

[0029] Fig.16is a view for describing an example of a first determination process according to an embodiment of the present disclosure.

[0030] Fig.17 is a diagram for describing another example of the first determination process according to an embodiment of the present disclosure.

[0031] Fig.18 is a diagram for describing another example of the first determination process according to an embodiment of the present disclosure.

[0032] Fig.19 : is a flowchart showing an example of the flow of the second determination process according to the embodiment of the present disclosure.

[0033] Fig. 20 is a diagram for describing another example of the second determination process according to an embodiment of the present disclosure.

[0034] Fig.21 2 is a view for describing an example of a third determination process according to an embodiment of the present disclosure.

[0035] Fig. 22 is a diagram for describing another example of the third determination process according to the embodiment of the present disclosure.

[0036] Fig.23 is a diagram for describing another example of the third determination process according to the embodiment of the present disclosure.

[0037] Fig.24 is a diagram for describing another example of the third determination process according to the embodiment of the present disclosure.

[0038] Fig.25 It is a view for describing an example of presentation information presented by the information processing device according to an embodiment of the present disclosure.

[0039] Fig.26 : is a view showing an example of a presentation image presented by the information processing apparatus according to the embodiment of the present disclosure.

[0040] Fig. 27 is a view for describing another example of presentation information presented by the information processing apparatus according to an embodiment of the present disclosure.

[0041] Fig.28 : is a flowchart showing an example of the flow of a model generation process according to an embodiment of the present disclosure.

[0042] Fig.29 is a diagram showing a configuration example of an information processing system according to an embodiment of the present disclosure.

[0043] Fig.30It is a view for describing an example of an application function registration method according to an embodiment of the present disclosure.

[0044] Fig.31 is a sequence diagram showing an example of the flow of information processing performed in the information processing system according to the embodiment of the present disclosure.

[0045] Fig.32 is a sequence diagram showing an example of the flow of information processing performed in the information processing system according to the embodiment of the present disclosure.

[0046] Fig.33 It is a view for describing the connection between a cloud-side information processing device and an edge-side information processing device according to an embodiment of the present disclosure.

[0047] Fig.34 : is a block diagram showing a configuration example of a cloud-side information processing device according to an embodiment of the present disclosure.

[0048] Fig.35 is a block diagram showing an internal configuration example of a camera according to an embodiment of the present disclosure.

[0049] Fig.36 is a view showing an example of the structure of an image sensor according to an embodiment of the present disclosure.

[0050] Fig.37 is a view showing a deployment instance of an AI model and an AI application according to an embodiment of the present disclosure.

[0051] Fig.38 is a diagram showing a configuration example of a cluster according to an embodiment of the present disclosure.

[0052] Fig.39 is a diagram showing an example of a hardware configuration of an information processing device according to an embodiment of the present disclosure.

[0053] Fig.40 It is a view for describing an example of the flow of a relearning process / update process according to an embodiment of the present disclosure.

[0054] Fig.41 is a view showing an example of a login screen of a marketplace according to an embodiment of the present disclosure.

[0055] Fig.42 is a view illustrating an example of a market developer screen according to an embodiment of the present disclosure.

[0056] Fig.43 is a view illustrating an example of a market user screen according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0057] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in this specification and the accompanying drawings, substantially the same elements are denoted by the same reference numerals, and repeated descriptions will be omitted.

[0058] Furthermore, although specific values ​​may be used for description in the present specification and the drawings, these values ​​are merely examples and other values ​​may be applied.

[0059] Each of one or more embodiments (including examples and variations) described below can be performed independently. On the other hand, at least a portion of the multiple embodiments described below can be appropriately combined and performed with at least a portion of other embodiments. The multiple embodiments may include novel features that are different from each other. Thus, multiple embodiments may help solve different goals or problems, and may show different effects from each other.

[0060] <<1. Introduction>>

[0061] <1.1. Overview of the proposed method>

[0062] As described above, with the popularization of AI, various services using learning models (also referred to as AI models) generated by machine learning are provided.

[0063] The learning model is deployed in, for example, a personal computer (PC) or a terminal device (such as a camera other than a cloud server) and is used for image recognition processing, etc. In the case where the learning model is deployed in a PC or a terminal device, there is a case where a weight reduction process is performed on the learning model in order to reduce the operation amount and size of the learning model.

[0064] Hereinafter, the learning model before the weight reduction process may be referred to as a pre-compression model M0 , and the learning model after the weight reduction process may be referred to as a post-compression model M1 .

[0065] When the weight reduction process is performed in this way, it is desirable to confirm whether the performance of the learning model before and after the process is equivalent to a certain extent. Alternatively, it is desirable to generate a post-compression model M1 having a certain degree of performance of the learning model before and after the process.

[0066] It should be noted that the degree of equivalence of performance before and after the weight reduction process varies depending on the process using the learning model (such as image recognition process, speech recognition process, etc.).

[0067] In the past, the weight reduction process was performed using the recognition accuracy of the post-compression model M1 as an indicator. That is, the weight reduction process was performed in such a way that the recognition accuracy of the post-compression model M1 became higher. However, in the conventional weight reduction process, whether the learning model has changed before and after the process is not considered.

[0068] Therefore, in the technology proposed in the present disclosure, the information processing apparatus evaluates the changes in the learning model before and after the weight reduction process to be equivalent.

[0069] For example, the information processing device evaluates the equivalence between the pre-compression model M0 (an instance of the first learning model) before weight reduction by the weight reduction method and the post-compression model M1 (an instance of the second learning model) after weight reduction by the weight reduction method. The information processing device decides the post-compression model M1 (also referred to as the final model MF) based on the evaluation result.

[0070] Figure 1 is a diagram for describing an example of a weight reduction process according to the technology proposed in the present disclosure. Figure 1 The weight reduction process shown in .

[0071] like Figure 1 As shown, the information processing device performs pre-compression model M0 1 Perform model weight reduction (step S1) and generate the post-compression model M1 1 .

[0072] Then, the information processing device evaluates the pre-compression model M0 1 and post-compression model M1 1 The information processing device determines the post-compression model M1 based on the result of the equivalence evaluation. 1 (Final model MF) (Step S3).

[0073] Here, equivalence means that, for example, the pre-compression model M0 is 1 and post-compression model M1 1 An index of how much is the same (unchanged). Equivalence can be calculated by utilizing, for example, exchangeable AI (XAI) techniques.

[0074] <1.2. Question>

[0075] Here, the problem of the weight reduction process will be described in detail. As above, in the conventional weight reduction process, the equivalence of the learning model is not considered before and after the process.

[0076] Figure 2 is a diagram for describing an example of weight reduction processing. Figure 2 As shown, for example, a weight reduction process can be performed to deploy the learning model to the terminal device. Here, an assumption that an information processing device (not shown) performs a weight reduction process will be described.

[0077] exist Figure 2In the example of the information processing device, the pre-compression model M0 of the execution category classification is 2 Quantization is performed to perform weight reduction processing and generate a post-compression model M1 2 At this time, for example, the information processing device uses the post-compression model M1 2 The recognition accuracy is used as an indicator to generate the post-compression model M1 2 .

[0078] exist Figure 2 In the example, the pre-compression model M0 2 The recognition accuracy is 90%, and the post-compression model M1 2 The recognition accuracy is 85%. As mentioned above, in terms of recognition accuracy, the pre-compression model M0 2 and post-compression model M1 2 Have roughly equal performance.

[0079] However, for the recognition accuracy of each class, for example, in class 2, the pre-compression model M0 2 The recognition accuracy is 100%, after compression model M1 2 The recognition accuracy is 75%.

[0080] As above, even in a case where there is no large difference in the recognition accuracy of all categories before and after the weight reduction process, there is a possibility that a large difference occurs in the recognition accuracy of each category before and after the weight reduction process.

[0081] In addition, there is a possibility that the feature quantities extracted by each neuron may differ. Figure 3 and Figure 4 Describe this.

[0082] Figure 3 is used to describe the pre-compression model M0 2 A view of the feature quantity extracted by neurons. Figure 4 is used to describe the post-compression model M1 2 A view of the feature quantity extracted by neurons.

[0083] like Figure 3 As shown, assuming the pre-compression model M0 2 Neuron N0 1 Extract "ear" as the feature quantity, neuron N0 2 Extract "mouth" as a feature. As a result, the pre-compression model M0 2 The output is a result with a probability of 99% that the input image is a "dog".

[0084] like Figure 4 As shown, assuming the post-compression model M1 2 Neuron N11 Extract "nose" as the feature, and neuron N1 2 Extract "claw" as the feature quantity. Note that the post-compression model M1 2 Neuron N1 1 Corresponding to the pre-compression model M0 2 Neuron N0 1 . Rear Compression Model M1 2 Neuron N1 2 Corresponding to the pre-compression model M0 2 Neuron N0 2 As a result, the post-compression model M1 2 The output is a 99% probability that the input image is a dog.

[0085] As mentioned above, even if the classification results are the same, the feature quantity extracted by each neuron is different from that of the model M0 before compression. 2 and post-compression model M1 2 It may also be different.

[0086] As above, even in the case where the difference in recognition accuracy of all categories before and after the weight reduction process is small, there is a possibility that the difference in recognition accuracy of individual categories is large or a variation in feature amounts extracted by individual neurons is present.

[0087] As mentioned above, even if the recognition accuracy before and after the weight reduction process does not change, it is possible that the model M0 before compression 2 and post-compression model M1 2 The model changes, that is, the equivalence deteriorates.

[0088] In addition, there is a possibility that the equivalence of fairness cannot be ensured before and after the lightweight processing.

[0089] Figure 5 is a diagram for describing another example of the weight reduction process. Figure 5 2 shows a process of reducing the weight of a learning model that classifies whether a person included in an input image is a “nurse” or a “doctor.” Note that here, description will be made based on the assumption that an information processing device (not shown) performs weight reduction processing and the like.

[0090] like Figure 5 As shown, for example, it is assumed that the information processing device uses the input image to pre-compress the model M0 3 Here, for example, it is assumed that the information processing device generates a heat map based on the feature quantity by using a technique such as "Grad-CAM" or "Grad-CAM++" as the XAI process. Thus, the information processing device is able to convert the pre-compression model M0 3 Visualization of the judgment criteria for the classification results.

[0091] from Figure 5 Heatmap of HM0 3 It can be seen that the pre-compression model M0 3 Classify based on the medical device, clothing color, sleeve length, etc. contained in the input image.

[0092] According to the above, it can be seen that if the pre-compression model M0 is not 3 Applying bias voltage ensures that the pre-compression model M0 3 fairness.

[0093] On the other hand, for example, it is assumed that the information processing apparatus 3 After compression model M1 3 Perform XAI processing. Here, with the pre-compression model M0 3 Similar to the case of, for example, it is assumed that the information processing apparatus generates a heat map based on the feature amount by using a technique such as “Grad-CAM” or “Grad-CAM++” as XAI processing.

[0094] from Figure 5 As can be seen from the heat map in the figure, the post-compression model M1 3 Classification is performed based on the face, hairstyle, etc. of a person included in the input image.

[0095] Therefore, there is a human biased compression model M1 3 possibility, and cannot ensure the post-processing model M1 3 fairness.

[0096] As mentioned above, even if the recognition accuracy of the test data before and after the weight reduction process does not change, it is possible that the model M0 before compression 3 and post-compression model M1 3 The possibility of changing the model, that is, the possibility of degradation in fairness.

[0097] As described above, in the conventional weight reduction process, although the recognition accuracy of the post-compression model M1 can be ensured to be equal to or higher than the desired accuracy, the change in the equivalence of the learning model before and after the weight reduction process is not considered.

[0098] Therefore, in the technology proposed in the present disclosure, as above, the information processing device evaluates the equivalence between the pre-compression model M0 and the post-compression model M1. The information processing device determines the post-compression model M1 (final model MF) based on the evaluation result of the equivalence. As a result, the information processing device can perform the weight reduction process in consideration of the change in the equivalence of the learning model before and after the weight reduction process.

[0099] <<2. Configuration Example of Information Processing Device>>

[0100] Figure 6 : is a block diagram showing a configuration example of an information processing device 10 according to an embodiment of the present disclosure. The information processing device 10 is a device that performs a weight reduction process and an equivalence evaluation process.

[0101] like Figure 6 As shown, the information processing device 10 includes a communication unit 11 , a storage unit 12 , a control unit 13 , and an input / output unit 14 .

[0102] [Communication unit 11]

[0103] The communication unit 11 is realized by, for example, a network interface card (NIC), etc. Then, the communication unit 11 is connected to a network in a wired or wireless manner, and transmits and receives information to and from another information processing apparatus.

[0104] [Storage unit 12]

[0105] The storage unit 12 is implemented by, for example, a semiconductor storage element such as a RAM (Random Access Memory) or a flash memory (Flash Memory), a storage device such as a hard disk or an optical disk. Figure 6 As shown, the storage unit 12 of the present embodiment includes a pre-compression model database (DB) 12_1, a post-compression model DB 12_2, and a data set DB 12_3.

[0106] The pre-compression model DB 12_1 stores the pre-compression model M0 before performing the weight reduction process. The post-compression model DB 12_2 stores the post-compression model M1 after the weight reduction process. The data set DB 12_3 stores the data set used for the XAI process (described later). In the case where the information processing device 10 learns the pre-compression model M0, the data set DB 12_3 can store the data set used to learn the pre-compression model M0.

[0107] [Input / output unit 14]

[0108] The input / output unit 14 is a user interface for exchanging information with a user. For example, the input / output unit 14 is an operation device for the user to perform various operations, such as a keyboard, a mouse, an operation key, and a touch panel. Alternatively, the input / output unit 14 is a display device, such as a liquid crystal display or an organic electroluminescent display (organic EL display). The input / output unit 14 may be an acoustic device such as a speaker or a buzzer. In addition, the input / output unit 14 may be a lighting device such as a light emitting diode (LED) lamp.

[0109] [Control unit 13]

[0110] For example, when a program (such as the information processing program according to the present disclosure) stored in the information processing device 10 is executed by a central processing unit (CPU), a microprocessing unit (MPU), or the like having a random access memory (RAM) or the like as a work area, the control unit 13 is realized. In addition, the control unit 13 is a controller and can be realized by an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0111] like Figure 6 As shown in FIG. 1 , the control unit 13 includes a compression processing unit 13_1, an XAI processing unit 13_2, an evaluation processing unit 13_3, a model determination unit 13_4, an output determination unit 13_5, and an input / output control unit 13_6, and implements or performs the functions and actions of the information processing described below. It should be noted that the internal configuration of the control unit 13 is not limited to Figure 6 The configuration shown in FIG. 1 and may be another configuration as long as it is a configuration that performs information processing described later. In addition, the connection relationship of each processing unit included in the control unit 13 is not limited to Figure 6 The connection relationship shown in , and can be another connection relationship.

[0112] (Compression processing unit 13_1)

[0113] The compression processing unit 13_1 compresses the size of the pre-compression model M0 by performing weight reduction processing on the pre-compression model M0, and generates a post-compression model M1. For example, the compression processing unit 13_1 acquires the pre-compression model M0 from the pre-compression model DB 12_1. The compression processing unit 13_1 stores the generated post-compression model M1 in the post-compression model DB 12_2.

[0114] The compression processing unit 13_1 performs at least one of weight reduction processes such as pruning, quantization, distillation, and neural architecture search (NAS).

[0115] Pruning is a weight reduction method for removing redundant connections of the pre-compression model M0. The compression processing unit 13_1 deletes a part of the network structure of the pre-compression model M0 through pruning to reduce the parameters of the post-compression model M1 or reduce the amount of calculation.

[0116] Quantization is a weight reduction method that replaces, for example, floating point parameters of the pre-compression model M0 or calculations performed by floating point with integers having a small bit length. It should be noted that mixed precision quantization is also known where the bit length changes for each layer of the pre-compression model M0.

[0117] Distillation is a method of reducing weights by causing a new model (post-compression model M1) to newly learn the relationship between input and output in the learned pre-compression model M0.

[0118] NAS is a method of automatically searching for a network structure. By using NAS, the compression processing unit 13_1 can create a lighter network without being limited to the existing network structure.

[0119] As above, the compression processing unit 13_1 performs weight reduction of the pre-compression model M0 by using at least one of various weight reduction methods. The compression processing unit 13_1 may perform weight reduction processing by combining a plurality of weight reduction methods (eg, by performing distillation after pruning and quantization).

[0120] For example, the compression processing unit 13_1 may perform weight reduction processing with different parameters on one pre-compression model M0 and generate multiple post-compression models M1. Alternatively, the compression processing unit 13_1 may perform weight reduction processing of different methods on one pre-compression model M0 and generate multiple post-compression models M1.

[0121] It is to be noted that the post-compression model M1 generated by the compression processing unit 13_1 is also referred to as a candidate model (an example of a first candidate learning model and a second candidate learning model).

[0122] (XAI processing unit 13_2)

[0123] The XAI processing unit 13_2 processes the pre-compression model M0 and the post-compression model M1 using the XAI technique. Examples of the XAI technique include the following techniques.

[0124] - Gradient Weighted Class Activation Mapping (Grad-CAM)

[0125] -Grad-CAM++

[0126] - Locally Interpretable Model-Agnostic Explanations (LIME)

[0127] -anchor

[0128] -Influence

[0129] -Activate Maximize

[0130] -Network analysis

[0131] It should be noted that the above-mentioned XAI technique is an example, and the XAI processing unit 13_2 may perform XAI processing by using other XAI techniques other than the above-mentioned XAI technique.

[0132] The XAI processing unit 13_2 performs XAI processing on the pre-compression model M0 and outputs the processing result to the estimation processing unit 13_3. The XAI processing unit 13_2 performs XAI processing on the post-compression model M1 and outputs the processing result to the estimation processing unit 13_3.

[0133] The XAI processing unit 13_2 may perform XAI processing by applying each of a plurality of XAI techniques to the pre-compression model M0 and the post-compression model M1.

[0134] (Evaluation Processing Unit 13_3)

[0135] The evaluation processing unit 13_3 evaluates the equivalence of the pre-compression model M0 and the post-compression model M1 based on the processing result of the XAI processing performed by the XAI processing unit 13_2. The evaluation processing unit 13_3 quantitatively evaluates the equivalence of the pre-compression model M0 and the post-compression model M1 by calculating the difference between the pre-compression model M0 and the post-compression model M1 as an equivalence (an example of an evaluation value).

[0136] The estimation processing unit 13_3 outputs the estimation result to the model determination unit 13_4.

[0137] It should be noted that the details of the equivalence evaluation process performed by the evaluation processing unit 13_3 will be described later.

[0138] (Model determination unit 13_4)

[0139] The model determination unit 13_4 determines the final model MF based on the evaluation result of equivalence performed by the evaluation processing unit 13_3. For example, in the case where the compression processing section 13_1 generates a plurality of post-compression models M1, the final model MF is determined by selecting one of the plurality of post-compression models M1.

[0140] For example, the model determination unit 13_4 outputs the final model MF to the post-compression model DB 12_2. For example, the model determination unit 13_4 outputs information related to the final model MF to the output determination unit 13_5.

[0141] It should be noted that the details of the determination processing performed by the model determination unit 13_4 will be described later.

[0142] (Output determination unit 13_5)

[0143] The output determination unit 13_5 determines whether to output the final model MF from the information processing device 10. For example, the output determination unit 13_5 determines whether to deploy the final model MF to a terminal device (not shown). Alternatively, the output determination unit 13_5 may determine whether the final model MF can be sold on a website for sale (such as a market).

[0144] For example, the output determination unit 13_5 determines whether to output the final model MF from the information processing device 10 based on the evaluation result of equivalence by the evaluation processing unit 13_3, more specifically, based on the evaluation result of equivalence indicating whether the equivalence is equal to or greater than a predetermined value. For example, when receiving at least one of the final models MF evaluated for equivalence as the first or more being sold on the market and deployed on the terminal device, the output determination unit 13_5 makes the determination.

[0145] As described above, the information processing device 10 of this embodiment uses an equivalent evaluation result different from the recognition accuracy as an evaluation index of the model (such as an evaluation index of whether to deploy). For example, the equivalent evaluation result can be used as a standard quality index of the product of the deployed final model MF.

[0146] (Input / output control unit 13_6)

[0147] The input / output control unit 13_6 controls the input / output unit 14, presents information to the user, and receives input from the user. For example, the input / output control unit 13_6 presents the evaluation result of equivalence performed by the evaluation processing unit 13_3 to the user. The details of the evaluation result of equivalence presented to the user by the input / output control unit 13_6 will be described later.

[0148] <<3. Examples of Information Processing>>

[0149] Next, an example of information processing performed by the information processing device 10 will be described.

[0150] <3.1. Equivalence evaluation process>

[0151] As above, the information processing device 10 performs an equivalence evaluation process of evaluating the equivalence of the pre-compression model M0 and the post-compression model M1 by using the XAI processing result. For example, the information processing device 10 calculates a value corresponding to the difference between the analysis result of the model that analyzes the pre-compression model M0 (such as the XAI processing result) and the analysis result of the model that analyzes the post-compression model M1 (such as the XAI processing result) as an equivalence value.

[0152] Hereinafter, the equivalence evaluation process performed by the information processing device 10 for each XAI process will be described.

[0153] <3.1.1. First equivalence evaluation process>

[0154] The first equivalence evaluation process is a process for performing equivalence evaluation using a heat map as basic information. In this case, the XAI processing unit 13_2 generates a heat map (saliency map) based on the feature quantities (such as feature maps) of the pre-compression model M0 and the post-compression model M1, and estimates the image area as the determination basis of the model. First, the XAI process performed by the XAI processing unit 13_2 (hereinafter, also referred to as the first XAI process) will be described.

[0155] Figure 7 is a diagram showing an example of a first XAI process according to an embodiment of the present disclosure.

[0156] The XAI processing unit 13_2 uses, for example, "Grad-CAM" or "Grad-CAM++"

[0157] The XAI technique visualizes the determined libraries of the pre-compression model M0 and the post-compression model M1.

[0158] -Grad-CAM: Gradient-based localization<https: / / arxiv.org / abs / 1610.02391>

[0159] -Grad-CAM++: Improved Visual Explanations for Deep Convolutional Networks<https: / / arxiv.org / abs / 1710.11063>

[0160] It should be noted that, although the description of the techniques of “Grad-CAM” and “Grad-CAM++” is appropriately omitted, the XAI processing unit 13_2 generates the heat map HM by the methods of “Grad-CAM” and “Grad-CAM++”. 1 and HM1 1 As basic information (see above reference).

[0161] exist Figure 7 In the example of FIG. 1 , the XAI processing unit 13_2 generates a heat map HM0 1 , the heat map HM0 1 is an indication of the pre-compression model M0 after the input of the input image IM1 1 The output of the basic information related to the basis of the heat map HM0 1 The pixels of the input image IM1 are pre-compressed into the model M0 1 The prediction results have contributed to the basic information visualization of the contribution.

[0162] In addition, the XAI processing unit 13_2 generates a heat map HM1 1 , the heat map HM1 1 is the post-compression model M1 after inputting the input image IM11 The output of the heat map HM1 1 is the pixel pair compression model M1 of the visualized input image IM1 1 The prediction results provide basic information about the contribution of the contributors.

[0163] Note that Figure 7 In the example, it is assumed that due to the heat map HM0 1 and HM1 1 The darker the shadow, the higher the contribution to the prediction result.

[0164] When the heat map HM0 is obtained from the XAI processing unit 13_2 1 and HM1 1 As a result of the processing, the evaluation processing unit 13_3 evaluates the heat map HM0 based on the heat map HM0. 1 and HM1 1 A first equivalence evaluation process is performed.

[0165] Figure 8 IM1 is a diagram showing an example of a first equivalence evaluation process according to an embodiment of the present disclosure. Here, it is assumed that there is a correct answer corresponding to the input image IM1.

[0166] For example, Figure 8 As shown in , it is assumed that the correct answer of the input image IM1 is given as the bounding box BB1. In this case, the evaluation processing unit 13_3 evaluates the equivalence using, for example, an energy-based pointing game (EBPG) as an evaluation index.

[0167] EBPG is an indicator heat map (here, heat map HM0 1 and HM1 1 ) is an index of how much energy fits into the bounding box of the target object (here, the bounding box BB1 of the input image IM1). EBPG is calculated based on the following equation (1), for example.

[0168]

[0169] like Figure 8 As shown, the evaluation processing unit 13_3 calculates, for example, a heat map HM0 of the bounding box BB1 of the input image IM1. 1 Here, assuming that the heat map HM0 1 The EBPG is 80%.

[0170] Similarly, the evaluation processing unit 13_3 calculates, for example, a heat map HM1 of the bounding box BB1 of the input image IM1. 1 Here, assuming that the heat map HM1 1 The EBPG is 70%.

[0171] In this case, for example, the evaluation processing unit 13_3 calculates the heat map HM0 1 Heat map of EBPG and HM1 1 The difference of EBPG as the model M0 before compression 1 and post-compression model M1 1 That is, the evaluation processing unit 13_3 calculates 80-70=10% as the pre-compression model M0 1 and post-compression model M1 1 The difference between.

[0172] Alternatively, for example, the evaluation processing unit 13_3 may calculate a value obtained by subtracting the calculated difference from 100% (100-10=90) as the pre-compression model M0. 1 With rear compression model M1 1 The degree of match (equivalence) between them.

[0173] For example, the evaluation processing unit 13_3 calculates the pre-compression model M0 by using a plurality of test data. 1 and post-compression model M1 1 The difference (or matching degree) between them is calculated, and the overall average of the absolute values ​​of the calculated differences is set as the pre-compression model M0 1 and post-compression model M1 1 The amount of change.

[0174] The evaluation processing unit 13_3 outputs the calculated variation to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1 Alternatively, as the pre-compression model M0 1 and post-compression model M1 1 The evaluation processing unit 13_3 may output, to the model determination unit 13_4, a value obtained by subtracting the calculated amount of change from 100%.

[0175] Fig. 9 IM1 is a diagram showing another example of the first equivalence evaluation process according to an embodiment of the present disclosure. Here, it is assumed that there is no correct answer corresponding to the input image IM1.

[0176] In this case, the evaluation processing unit 13_3 evaluates the heat map HM0 based on the heat map HM0. 1 and HM1 1 The equivalence is evaluated by the degree of match.

[0177] For example, the evaluation processing unit 13_3 calculates the heat map HM0 by 1 and HM1 1The heat map HM0 is set by setting the values ​​of pixels whose contribution is equal to or higher than a predetermined threshold to "1" and setting the values ​​of other pixels to "0". 1 and HM1 1 Binarization and generation of a binary image BM0 1 and BM1 1 It should be noted that Fig. 9 In the binary image BM0 1 and BM1 1 The black area R 1 and R 2 is the area where the pixel value is "1".

[0178] The evaluation processing unit 13_3 calculates the binary image BM0 by using the union intersection (IoU) 1 and BM1 1 Here, IoU is an index indicating how much two regions overlap.

[0179] like Figure 8 As shown in FIG. 1 , the evaluation processing unit 13_3 calculates the area R 1 and R 2 The IoU in is 70%.

[0180] The evaluation processing unit 13_3 sets the value of IoU (here, 70%) to the pre-compression model M0 1 and post-compression model M1 1 Alternatively, the evaluation processing unit 13_3 sets the difference value of IoU (the value obtained by subtracting the value of IoU from 100%). Here, 100-70=30% is used as the pre-compression model M0 1 and post-compression model M1 1 The amount of change.

[0181] For example, the evaluation processing unit 13_3 calculates the pre-compression model M0 using a plurality of test data. 1 and post-compression model M1 1 The matching degree (or difference) is calculated, and the total average value of the absolute value of the matching degree is set as the pre-compression model M0 1 and post-compression model M1 1 The matching degree (equivalence).

[0182] The evaluation processing unit 13_3 outputs the calculated equivalence to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1Alternatively, the evaluation processing unit 13_3 may output a value obtained by subtracting the calculated equivalent value from 100% to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1 The evaluation results of equivalent value.

[0183] In this way, the information processing device 10 is based on the heat map HM0 1 and HM1 1 To evaluate the model M0 before compression 1 and post-compression model M1 1 That is, the information processing device 10 is based on the pre-compression model M0 1 and post-compression model M1 1 The difference in the focus data position is used to evaluate the model M0 before compression. 1 and post-compression model M1 1 equivalence.

[0184] This allows the information processing device 10 to more easily confirm changes in the learning model, such as changes in the position of the attention data before and after the weight reduction process.

[0185] <3.1.2. Second equivalence evaluation process>

[0186] The second equivalence evaluation process is a process of performing equivalence evaluation by using an image block as reference information.

[0187] In this case, the XAI processing unit 13_2 generates an image block determined as a model based on the feature amounts of the pre-compression model M0 and the post-compression model M1. That is, the XAI processing unit 13_2 generates basic information indicating which feature (image block) is important for prediction by the model.

[0188] For example, the XAI processing unit 13_2 approximates the model using a linear model describing the surroundings of the target data. The XAI processing unit 13_2 measures the importance of each feature (image block) by the magnitude of the coefficient of the linear model.

[0189] Fig.10 13_2 is a diagram showing an example of a second XAI process according to an embodiment of the present disclosure. The XAI processing unit 13_2 visualizes the determination bases of the pre-compression model M0 and the post-compression model M1 by using an XAI technique such as “LIME” or “Anchor”.

[0190] -LIME: "Why Should ITrust You?": Explaining the Predictions of AnyClassifier<https: / / arxiv.org / abs / 1602.04938>

[0191] -Anchors: High-Precision Model-AgnosticExplanations<https: / / homes.cs.washington.edu / ~marcotcr / aaai18.pdf>

[0192] Note that although the description of the techniques of “LIME” and “anchor point” is appropriately omitted, the XAI processing unit 13_2 generates the image block PM by the method of “LIME” or “anchor point”. 1 and PM1 1 As basic information (see above reference).

[0193] exist Fig.10 In the example, the XAI processing unit 13_2 first divides the input image IM1 into super pixels and generates a divided image PM1.

[0194] Based on the segmented image PM1, the XAI processing unit 13_2 generates an image block PM0 1 , the image block PM0 1 It indicates that after the input image IM1 is input and the pre-compression model M0 1 The output of the image block PM0 1 is indicated in the use of pre-compression model M0 1 An image of an image block that is important for the inference of the input image IM1 in the inference.

[0195] Based on the segmented image PM1, the XAI processing unit 13_2 generates an image block PM1 1 , the image block PM1 1 is the post-compression model M1 after inputting the input image IM1 1 The output of the image block PM1 is related to the basic information. 1 Is indicated to compress model M1 after use 1 An image of an image block that is important for the inference of the input image IM1 in the inference.

[0196] The XAI processing unit 13_2 generates the image slice PM0 1 and PM1 1The result of the processing is output to the evaluation processing unit 13_3.

[0197] When the image slice PM0 is obtained from the XAI processing unit 13_2 1 and PM1 1 As a result of the processing, the evaluation processing unit 13_3 evaluates the image slice PM0 based on the image slice PM0. 1 and PM1 1 A second equivalence evaluation process is performed.

[0198] Fig.11 : is a view showing an example of a second equivalence evaluation process according to an embodiment of the present disclosure.

[0199] like Fig.11 As shown, for example, the evaluation processing unit 13_3 performs the evaluation based on the image slice PM0. 1 and PM1 1 For example, the evaluation processing unit 13_3 calculates the image block PM0 by using IoU 1 and PM1 1 overlap.

[0200] like Fig.11 As shown, the evaluation processing unit 13_3 calculates the image block PM0 1 and PM1 1 The IoU is 70%.

[0201] The evaluation processing unit 13_3 sets the value of IoU (here, 70%) to the pre-compression model M0 1 and post-compression model M1 1 Alternatively, the evaluation processing unit 13_3 sets the difference value of IoU (the value obtained by subtracting the value of IoU from 100%). Here, 100-70=30% is used as the pre-compression model M0 1 and post-compression model M1 1 The amount of change.

[0202] For example, the evaluation processing unit 13_3 calculates the pre-compression model M0 using a plurality of test data. 1 and post-compression model M1 1 The matching degree (or difference) is calculated, and the total average value of the absolute value of the matching degree is set as the pre-compression model M0 1 and post-compression model M1 1 The matching degree (equivalence).

[0203] The evaluation processing unit 13_3 outputs the calculated equivalence to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1Alternatively, the evaluation processing unit 13_3 may output a value obtained by subtracting the calculated equivalent value from 100% to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1 The evaluation results of equivalent value.

[0204] In this way, the information processing device 10 performs the image processing based on the image patch PM0. 1 and PM1 1 Evaluating the pre-compression model M0 1 and post-compression model M1 1 That is, the information processing device 10 is based on the pre-compression model M0 1 and post-compression model M1 1 The difference in the focus data position is used to evaluate the model M0 before compression. 1 and post-compression model M1 1 equivalence.

[0205] This allows the information processing device 10 to more easily confirm changes in the learning model, such as changes in the position of the attention data before and after the weight reduction process.

[0206] <3.1.3. Third equivalence evaluation process>

[0207] The third equivalence evaluation process is a process of performing equivalence evaluation by using the images of the data set as basic information.

[0208] In this case, for example, the XAI processing unit 13_2 generates influence information related to influence data that effectively influences learning among the learning data held by the dataset DB 12_3. That is, the XAI processing unit 13_2 generates basic information indicating which piece of learning data (such as image data) has caused a large change in the prediction by the model.

[0209] For example, in the learning of the pre-compression model M0 and the post-compression model M1, the XAI processing unit 13_2 calculates the influence (degree of influence) of each piece of learning data that influences the learning.

[0210] Fig.12 13_2 is a diagram showing an example of a third XAI process according to an embodiment of the present disclosure. The XAI processing unit 13_2 calculates the learning data for the pre-compression model M0 by using an XAI technique such as “Influence”. 1 and post-compression model M1 1 degree of impact.

[0211] -Understanding Black-box Predictions via Influence Functions<https: / / arxiv.org / abs / 1703.04730>

[0212] It should be noted that, although the description of the technique of “Influence” is appropriately omitted, the XAI processing unit 13_2 calculates the degree of influence (importance) of the learning data by the method of “Influence” (see the above document).

[0213] exist Fig.12 In the example, the XAI processing unit 13_2 performs the pre-compression model M0 1 and post-compression model M1 1 Each test data included in each calculation test data set 12_32 (such as Fig.12 The importance of learning data in the image data of cats, rabbits and dogs in .

[0214] Note that the learning data is included in the learning data set 12_31. In addition, for example, the learning data set 12_31 and the test data set 12_32 are included in the data set DB 12_3.

[0215] like Fig.12 As shown, the XAI processing unit 13_2 generates the importance level R0 of the learning data according to the calculated importance. 1 and R1 1 . Importance level R0 1 is based on the learning data relative to the pre-compression model M0 1 The level of influence generated. Importance level R1 1 is based on the learning data relative to the post-compression model M1 1 The level of influence generated.

[0216] The XAI processing unit 13_2 generates the importance level R0 1 and R1 1 The result of the processing is output to the evaluation processing unit 13_3.

[0217] When the importance level R0 is obtained from the XAI processing unit 13_2 1 and R1 1 As a result of the processing, the evaluation processing unit 13_3 determines the importance level R0 based on the importance level R0. 1 and R1 1 The third equivalence evaluation process is executed.

[0218] Fig.13 : is a view showing an example of a third equivalence evaluation process according to an embodiment of the present disclosure.

[0219] like Fig.13 As shown, for example, the evaluation processing unit 13_3 evaluates the importance level R0 based on the importance level R1. 1 and R1 1 For example, the evaluation processing unit 13_3 calculates the Jaccard index as the importance level R0 1 and R1 1 similarity (matching degree).

[0220] For example, the evaluation processing unit 13_3 calculates the importance level R0 for each test data. 1 and R1 1 Similarity (Jaccard index). Fig.13 In the example, the test data is "cat" and its importance ranking is R0 1 and R1 1 The similarity (Jaccard index) is 1.0. The test data is the importance ranking R0 of "rabbit". 1 and R1 1 The similarity (Jaccard index) is 1.0. The test data is "dog" and its importance ranking is R0 1 and R1 1 The similarity (Jaccard index) is 0.818.

[0221] For example, the evaluation processing unit 13_3 sets the average value of the similarities in the test data set to the pre-compression model M0 1 and post-compression model M1 1 The similarity (equivalence) between . Fig.13 In the example of , the evaluation processing unit 13_3 sets (1.0+1.0+0.818) / 3=0.939 as the average value of the test data set as the pre-compression model M0 1 With rear compression model M1 1 Similarity (equivalence) between them.

[0222] Alternatively, the evaluation processing unit 13_3 sets the difference of the mean value (a value obtained by subtracting the mean value from 1). Here, 1-0.939=0.061) as the pre-compression model M0 1 and post-compression model M1 1 The amount of change.

[0223] The evaluation processing unit 13_3 outputs the calculated equivalence to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1Optionally, the evaluation processing unit 13_3 may output a value obtained by subtracting the calculated equivalent value from 1 to the model determination unit 13_4 as the pre-compression model M0. 1 and post-compression model M1 1 The evaluation results of the equivalent value.

[0224] In this way, the information processing device 10 ranks R0 based on the importance of each test data. 1 and R1 1 Evaluating the pre-compression model M0 1 and post-compression model M1 1 That is, the information processing device 10 is based on the pre-compression model M0 1 and post-compression model M1 1 The difference in learning data has a significant impact on evaluating the pre-compression model M0 1 and post-compression model M1 1 equivalence.

[0225] As a result, the information processing apparatus 10 can more easily confirm that changes in the learning model, such as changes in the influencing data, have an effective influence on the learning before and after the weight reduction process.

[0226] Note that although the XAI processing unit 13_2 uses the Jaccard index to calculate the importance level R0 1 and R1 1 However, the XAI processing unit 13_2 can calculate the similarity by using another method. For example, the XAI processing unit 13_2 can sort the importance by R0 1 and R1 1 The importance included in is considered as the vector of each test data, and the similarity between the vectors is calculated as the importance ranking R0 1 and R1 1 For example, Euclidean distance, cosine similarity, etc. can be used as the similarity between vectors.

[0227] <3.14 Fourth equivalence evaluation process>

[0228] The third equivalence evaluation process is a process of performing equivalence evaluation by using feature amounts extracted in neurons or layers.

[0229] In this case, the XAI processing unit 13_2 generates, for example, feature amounts extracted by neurons or layers as feature amount images.

[0230] Fig.14 : is a view showing an example of a third XAI process and a third equivalence evaluation process according to an embodiment of the present disclosure.

[0231] The XAI processing unit 13_2 converts the pre-compression model M0 into a 2 and post-compression model M1 2 Visualize the features extracted by neurons or layers.

[0232] -Understanding Neural Networks through Deep Visualization<https: / / arxiv.org / abs / 1506.06579>

[0233] -Understanding Deep Image Representations by Inverting Them<https: / / arxiv.org / abs / 1412.0035>

[0234] Note that, although the description of a technique such as “activation maximization” is appropriately omitted, the XAI processing unit 13_2 generates a feature amount image that visualizes feature amounts by a method such as “activation maximization” (see the above document).

[0235] exist Fig.14 In the example, the XAI processing unit 13_2 uses the pre-compression model M0 2 Neuron n0 0 To n0 3 The extracted features are visualized and feature image AM is generated. 0 to 0 AM 3 For example, the XAI processing unit 13_2 combines the neuron n0 0 To n0 3 The output of the input image is maximized to generate the feature image AM0 0 to 0 AM 3 .

[0236] Similarly, the XAI processing unit 13_2 will be composed of the post-compression model M1 2 Neuron n1 0 To n1 3 The extracted features are visualized and feature image AM1 is generated 0 to AM1 3 Note that the post-compression model M1 2 Neuron n1 0 To n1 3 They are respectively the same as the former compression model M0 2 Neuron n0 0 To n0 3 The corresponding neurons.

[0237] The XAI processing unit 13_2 generates the feature image AM0 0 Output to AM0 3 And AM1 0 to AM1 3 Output to the evaluation processing unit 13_3 as a processing result.

[0238] When the feature amount image AM0 is acquired from the XAI processing unit 13_2 0 to 0 AM 3 and AM1 0 to AM1 3 As a result of the processing, the evaluation processing unit 13_3 calculates the value of the feature amount image AM0 based on the feature amount image AM0. 0 to 0 AM 3 and AM1 0 to AM1 3 The fourth equivalence evaluation process is executed.

[0239] like Fig.14 As shown, the evaluation processing unit 13_3 performs the evaluation based on, for example, the feature amount image AM0 0 to 0 AM 3 and AM1 0 to AM1 3 The difference between evaluates equivalence.

[0240] First, the evaluation processing unit 13_3 generates, for example, a feature amount image AM. 0 to 0 AM 3 and feature image AM1 0 to AM1 3 For example, the evaluation processing unit 13_3 generates the difference images DM0 to DM3 for each corresponding neuron.

[0241] Then, the evaluation processing unit 13_3 performs comparison of whether the pixel values ​​contained in the difference images DM0 to DM3 are equal to or greater than a predetermined threshold, and sets the ratio of pixels having pixel values ​​of the predetermined threshold or greater to all pixels as the amount of change of the difference images DM0 to DM3.

[0242] For example, the evaluation processing unit 13_3 generates a feature amount image AM0 3 and feature image AM1 3 The evaluation processing unit 13_3 counts the number of pixels having pixel values ​​equal to or greater than a predetermined threshold value among all pixels (here, for example, 36 pixels) included in the differential image DM3. Here, it is assumed that there are four pixels whose pixel values ​​are equal to or greater than the predetermined threshold value.

[0243] In this case, the evaluation processing unit 13_3 assumes that the neuron n03 and n1 3 The amount of change with respect to the difference image DM3 is 4 / 36=0.111.

[0244] Alternatively, the evaluation processing unit 13_3 sets the difference in the amount of change (a value obtained by subtracting the change value from 1). Here, 1-0.111=0.899) as the value of the neuron n0 3 and n1 3 The equivalent of.

[0245] For example, the evaluation processing unit 13_3 calculates the change amount of all neurons. For example, the evaluation processing unit 13_3 sets at least one of the sum or average of the calculated change amounts as the pre-compression model M0 2 and post-compression model M1 2 The amount of change.

[0246] Alternatively, for example, the evaluation processing unit 13_3 calculates the equivalence of all neurons. For example, the evaluation processing unit 13_3 sets at least one of the total value or the average value of the calculated equivalence value as the pre-compression model M0 2 and post-compression model M1 2 of equivalent value.

[0247] Alternatively, the evaluation processing unit 13_3 may also use the pre-compression model M0 as an example. 2 and post-compression model M1 2 The difference in the amount of change (the value obtained by subtracting the change value from 1) is set as the pre-compression model M0 2 and post-compression model M1 2 of equivalent value.

[0248] For example, as the pre-compression model M0 2 and post-compression model M1 2 The evaluation processing unit 13_3 may output at least one of the calculated equivalence or the amount of change to the model determination unit 13_4 based on the evaluation result of the equivalence.

[0249] Note that although it is assumed here that the XAI processing unit 13_2 generates feature amount images AM0 and AM1, the XAI processing unit 13_2 may generate feature amounts extracted by neurons or layers, for example, as “semantic concepts”.

[0250] Fig.15 is a diagram illustrating another example of the third XAI process and the third equivalence evaluation process according to an embodiment of the present disclosure.

[0251] The XAI processing unit 13_2 converts the pre-compression model M0 into the XAI processing unit 13_2 by using the XAI technology of “network decomposition”, for example. 2 and post-compression model M12 Visualize the features extracted by neurons or layers.

[0252] -Network Dissection: Quantifying Interpretability of Deep Visual Representations<https: / / arxiv.org / abs / 1704.05796>

[0253] Note that, although the description of the technique of “network decomposition” is appropriately omitted, the XAI processing unit 13_2 generates a semantic concept of a visualization feature amount by the method of “network decomposition” (see the above document).

[0254] exist Fig.15 In the example, the XAI processing unit 13_2 uses the pre-compression model M0 2 Neuron n0 4 To n0 7 The extracted feature quantity is visualized, and a semantic concept SC0 is generated. In addition, the XAI processing unit 13_2 makes the post-compression model M1 2 Neuron n1 4 The feature quantities extracted to n17 are visualized and the semantic concept SC1 is generated. Note that although Fig.15 The XAI processing unit 13_2 in FIG. 1 is illustrated corresponding to the neuron n0 4 To n0 7 and n1 4 To the semantic concepts SC0 and SC1 of n17, but the XAI processing unit 13_2 generates the semantic concepts SC0 and SC1 for all neurons n0 and n1.

[0255] The XAI processing unit 13_2 outputs the generated semantic concepts SC0 and SC1 to the evaluation processing unit 13_3 as a processing result.

[0256] When acquiring the semantic concepts SC0 and SC1 from the XAI processing unit 13_2 as the processing result, the evaluation processing unit 13_3 performs a fourth equivalence evaluation process based on the semantic concepts SC0 and SC1.

[0257] The evaluation processing unit 13_3 evaluates equivalence based on the similarity between the semantic concepts SC0 and SC1. For example, the evaluation processing unit 13_3 calculates the Jaccard index as the similarity (equivalence) between the semantic concepts SC0 and SC1.

[0258] Alternatively, the evaluation processing unit 13_3 sets the difference in similarity (a value obtained by subtracting the similarity from 1) as the amount of change of the semantic concepts SC0 and SC1.

[0259] For example, as the pre-compression model M0 2 and post-compression model M1 2 The evaluation processing unit 13_3 may output at least one of the calculated equivalence or the amount of change to the model determination unit 13_4 based on the evaluation result of the equivalence.

[0260] In this way, the information processing device 10 evaluates the pre-compression model M0 based on the feature quantity extracted by each neuron. 2 and post-compression model M1 2 That is, the information processing device 10 evaluates the equivalence based on the changes in the internal structures of the pre-compression model M0 and the post-compression model M1.

[0261] As a result, the information processing device 10 can more easily confirm changes in the learning model, such as changes in the feature amount extracted by each neuron before and after the weight reduction process.

[0262] Note that here, the XAI processing unit 13_2 uses the Jaccard index to calculate the similarity between the semantic concepts SC0 and SC1. However, the XAI processing unit 13_2 can calculate the similarity by using another method. For example, the XAI processing unit 13_2 can regard the semantic concepts SC0 and SC1 as vectors, and calculate the similarity between the vectors as the similarity between the semantic concepts SC0 and SC1. For example, the Euclidean distance, cosine similarity, etc. can be used as the similarity between the vectors.

[0263] <3.2. Determine the processing>

[0264] As above, the information processing device 10 performs the determination process of selecting (determining) the final model MF by using the result of the equivalence evaluation process.

[0265] For example, the information processing device 10 selects the post-compression model M1 (final model MF) from a plurality of candidate models. Alternatively, the information processing device 10 generates the final model MF by repeatedly performing the weight reduction process, the XAI process, and the equivalence evaluation process according to the result of the equivalence evaluation process.

[0266] Here, first to third determination processes will be described as examples of determination processes performed by the information processing device 10 .

[0267] <3.2.1. First determination process>

[0268] The first determination process is a process of selecting a final model MF from a plurality of candidate models. In this case, for example, the compression processing unit 13_1 performs a weight reduction process using different weight reduction parameters and generates a plurality of candidate models from a pre-compression model M0. That is, the plurality of candidate models are models generated by weight reduction processes having the same weight reduction method and different weight reduction parameters.

[0269] Fig.16 is a view for describing an example of a first determination process according to an embodiment of the present disclosure.

[0270] exist Fig.16 In the example shown, the compression processing unit 13_1 processes the pre-compression model M0. 3 Perform the first weight reduction process and generate candidate model M1 31 In addition, the compression processing unit 13_1 performs compression on the pre-compression model M0 3 Perform the second weight reduction process to generate candidate model M1 32 .

[0271] The first weight reduction process and the second weight reduction process are the same weight reduction method ( Fig.16 ) and have different weight reduction parameters (e.g., compression ratio in each layer). 31 and M1 32 The first and second weight reduction processes are performed in a manner having the same size.

[0272] The XAI processing unit 13_2 processes the candidate model M1 31 and M1 32 In addition, the evaluation processing unit 13_3 performs equivalence evaluation based on the result of the XAI processing and generates an evaluation result for the candidate model M1. 31 and M1 32 Every computational equivalence in .

[0273] exist Fig.16 In the example, assume that the candidate model M1 31 The equivalence of is 90% and the candidate model M1 32 The equivalence of is 70%. In addition, assume that the candidate model M1 31 The recognition accuracy is 80%, and the candidate model M1 32 The recognition accuracy is 82%.

[0274] In this case, for example, when determining a model based on recognition accuracy, the candidate model M1 32 is determined as the final model MF. However, the candidate model M1 32 The equivalence of is 70% and is lower than the candidate model M131 In addition, the candidate model M1 31 The recognition accuracy of 32 The recognition accuracy is not much different.

[0275] Thus, for example, the model determination unit 13_4 of the present embodiment selects the candidate model M1 having high equivalence. 31 Set as the final model MF.

[0276] As described above, the model determination unit 13_4 of this embodiment determines the final model MF based on equivalence instead of (or in addition to) the recognition accuracy. As a result, the information processing device 10 can select the candidate model M1 with a small change in the model before and after processing. 31 Set as the final model MF.

[0277] Note that although it is assumed here that the model determination unit 13_4 selects the candidate model M1 having high equivalence 31 The final model MF is set, but the determination method of the model determination unit 13_4 is not limited thereto. The model determination unit 13_4 only needs to determine the final model MF in consideration of equivalence.

[0278] For example, the model determination unit 13_4 may set the candidate model M1 having the highest recognition accuracy (or equivalence) among the candidate models M1 having the recognition accuracy equal to or greater than a predetermined threshold and the equivalence equal to or greater than a predetermined threshold as the final model MF. Note that the threshold for determining the recognition accuracy and the threshold for determining the equivalence may be different values ​​or the same value.

[0279] In addition, although Fig.16 The number of candidate models M1 in is two, but the number of candidate models M1 is not limited to two. The number of candidate models M1 can be one or three or more. In addition, although Fig.16 The weight reduction method used in the weight reduction process is pruning, but the weight reduction method is not limited to pruning. For example, the compression processing unit 13_1 can generate the candidate model M1 by using various weight reduction methods such as quantization.

[0280] Although in Fig.16 The compression processing unit 13_1 generates the candidate model M1 by using one weight reduction method, but the compression processing unit 13_1 may generate the candidate model M1 by using a plurality of weight reduction methods. The determination process of the model determination unit 13_4 determining the final model MF in this case will be described.

[0281] Fig.17 is a diagram for describing another example of the first determination process according to an embodiment of the present disclosure.

[0282] exist Fig.17 In the example shown, the compression processing unit 13_1 uses the first weight reduction method to reduce the pre-compression model M0. 4 Perform weight reduction processing and generate candidate model M1 41 In addition, the compression processing unit 13_1 performs weight reduction on the pre-compression model M0 using a second weight reduction method that is different from the first weight reduction method. 4 Perform weight reduction processing to generate candidate model M1 42 .

[0283] The XAI processing unit 13_2 processes the candidate model M1 41 and M1 42 The XAI process is performed. In addition, the evaluation processing unit 13_3 performs equivalence evaluation based on the result of the XAI process and calculates the candidate model M1 41 and M1 42 The equivalence of each in .

[0284] exist Fig.17 In the example, assume that the candidate model M1 41 The equivalence of is 90% and the candidate model M1 42 The equivalence of is 70%. In addition, assume that the candidate model M1 41 The recognition accuracy is 80%, and the candidate model M1 42 The recognition accuracy is 82%.

[0285] In this case, for example, when determining a model based on recognition accuracy, the candidate model M1 42 is determined as the final model MF. However, the candidate model M1 42 The equivalence of is 70% and is lower than the candidate model M1 41 In addition, the candidate model M1 41 The recognition accuracy of 42 The recognition accuracy is not much different.

[0286] Thus, for example, the model determination unit 13_4 of the present embodiment selects the candidate model M1 having high equivalence. 41 Set as the final model MF.

[0287] As described above, the model determination unit 13_4 of this embodiment determines the final model MF based on equivalence instead of (or in addition to) the recognition accuracy. As a result, the information processing device 10 can select the candidate model M1 with a small change in the model before and after processing. 41 Set as the final model MF.

[0288] Note that although it is assumed here that the model determination unit 13_4 selects the candidate model M1 having high equivalence41 The final model MF is set, but the determination method of the model determination unit 13_4 is not limited thereto. The model determination unit 13_4 only needs to determine the final model MF in consideration of equivalence.

[0289] For example, the model determination unit 13_4 may set the candidate model M1 having the highest recognition accuracy (or equivalence) among the candidate models M1 having the recognition accuracy equal to or greater than a predetermined threshold and the equivalence equal to or greater than a predetermined threshold as the final model MF. Note that the threshold for determining the recognition accuracy and the threshold for determining the equivalence may be different values ​​or the same value.

[0290] In addition, although Fig.17 The number of candidate models M1 in is two, but the number of candidate models M1 is not limited to two. The number of candidate models M1 may be one or three or more.

[0291] In addition, although Fig.17 The compression processing unit 13_1 generates the candidate model M1 by using two weight reduction methods, but the weight reduction methods used for the weight reduction process by the compression processing unit 13_1 are not limited to two. The compression processing unit 13_1 may generate the candidate model M1 by using three or more weight reduction methods.

[0292] In addition, although Fig.17 The compression processing unit 13_1 generates one candidate model M1 by using one weight reduction method, but the number of candidate models M1 generated by the compression processing unit 13_1 is not limited to one. For example, the compression processing unit 13_1 may generate a plurality of candidate models M1 by using one weight reduction method. In this case, it is assumed that the weight reduction parameter used for the weight reduction process is different between the plurality of candidate models M1.

[0293] Furthermore, in the above-described first determination process, the compression processing unit 13_1 generates a plurality of candidate models M1 from one pre-compression model M0. However, there may be a plurality of pre-compression models M0 for which the weight reduction process is performed by the compression processing unit 13_1. The determination process in which the model determination unit 13_4 determines the final model MF in this case will be described.

[0294] Fig.18 is a diagram for describing another example of the first determination process according to an embodiment of the present disclosure.

[0295] exist Fig.18 In the example shown in FIG. 1 , the compression processing unit 13_1 performs compression on the pre-compression model M0. 51 The first weight reduction process is performed, and a candidate model M1 is generated 51In addition, the compression processing unit 13_1 performs compression on the pre-compression model M0 52 Perform the second weight reduction process to generate candidate model M1 52 .

[0296] The first weight reduction process and the second weight reduction process are the same weight reduction method ( Fig.18 For example, the compression processing unit 13_1 takes the candidate model M1 51 and M1 52 The first and second weight reduction processes are performed in a manner having the same size.

[0297] The XAI processing unit 13_2 processes the candidate model M1 51 and M1 52 In addition, the evaluation processing unit 13_3 performs equivalence evaluation based on the result of the XAI processing and generates an evaluation result for the candidate model M1. 51 and M1 52 Every computational equivalence in .

[0298] exist Fig.18 In the example, assume that the candidate model M1 51 The equivalence of is 90% and the candidate model M1 52 The equivalence of is 70%. In addition, assume that the candidate model M1 51 The recognition accuracy is 75%, and the candidate model M1 52 The recognition accuracy is 80%.

[0299] In this case, for example, when determining a model based on recognition accuracy, the candidate model M1 52 is determined as the final model MF. However, the candidate model M1 52 The equivalence of is 70% and is lower than the candidate model M1 51 equivalence.

[0300] Thus, the model determination unit 13_4 of this embodiment selects, for example, a candidate model M1 having a high equivalence. 51 Set as the final model MF.

[0301] As described above, the model determination unit 13_4 of this embodiment determines the final model MF based on equivalence instead of (or in addition to) the recognition accuracy. As a result, the information processing device 10 can select the candidate model M1 with a small change in the model before and after processing. 51 Set as the final model MF.

[0302] Note that although it is assumed here that the model determination unit 13_4 selects the candidate model M1 having high equivalence 51The final model MF is set, but the determination method of the model determination unit 13_4 is not limited thereto. The model determination unit 13_4 only needs to determine the final model MF in consideration of equivalence.

[0303] For example, the model determination unit 13_4 may set the candidate model M1 having the highest recognition accuracy (or equivalence) among the candidate models M1 having the recognition accuracy equal to or greater than a predetermined threshold and the equivalence equal to or greater than a predetermined threshold as the final model MF. Note that the threshold for determining the recognition accuracy and the threshold for determining the equivalence may be different values ​​or the same value.

[0304] In addition, although Fig.18 The number of pre-compression models M0 and the number of candidate models M1 are two, but the number of pre-compression models M0 and the number of candidate models M1 are not limited to two. The number of pre-compression models M0 and the number of candidate models M1 can be one or three or more. In addition, multiple candidate models M1 can be generated from one pre-compression model M0.

[0305] In addition, although Fig.18 The weight reduction method used in the weight reduction process is pruning, but the weight reduction method is not limited to pruning. For example, the compression processing unit 13_1 can generate the candidate model M1 by using various weight reduction methods such as quantization. In addition, the weight reduction process of different weight reduction methods can be performed on different pre-compression models M0.

[0306] <3.2.2. Second determination process>

[0307] The second determination process is a process of determining the final model MF by adjusting the weight reduction parameter of the weight reduction process according to the result of the equivalence evaluation process. In the second determination process, the weight reduction process, the XAI process, and the equivalence evaluation process are repeatedly performed, and the final model MF is generated.

[0308] Fig.19 : is a flowchart showing an example of the flow of the second determination process according to the embodiment of the present disclosure. Fig.19 The second determination process shown in is executed by each unit of the information processing device 10 .

[0309] like Fig.19 As shown in, for example, the compression processing unit 13_1 of the information processing device 10 sets the weight reduction parameter to an initial value (step S101). Such an initial value may be a preset value or a value specified by the user.

[0310] The compression processing unit 13_1 performs weight reduction processing by using the set weight reduction parameter (step S102). For example, the compression processing unit 13_1 performs weight reduction processing on the pre-compression model M0, and generates a candidate model M1.

[0311] Then, the XAI processing unit 13_2 of the information processing device 10 performs XAI processing (step S103). For example, the XAI processing unit 13_2 performs XAI processing on the pre-compression model M0 and the candidate model M1.

[0312] The evaluation processing unit 13_3 of the information processing device 10 performs equivalence evaluation (step S104). For example, the evaluation processing unit 13_3 performs equivalence evaluation by calculating equivalence of the pre-compression model M0 and the candidate model M1.

[0313] The model determination unit 13_4 of the information processing device 10 determines whether the equivalence value is greater than a threshold value (step S105). In the case where the equivalence is equal to or less than the threshold value (step S105; No), the model determination unit 13_4 changes the weight reduction parameter (step S106), the process returns to step S103, and the compression processing unit 13_1 performs the weight reduction process by using the changed weight reduction parameter.

[0314] On the other hand, in the case where the equivalence is greater than the threshold value (step S105; Yes), the model determination unit 13_4 determines the candidate model M1 as the final model MF (step S107).

[0315] As described above, the information processing device 10 of this embodiment generates the post-compression model M1 (final model MF) in a manner that the equivalence exceeds the threshold value instead of the recognition accuracy (or in addition to the recognition accuracy). As a result, the information processing device 10 can generate the post-compression model M1 (final model MF) with a small change in the model before and after processing.

[0316] Here, it is assumed that the information processing device 10 sets the candidate model M1 having an equivalence greater than a threshold as the final model MF, but the determination method of the information processing device 10 is not limited thereto. The information processing device 10 may determine the final model MF by considering equivalence.

[0317] For example, when the recognition accuracy of the candidate model M1 is greater than a predetermined threshold value (sometimes different from the threshold value used for equivalent judgment) and the equivalent value is greater than the threshold value, the information processing device 10 may terminate the repetitive processing. Alternatively, when the equivalent change amount is less than a predetermined threshold value, the information processing device 10 may terminate the repetitive processing. In addition, the information processing device 10 may terminate the repetitive processing when the repetitive processing is performed a predetermined number of times.

[0318] In addition, here, the information processing device 10 changes the weight reduction parameter when the equivalence is equal to or less than the threshold value, but the weight reduction method may be changed instead of the weight reduction parameter.

[0319] In addition, although it is assumed here that the weight reduction method used in the weight reduction process is pruning, the weight reduction method is not limited to pruning. For example, the compression processing unit 13_1 can generate the candidate model M1 by using various weight reduction methods such as quantization.

[0320] Furthermore, although it is assumed that the information processing device 10 changes the weight reduction parameter based on the equivalence of the pre-compression model M0 and the candidate model M1, the information processing device 10 may change the weight reduction parameter based on an indicator other than the equivalence of the pre-compression model M0 and the candidate model M1.

[0321] For example, the information processing device 10 may change the weight reduction parameter based on the equivalence of the layers of the pre-compression model M0 and the candidate model M1.

[0322] Fig. 20 1 is a diagram for describing another example of the second determination process according to the embodiment of the present disclosure. Here, the compression processing unit 13_1 performs weight reduction processing on the pre-compression model M07 to generate a candidate model M17.

[0323] The evaluation processing unit 13_3 calculates the pre-compression model M0 7 and candidate model M1 7 For example, in Fig. 20 In the example, the XAI processing unit 13_2 respectively targets the pre-compression model M0 7 Layer L0 2 and L0 3 Generate heat map HM0 72 and HM0 73 In addition, the XAI processing unit 13_2 respectively targets the candidate model M1 7 Layer L1 2 and L1 3 Generate heat map HM1 72 and HM1 73 .

[0324] The evaluation processing unit 13_3 calculates the equivalence of each layer. For example, assuming that layer L0 2 and L1 2 The equivalence is 90%, and layer L0 3 and L1 3 The equivalence is 60%.

[0325] In this case, for example, the model determination unit 13_4 increases the layer L0 2 The compression ratio and reduce the layer L03 The compression processing unit 13_1 uses the weight reduction parameter changed by the model determination unit 13_4 to perform compression on the pre-compression model M0. 7 Perform weight reduction processing.

[0326] For example, the information processing device 10 repeatedly changes the weight reduction parameter and performs the weight reduction process until the equivalence of each layer is greater than a specified value. Alternatively, the information processing device 10 may repeatedly change the weight reduction parameter and perform the weight reduction process until the difference in equivalence of each layer is less than a specified value.

[0327] Alternatively, the information processing apparatus 10 may set the number of repetitions and perform the repetitive processing, or may end the repetitive processing in a case where the amount of change in equivalence of each layer becomes equal to or smaller than a predetermined value.

[0328] <3.2.3. Third determination process>

[0329] The third determination processing is a processing of generating the final model MF by using an equivalent evaluation result (such as a result of the XAI processing) as an index (a parameter of the evaluation function) used in the weight reduction processing.

[0330] Fig.21 1 is a view for describing an example of the third determination process according to the embodiment of the present disclosure. Here, a case where the compression processing unit 13_1 adopts distillation as a weight reduction method will be described.

[0331] As the weight reduction process, for example, the compression processing unit 13_1 receives the input image IM8 as input to the candidate model M1 8 The inference result becomes closer to the correct data of the input image IM8 by learning the candidate model M1 8 Here, the correct data for the input image IM8 is a hard target (the score for correct labeling is 1.0, and the score for incorrect labeling is 0.0).

[0332] Furthermore, for example, the compression processing unit 13_1 learns the candidate model M1 8 , in such a way that when the input image IM8 is input to the candidate model M1 8 The inference result becomes closer when the input image IM8 is input to the pre-compression model M0 8 Pre-compression model M0 8 The inference result of is a soft target (the score has a predetermined distribution).

[0333] As above, in general distillation, the correct data of the input image IM8 and the pre-compression model M0 are used. 8 The inference results are used as training data to execute the candidate model M18 of learning.

[0334] In the third determination process according to the present embodiment, in addition to the above-mentioned correct data and inference results, the candidate model M1 is determined using the XAI processing result as an evaluation index for learning (such as training data). 8 of learning.

[0335] For example, in Fig.21 In the process, the XAI processing unit 13_2 generates a pre-compression model M0 8 Heatmap of HM0 8 and candidate model M1 8 Heatmap of HM0 8 The compression processing unit 13_1 learns the candidate model M1 8 , so that the candidate model M1 8 Heatmap of HM0 8 Close to pre-compression model M0 8 Heatmap of HM0 8 .

[0336] In this way, it is assumed that the information processing device 10 generates a candidate model M1 as a student model by using the pre-compression model M0 as a teacher model. In this case, in addition to the correct data of the input image IM8 and the inference result of the pre-compression model M0 of the input image IM, the information processing device 10 also uses the XAI processing result of the pre-compression model M0 (here, the heat map HM0) to learn the candidate model M1 as training data.

[0337] That is, the information processing device 10 learns the candidate model M1 so that the inference result of the candidate model M1 is closer to the correct data, and the inference result of the pre-compression model M0 and the heat map HM1 of the candidate model M1 are closer to the heat map HM0 of the pre-compression model M0.

[0338] Alternatively, the information processing device 10 may learn the candidate model M1 in such a manner that the equivalence between the pre-compression model M0 and the candidate model M1 becomes greater. That is, the information processing device 10 learns the candidate model M1 in such a manner that the inference result of the candidate model M1 is closer to the correct data and the inference result of the pre-compression model M0 and the equivalence between the pre-compression model M0 and the candidate model M1 become greater.

[0339] For example, the model determination unit 13_4 sets the candidate model M1 generated by the compression processing unit 13_1 as the final model MF.

[0340] In addition, it is assumed here that the information processing device 10 performs XAI processing and generates a heat map HM, but the XAI processing performed by the information processing device 10 is not limited to this. For example, the information processing device 10 may perform XAI processing and generate an image block PM, or may generate a feature quantity image. Alternatively, the information processing device 10 may perform XAI processing and generate a semantic concept.

[0341] In addition, although it is assumed here that the compression processing unit 13_1 performs weight reduction processing by using distillation, the weight reduction method used by the compression processing unit 13_1 is not limited thereto. For example, the compression processing unit 13_1 may perform weight reduction processing by using NAS. Also in this case, the information processing device 10 incorporates the result of the XAI processing into the evaluation function of the NAS, performs weight reduction processing in consideration of the XAI processing result (in other words, equivalence), and generates the final model MF.

[0342] Fig. 22 and Fig.23 is a diagram for describing another example of the third determination process according to the embodiment of the present disclosure.

[0343] For example, it is assumed that the compression processing unit 13_1 incorporates the XAI processing result (here, the heat map HM) into the evaluation function of the reinforcement learning type NAS.

[0344] In this case, if Fig. 22 As shown in FIG. 1 , the compression processing unit 13_1 obtains the correct heat map HM2 of the input image IM and the output heat map (such as the candidate model M1 9 Heatmap of HM1 9 ) and learn the candidate model M1 in a distance-reducing way 9 .

[0345] Note that in the absence of a correct heat map HM2, the compression processing unit 13_1 can generate correct data M2 (such as image data including a bounding box containing an object recognized as correct) by using, for example, object recognition, etc. The compression processing unit 13_1 learns the candidate model M1 in a manner that reduces the distance between the correct data M2 and the output heat map. 9 .

[0346] like Fig.23 As shown in , in the absence of a correct heat map HM2, the compression processing unit 13_1 represents the goodness of the output heat map by an index (e.g., the degree of approximation to the RGB edge) and performs learning in such a way that the index becomes better (e.g., the value becomes larger).

[0347] Fig.24 is a diagram for describing another example of the third determination process according to the embodiment of the present disclosure.

[0348] For example, it is assumed that the compression processing unit 13_1 incorporates the XAI processing result (here, the heat map HM) in the evaluation function of the gradient NAS.

[0349] In this case, the compression processing unit 13_1 causes the network structure to output a heat map (eg, candidate model M1 10 Heatmap of HM1 10 ) and the correct heatmap HM2 in a manner that generates a candidate model M1 0 .

[0350] In this way, the information processing device 10 uses the XAI processing result or equivalent as an evaluation index (evaluation function) of the compression processing of the compression processing unit 13_1. As a result, the information processing device 10 can reduce the weight of the learning model while controlling the change of the model before and after the weight reduction processing.

[0351] <<4. Examples of information presentation>>

[0352] The information processing device 10 can present the processing result of each unit to the user as presentation information. For example, the information processing device 10 can present the XAI processing result of the post-compression model M1 to the user as presentation information.

[0353] Fig.25 : is a view for describing an example of presentation information presented by the information processing device 10 according to the embodiment of the present disclosure.

[0354] As described above, the information processing device 10 of this embodiment performs XAI processing on the post-compression model M1. Thus, the information processing device 10 presents the processing result of the XAI processing on the post-compression model M1 to the user as presentation information.

[0355] For example, Fig.25 As shown, it is assumed that the information processing device 10 generates the heat map HM as an XAI process. In this case, the information processing device 10 presents the heat map HM to the user.

[0356] As a result, the user can confirm the fairness of the post-compression model M1 and can confirm the judgment criteria of the estimation result.

[0357] Note that although it is assumed here that the XAI processing result is a heat map HM, the XAI processing result is not limited to the heat map HM and may be, for example, an image block PM or a semantic concept.

[0358] Furthermore, the information processing device 10 may present the XAI processing results of the pre-compression model M0 and the post-compression model M1 to the user.

[0359] Fig.26 is a diagram showing an example of a presentation image presented by the information processing device 10 according to an embodiment of the present disclosure.

[0360] For example, Fig.26 The rendered image shown in the figure may include a schematic diagram of the model before weight reduction (pre-compression model M0) and a schematic diagram of the model after weight reduction (post-compression model M1). In addition, the rendered image may include the pre-compression model M0 and the post-compression model M1 ( Fig.26 XAI processing results of the image block PM in .

[0361] As a result, the user can confirm the fairness of the post-compression model M1 and the judgment criteria of the inference result. In addition, the user can confirm the change in the fairness of the model and the change in the determination basis before and after the weight reduction process.

[0362] Furthermore, the presented image includes the equivalence and recognition accuracy calculated by the evaluation processing unit 13_3, the correctness / incorrectness of the inference result for the input image, and the like.

[0363] Since the demonstration image includes equivalence, the user can quantitatively confirm the changes in the model before and after the weight reduction.

[0364] Furthermore, the presentation image may include a button A1 for receiving a change of an input image, a button A2 for receiving a change of a weight reduction method, and a button A3 for receiving a change of a method of equivalence evaluation (method of XAI processing).

[0365] For example, when the input image changes, the XAI processing result included in the presentation image changes. As a result, the user can confirm the XAI processing results for various input images.

[0366] Furthermore, since the information processing apparatus 10 receives a change in the weight reduction method and a change in the equivalent evaluation method from the user, the user can select the weight reduction method and the method of equivalent evaluation (or the method of XAI processing).

[0367] In addition, although Fig.26 The information processing apparatus 10 presents information (such as equivalence) related to one post-compression model M1 to the user, but the information processing apparatus 10 may present information related to a plurality of post-compression models M1 (the above-mentioned candidate models M1) to the user.

[0368] In this case, for example, Fig.16 As shown in FIG. 1 , the information processing device 10 may present a plurality of candidate models M1 to the user in association with each other. 31 and M1 32Alternatively, the information processing device 10 may also present the user with presentation information related to the plurality of post-compression models M1 using graphics or the like.

[0369] Fig. 27 is a view for describing another example of presentation information presented by the information processing device 10 according to the embodiment of the present disclosure.

[0370] Fig. 27 The presentation information shown in is a graph showing the relationship between the weight reduction parameter and the equivalent amount. Here, the case where the weight reduction parameter is the compression ratio is shown.

[0371] That is, in Fig. 27 In the example shown in , the correspondence between the weight-reduced candidate models M1 with different weight-reduction parameters and the equivalence is represented as a curve graph.

[0372] As described above, the information processing device 10 presents a plurality of candidate models M1 together with equivalent values ​​to the user, so that the user can decide an appropriate final model MF (or weight reduction parameter) based on the equivalent values. In this way, the user can determine the final model MF from the plurality of candidate models M1. In this case, the model determination unit 13_4 determines the final model MF according to the instruction from the user.

[0373] In addition, here, it is assumed that the information processing device 10 prompts the user with information related to the candidate model M1 with different weight reduction parameters, but the information prompted by the information processing device 10 is not limited to this. For example, the information processing device 10 can present to the user information related to the candidate model M1 whose weight is reduced by different weight reduction methods. For example, the information processing device 10 can present to the user the candidate model M1 whose weight is reduced by different weight reduction methods and corresponding equivalence.

[0374] As a result, the user can determine an appropriate final model MF (or weight reduction method) based on equivalence.

[0375] <<5. Model generation process>>

[0376] Fig.28 1 is a flowchart showing an example of the flow of the model generation process according to the embodiment of the present disclosure. The model generation process is executed by the information processing device 10 in the case of generating the final model MF.

[0377] like Fig.28As shown in , the information processing device 10 acquires the pre-compression model M0 (step S201). The information processing device 10 acquires the pre-compression model M0 from the pre-compression model DB 12_1, for example. Alternatively, the information processing device 10 may acquire the pre-compression model M0 by learning the pre-compression model M0 using a learning dataset acquired from the dataset DB 12_3.

[0378] The information processing device 10 obtains the post-compression model M1 (step S202). The information processing device 10 obtains the post-compression model M1 from the post-compression model DB 12_2, for example. Alternatively, the information processing device 10 may obtain the post-compression model M1 by performing a weight reduction process on the pre-compression model M0.

[0379] The information processing device 10 performs the XAI process (step S203). The information processing device 10 performs the XAI process on the pre-compression model M0 and the post-compression model M1.

[0380] The information processing device 10 evaluates the equivalence of the pre-compression model M0 and the post-compression model M1 based on the XAI processing result (step S204 ).

[0381] The information processing device 10 determines the final model MF based on the result of the equivalence evaluation (step S205 ).

[0382] As described above, the information processing device 10 determines the final model MF based on the equivalent evaluation, and thus can further reduce the change in the model before and after the processing in the weight reduction processing.

[0383] <<6. Application Examples>>

[0384] Next, we will refer to Figure 29 to Figure 43 An information processing system 100 to which the information processing device 10 according to an embodiment of the present disclosure is applied is described.

[0385] <6.1. Information Processing System>

[0386] <6.1.1. Overall structure of the system>

[0387] Fig.29 is a diagram showing a configuration example of the information processing system 100 according to an embodiment of the present disclosure.

[0388] like Fig.29 As shown in FIG. 1 , the information processing system 100 includes a cloud server 1, at least one user terminal 2, and at least one camera 3 (in Fig.29 In an example, three or more cameras), fog server 4 and management server 5. The cloud server 1, user terminal 2, fog server 4 and management server 5 can communicate with each other via, for example, a network 6 (such as the Internet).

[0389] The cloud server 1 , the user terminal 2 , the fog server 4 , and the management server 5 may be configured as an information processing device including a microcomputer having a central processing unit (CPU), a read only memory (ROM), a random access memory (RAM), and the like.

[0390] The user terminal 2 is an information processing device assumed to be used by a user who is a recipient of a service using the information processing system 100. In addition, the management server 5 is an information processing device assumed to be used by a service provider.

[0391] The camera 3 is a terminal device including an image sensor such as a charge coupled device (CCD) type image sensor or a complementary metal oxide semiconductor (CMOS) type image sensor. The camera 3 captures an image of a subject and acquires image data (captured image data) as digital data.

[0392] As mentioned above, the camera 3 also has a function of processing the captured image using artificial intelligence (AI) (image recognition processing, image detection processing, etc.). For example, the above-mentioned final model (AI model) can be used for processing using AI.

[0393] In the following description, various processing of images (such as image recognition processing and image detection processing) will be simply referred to as “image processing.” For example, various processing of images processed by using AI (or AI model) will be described as “AI image processing.”

[0394] The camera 3 is configured to be able to communicate data with the fog server 4. The camera 3 transmits various data such as processing result information indicating processing results (image processing, etc.) using AI to the fog server 4, and receives various data from the fog server 4, for example.

[0395] Here, for example, assuming Fig.29 The information processing system 100 shown in FIG. 1 is used to allow a user to view information included in an image captured by a camera 3 via a user terminal 2. In this case, for example, the fog server 4 or the cloud server 1 generates analysis information of the subject based on the processing result information obtained by image processing of each camera 3, and provides the analysis information to the user via the user terminal 2.

[0396] In this case, the camera 3 can be used as various monitoring cameras. For example, the use as a monitoring camera includes the following uses.

[0397] -Indoor surveillance cameras for stores, offices, houses, etc.

[0398] -Outdoor surveillance cameras for monitoring parking lots, urban areas, etc. (including traffic surveillance cameras, etc.)

[0399] - Monitoring cameras for production lines in factory automation (FA) or industrial automation (IA)

[0400] - Surveillance cameras that monitor the interior or exterior of the vehicle

[0401] For example, when used as a surveillance camera in a store, it is conceivable to arrange multiple cameras 3 at predetermined positions in the store and enable users to check customer groups (such as gender and age groups), actions (flow lines) in the store, etc.

[0402] In this case, the fog server 4 or the cloud server 1 can generate, as the above-mentioned analysis information, information on the customer group of customers, information on the flow path in the store, information on the congestion status of the cash register (waiting time at the cash register, etc.), etc.

[0403] Alternatively, when used as a traffic monitoring camera, it is considered that a plurality of cameras 3 are arranged near the road so that the user can recognize information such as the number (vehicle number), vehicle color, and vehicle type of the overtaking vehicle.

[0404] In this case, the fog server 4 or the cloud server 1 generates information such as the number, vehicle color, vehicle type, etc. as the above-mentioned analysis information.

[0405] In addition, in the case of using a traffic monitoring camera in a parking lot, it is conceivable that the camera 3 is set in a manner capable of monitoring each parked vehicle and enables the user to monitor whether there is a person behaving suspiciously around each vehicle. In this case, in the case of a suspicious person, it is conceivable that the information processing system 100 notifies the user of the presence of the suspicious person, the attributes of the suspicious person (gender and age group), etc.

[0406] Furthermore, it is also conceivable that the information processing system 100 monitors empty spaces in an urban area or a parking lot and notifies the user of the location of a space where the vehicle can be parked.

[0407] The fog server 4 is configured for each monitored object. In the above-mentioned use of monitoring a store, for example, it is configured in the store of the monitored object together with the camera 3. In this way, when the fog server 4 is set for each monitored object such as a store, the cloud server 1 does not need to directly receive the transmission data from the multiple cameras 3 of the monitored object, thereby reducing the processing burden of the cloud server 1.

[0408] Note that, in the case where there are multiple stores to be monitored and all the stores belong to the same group, one fog server 4 may be provided for the multiple stores instead of providing one fog server 4 for each store. That is, the fog server 4 does not have to be provided for each monitored object, and one fog server 4 may be provided for multiple monitored objects.

[0409] Note that, in the case where the cloud server 1 or the camera 3 has processing capabilities, the cloud server 1 or the camera 3 may also have the function of the fog server 4. In this case, the fog server 4 may be omitted in the information processing system 100. In this case, the camera 3 may be directly connected to the network 6, and the cloud server 1 may directly receive transmission data from the plurality of cameras 3.

[0410] In the following description, various devices can be roughly divided into cloud-side information processing devices and edge-side information processing devices.

[0411] The cloud-side information processing device is a device group that provides services assumed to be used by a plurality of users, corresponding to the cloud server 1 and the management server 5. The above-described information processing device 10 can be applied to a part of the functions of the cloud-side information processing device.

[0412] Furthermore, the edge-side information processing device corresponds to the camera 3 and the fog server 4 , and can be regarded as a device group arranged in an environment prepared by a user who uses a cloud service.

[0413] However, both the cloud-side information processing device and the edge-side information processing device may be in an environment prepared by the same user.

[0414] Note that the fog server 4 may be an on-premises server.

[0415] <6.1.2. Registration of AI models and AI software

[0416] As described above, in the information processing system 100 of this embodiment, image processing using an AI model and AI utilization software is performed in the camera 3 of the edge side information processing device. In addition, in the cloud server 1 of the cloud side information processing device, advanced application functions are realized by utilizing the result information of the image processing on the camera 3 side.

[0417] Here, various methods for registering an application function in the cloud server 1 (or fog server 4) as a cloud-side information processing device can be considered. Fig.30 Describe the instance.

[0418] Fig.30 is a view for describing an example of an application function registration method according to an embodiment of the present disclosure. Fig.30 The illustration of the fog server 4 is omitted in FIG. 1 , but the fog server 4 may be included in the configuration. In this case, the fog server 4 may share a portion of the functions on the edge side.

[0419] like Fig.30As shown, the cloud-side information processing device 100C includes the above-mentioned cloud server 1 and the management server 5. In addition, the edge-side information processing device 100E has the above-mentioned camera 3.

[0420] Note that each of the cameras 3 can be regarded as a device including a control unit that performs overall control of the camera 3, and the camera 3 can be regarded as a device including another device, which is an image sensor 303, and the image sensor 303 includes an arithmetic processing unit that performs various processes (including AI image processing) on ​​the captured image. That is, it can be understood that the image sensor 303 as another edge-side information processing device is installed inside the camera 3 as an edge-side information processing device.

[0421] In addition, the above-mentioned user terminal 2 may include an application developer terminal 2A, an application user terminal 2B, an AI model developer terminal 2C, etc.

[0422] As described above, the user terminal 2 is an information processing device used by users who use various services provided by the cloud-side information processing device 100C. The application developer terminal 2A is an information processing device used by users who develop applications for AI image processing. The application user terminal 2B is an information processing device used by users who use applications. The AI ​​model developer terminal 2C is an information processing device used by users who develop AI models for AI image processing.

[0423] Note that one information processing device may have multiple functions (such as application user terminal 2B, AI model developer terminal 2C, etc.) That is, for example, one information processing device may be an application user terminal 2B and an AI model developer terminal 2C.

[0424] Note that the application developer terminal 2A can certainly be used by users who develop applications that do not use AI image processing.

[0425] The cloud-side information processing device 100C includes a learning data set for learning by AI (e.g., corresponding to the above-mentioned data set DB 12_3). A user who develops an AI model (hereinafter, also referred to as an AI model developer) can communicate with the cloud-side information processing device 100C by using the AI ​​model developer terminal 2C and download the learning data set.

[0426] At this time, the learning data set can be provided for a fee. For example, the AI ​​model developer can purchase the learning data set in a state where it is possible to purchase different functions and materials registered in the market (electronic market) prepared by the cloud-side information processing device 100. For example, the AI ​​model developer can purchase materials, etc. by registering personal information in the market.

[0427] After developing an AI model by using a learning data set, the AI ​​model developer registers the developed AI model in the market by using an AI model developer terminal 2C. Therefore, when the AI ​​model is downloaded, an incentive can be paid to the AI ​​model developer.

[0428] At this time, the above-mentioned equivalence can be set as a condition for registration in the market. For example, in the case where the developed AI model is the final model MF obtained by weight reduction of the pre-compression model M0, if the evaluation result (such as equivalence) obtained by the equivalence evaluation process is equal to or greater than a preset threshold, the registration of the final model MF in the market is allowed.

[0429] Therefore, for the final model MF registered in the market, fairness equivalent to that of the pre-compression model M0 is ensured.

[0430] In addition, the user who develops the application downloads the AI ​​model from the market by using the application developer terminal 2A, and develops an application (hereinafter, referred to as "AI application") using the AI ​​model. At this time, as above, incentives can be paid to the AI ​​model developer.

[0431] The application development user registers the developed AI application on the market through the application developer terminal 2A. Therefore, when the AI ​​application is downloaded, an incentive can be paid to the user who has developed the AI ​​application.

[0432] At this time, the above-mentioned equivalence can be set as a condition for registration in the market. For example, it is assumed that the application development user performs weight reduction with the developed AI model as the pre-compression model M0 when creating an AI application, and generates a final model MF. In this case, for example, when the evaluation result (such as equivalence) of the equivalence evaluation process of the final model MF is equal to or greater than a preset threshold, the registration of the AI ​​application including the final model MF in the market is allowed.

[0433] Therefore, for the final model MF included in the AI ​​application registered in the market, fairness equivalent to the fairness of the pre-compression model M0 is ensured.

[0434] By using the application user terminal 2B, the user who uses the AI ​​application performs an operation of deploying (deploying) at least one of the AI ​​application or the AI ​​model from the market to the edge-side information processing device 100E managed by him / herself. At this time, incentives can be paid to the AI ​​model developer.

[0435] As a result, the edge-side information processing device 100E, specifically, for example, the camera 3 can perform AI image processing using at least one of the AI ​​application or the AI ​​model. In addition, the edge-side information processing device 100E can not only capture images, but also detect customers and vehicles through AI image processing.

[0436] Here, the operation of deploying at least one of the AI ​​application or the AI ​​model (hereinafter, also referred to as the deployment operation) refers to an operation that enables the target (device) as the execution subject to use the AI ​​application and the AI ​​model. In other words, the deployment operation indicates that the AI ​​application or the AI ​​model is installed on the target as the execution subject in a manner that enables execution of at least a part of the program as the AI ​​application.

[0437] Furthermore, in the camera 3, the attribute information of the customer can be extracted by AI image processing from the captured image captured by the camera 3. The attribute information is transmitted from the camera 3 to the cloud-side information processing device via the network 6, for example.

[0438] A cloud application is developed in the cloud-side information processing device 100C. The user can use the cloud application via the network 6. Then, as the cloud application, for example, an application for analyzing the flow line of a customer by using the customer's attribute information and a captured image is prepared. Such a cloud application is uploaded by, for example, an application development user or the like.

[0439] The application utilizing user performs flow line analysis using, for example, a cloud application through the application user terminal 2B. As a result, the application utilizing user can perform flow line analysis of customers of his own store and view the analysis results.

[0440] As a viewing of the analysis results, the application utilizes a video in which the flow lines of customers are graphically presented on a map of the store, for example, by the user checking. Alternatively, the density of users, etc. can be presented by displaying the results of the flow line analysis in the form of a heat map, and the analysis results can be viewed. In addition, this information can be classified for each piece of attribute information of the customer in the display.

[0441] In the cloud-side market, an AI model optimized for each user can be registered. For example, in the cloud-side information processing device 100C, images captured by the camera 3 disposed in a store managed by a specific user are appropriately uploaded and stored.

[0442] In the cloud-side information processing device 100C, a re-learning process of the AI ​​model is performed every time a certain number of uploaded captured images are accumulated, and a process of updating the AI ​​model and re-registering with the market is performed.

[0443] Note that the relearning process of the AI ​​model can be selected by the user as an option on the marketplace, for example.

[0444] For example, by configuring the camera 3 with an AI model that is relearned using dark images from the camera 3 installed in the store, the recognition rate of image processing for images taken in dark places can be improved. In addition, by configuring the camera 3 with an AI model that is relearned using bright images from the camera 3 installed outside the store, the recognition rate of image processing for images taken in bright places can be improved.

[0445] That is, by redeploying the updated AI model in camera 3, the application user can continuously obtain optimized processing result information.

[0446] In addition, the re-learning process of the AI ​​model will be described later.

[0447] In addition, in the case where personal information is included in the information (such as a captured image) uploaded from the camera 3 to the cloud-side information processing device 100C, data from which information related to privacy is deleted may be uploaded from the aspect of privacy protection. Alternatively, the data from which information related to privacy is deleted may be made available to an AI model development user or an application development user.

[0448] exist Fig.31 and Fig.32 The flow of the above-mentioned processing is shown in the flowchart in . Fig.31 and Fig.32 : is a sequence diagram showing an example of the flow of information processing performed by the information processing system 100 according to an embodiment of the present disclosure.

[0449] The AI ​​model developer checks the list of data sets registered in the market by using an AI model developer terminal 2C having a display unit including, for example, a liquid crystal display (LCD), an organic electroluminescent (EL) panel, etc. The AI ​​model developer selects a desired data set by using the AI ​​model developer terminal 2C. In response to this, the AI ​​model developer terminal 2C sends a download request for the selected data set to the cloud-side information processing device 100C (step S210).

[0450] In response to this, the cloud-side information processing device 100 receives the request (step S10 ), and transmits the requested data set to the AI ​​model developer terminal 2C (step S20 ).

[0451] The AI ​​model developer terminal 2C receives the data set (step S220 ). Therefore, the AI ​​model developer can use the data set to develop an AI model.

[0452] After the AI ​​model developer completes the development of the AI ​​model, the AI ​​model developer performs an operation of registering the developed AI model in the market (e.g., specifying the name of the AI ​​model, the address where the AI ​​model is placed, etc.). As a result, the AI ​​model developer terminal 2C sends a request to the cloud-side information processing device 100C to register the AI ​​model with the market (step S230).

[0453] In response to this, the cloud-side information processing device 100C receives the registration request (step S30) and performs registration processing of the AI ​​model (step S40). As a result, for example, the AI ​​model is displayed on the market. Thereafter, users other than the AI ​​model developer can download the AI ​​model from the market.

[0454] For example, an application developer who wants to develop an AI application checks a list of AI models registered in the market by using the application developer terminal 2A. The application developer terminal 2A responds to the application developer's operation (such as an operation of selecting one of the AI ​​models in the market) and sends a download request for the selected AI model to the cloud-side information processing device 100C (step S310).

[0455] The cloud-side information processing device receives the request (step S50), and sends the AI ​​model to the application developer terminal 2A (step S60).

[0456] The application developer terminal 2A receives the AI ​​model (step S320). As a result, the application developer can develop an AI application using the AI ​​model developed by others.

[0457] When the development of the AI ​​application is completed, the application developer performs operations to register the AI ​​application in the market (such as specifying the name of the AI ​​application, the address where the AI ​​model is placed, etc.). The application developer terminal 2A sends a registration request for the AI ​​application to the cloud-side information processing device (step S330).

[0458] The cloud-side information processing device 100C receives the registration request (step S70), and registers the AI ​​application (step S80). As a result, for example, the AI ​​application is displayed on the market. Thereafter, users other than the application developer can select and download the AI ​​application on the market.

[0459] Then, if Fig.32 As shown in , for example, the application user terminal 2B performs purpose selection according to an instruction from a user who intends to use an AI application (step S410). In this purpose selection, the selected purpose is sent to the cloud-side information processing device 100C.

[0460] In response to this, the cloud-side information processing device 100C selects an AI application corresponding to the purpose (step S90 ), and performs a preparation process (deployment preparation process) for deploying the AI ​​application and the AI ​​model to each device (step S100 ).

[0461] As a deployment preparation process, the cloud-side information processing device 100C determines an AI model, etc. For example, the cloud-side information processing device 100C performs determination of an AI model, etc. based on information of a device for which the deployment process of the AI ​​model or AI application is targeted (such as information of the camera 3 and the fog server 4), performance requested by a user, and the like.

[0462] As deployment preparation processing, based on performance information of each device and request information from the user, the cloud-side information processing device 100C determines which device will execute each software (SW) component included in the AI ​​application for realizing the function desired by the user.

[0463] Each SW component can be a container (described later) or can be a microservice. Note that the SW components can also be implemented by utilizing WebA assembly technology.

[0464] As each SW component, for example, in the case of an AI application that counts the number of customers per attribute such as gender and age, the following SW components and the like may be included.

[0465] -SW component, which detects faces from captured images by using AI models;

[0466] -SW component, extracts human attribute information from face detection results

[0467] - SW component of the aggregation result

[0468] -SW component to visualize the aggregation results

[0469] Some examples of the deployment preparation process will be described again later.

[0470] The cloud-side information processing device 100C performs a process of configuring each SW component in each device (step S110 ). In this process, the AI ​​application and the AI ​​model are sent to each device such as the camera 3 .

[0471] In response to this, the camera 3 executes the deployment process of the AI ​​application and the AI ​​model (step S510 ). As a result, the AI ​​image processing can be performed on the captured image captured by the camera 3 .

[0472] Note that although Fig.32 Not shown in the figure, but the deployment process of AI applications and AI models is performed in a similar manner in the fog server 4 as needed.

[0473] However, in the case where all kinds of processing are executed in the camera 3 , the deployment processing for the fog server 4 is not performed.

[0474] The camera 3 acquires an image by performing an imaging operation (step S520 ). Then, the camera 3 performs AI image processing on the acquired image (step S530 ), and acquires, for example, an image recognition result.

[0475] The camera 3 transmits the captured image and the result information of the AI ​​image processing (step S540). In this information transmission, the camera 3 may transmit both the captured image and the result information of the AI ​​image processing, or may transmit either one of them.

[0476] The cloud-side information processing device 100C that has received the information performs analysis processing (step S120 ), and for example, performs user flow line analysis, vehicle analysis processing for traffic monitoring, and the like as the analysis processing.

[0477] The cloud-side information processing device 100C performs a process of presenting the analysis result (step S130 ). This process is realized, for example, when the user uses the above-mentioned cloud application.

[0478] In response to the presentation process of the analysis result, the application user terminal 2B performs a process of displaying the analysis result on a monitor or the like (step S420 ).

[0479] Through the processing up to this point, the user who is the user of the AI ​​application can acquire the analysis result corresponding to the purpose selected in step S410 .

[0480] In addition, the cloud-side information processing device 100C may also update the AI ​​model after step S130. By updating and deploying the AI ​​model, an analysis result suitable for the user's usage environment may be obtained.

[0481] <6.1.3. System Functional Overview>

[0482] In the present embodiment, as a service using the information processing system 100, it is assumed that a user as a customer can select a service of the type of function for AI image processing of each camera 3. The selection of the function type is, for example, selection of an image recognition function, an image detection function, etc. Alternatively, the selection of the function type may include selecting a more detailed type in a manner of presenting an image recognition function or an image detection function for a specific object.

[0483] For example, as a business model, a service provider sells a camera 3 and a fog server 4 with an image recognition function through AI to a user, and installs the camera 3 and the fog server 4 in a monitored place. Then, a service for providing the above analysis information to the user is developed.

[0484] At this time, the usage required by the system (such as the usage of store monitoring and the usage of traffic monitoring) is different between users. Thus, the information processing system 100 can selectively set the AI ​​image processing function of the camera 3 so as to obtain analysis information corresponding to the usage desired by the customer.

[0485] In the present embodiment, it is assumed that the management server 5 has a function of selectively setting such an AI image processing function of the camera 3 .

[0486] In addition, the cloud server 1 or the fog server 4 may also have the function of the management server 5 .

[0487] Here, we will refer to Fig.33 The connection between the cloud server 1 and the management server 5 as the cloud-side information processing device 100C and the camera 3 as the edge-side information processing device 100E is described.

[0488] Fig.33 It is a view for describing the connection between the cloud-side information processing device 100C and the edge-side information processing device 100E according to an embodiment of the present disclosure.

[0489] In the cloud-side information processing device 100C, a relearning function 102C of functions available via the hub 101C, a device management function 103C, and a market function 104C are implemented.

[0490] The hub 101C performs secure, highly reliable communication with the edge information processing device 100E. As a result, the hub 101C can provide various functions to the edge information processing device 100E.

[0491] The relearning function 102C is a function of performing relearning and providing a newly optimized AI model. Thus, an appropriate AI model based on new learning materials is provided.

[0492] The device management function 103C is a function of managing the camera 3 and the like as the edge-side information processing device 100E. The device management function 103C provides functions such as management and monitoring of the AI ​​model deployed in the camera 3 and fault detection and troubleshooting, for example.

[0493] The device management function 103C is also a function of managing information of the camera 3 and the fog server 4. The information of the camera 3 and the fog server 4 is information of a chip used as an arithmetic processing unit, storage capacity such as a CPU and a memory, information of storage capacity and usage rate, and information of software such as an operating system (OS) installed in each device.

[0494] Furthermore, the device management function 103C protects secure access by authenticated users.

[0495] The market function 104C provides a function of registering AI models developed by AI model developers and AI applications developed by application developers. The market function 104C provides a function of deploying these developed objects in the allowed edge-side information processing device 100E, etc. In addition, the market function 104C also provides functions related to paying incentives corresponding to the deployment of the developed objects.

[0496] The camera 3 as the edge-side information processing device 100E includes an edge runtime 301 , an AI application / AI model 302 , and an image sensor 303 .

[0497] The edge runtime 301 serves as embedded software for managing applications deployed in the camera 3 and communicating with the cloud-side information processing device 100C.

[0498] As above, the AI ​​application / AI model 302 is acquired by deploying the AI ​​application / AI model registered in the market in the cloud-side information processing device 100C. As a result, the camera 3 acquires result information of the AI ​​image processing corresponding to the purpose by using the captured image.

[0499] Fig.34 1 is a block diagram showing a configuration example of a cloud-side information processing device 100C according to an embodiment of the present disclosure. Fig.34 An overview of the functions of the cloud-side information processing device 100C is described. In addition, the cloud server 1, the management server 5, and other devices are collectively referred to as the cloud-side information processing device 100C.

[0500] The cloud-side information processing device 100C includes a license authentication function F1 , an account service function F2 , a device monitoring function F3 , a market function F4 , and a camera service function F5 .

[0501] The license approval function F1 is a function that performs processing related to various types of authentication. Specifically, in the license authentication function F1, processing related to device authentication of each camera 3 and processing related to authentication of each of the AI ​​model, software, and firmware used in the camera 3 are performed.

[0502] The software here refers to the software required to properly implement the AI ​​image processing in the camera 3.

[0503] In order to appropriately perform AI image processing based on the captured image and send the result of AI image processing to the fog server 4 or the cloud server 1 in an appropriate format, it is necessary to control the data input to the AI ​​model and appropriately process the output data of the AI ​​model. The above-mentioned software is software that includes peripheral processing required to appropriately implement AI image processing. This software is software for realizing the desired function by using the AI ​​model, and corresponds to the above-mentioned AI application.

[0504] Note that AI applications are not limited to applications using one AI model, and applications using two or more AI models are also conceivable. For example, there may be an AI application having the following processing flow: information of a recognition result obtained by an AI model performing AI image processing is input as input data (such as image data, and hereinafter referred to as "recognition result information") to another AI model, and a second AI image processing is performed.

[0505] In the permission authentication function F1 , in order to authenticate the camera 3 , when the camera 3 is connected via the network 6 , a process of issuing a device identification (ID) to each camera 3 is performed.

[0506] In addition, regarding the authentication of AI models and software, a process of issuing a unique ID (AI model ID or software ID) is performed for each of the AI ​​models that request AI application registration from the AI ​​model developer terminal 2C and the software developer terminal (not shown). Note that the software developer terminal is an information processing device used by the developer of the AI ​​application.

[0507] Furthermore, in the license authentication function F1, processing for issuing various keys, certificates, and the like to the manufacturer of the camera 3 (specifically, the manufacturer of the image sensor 303 described later), the AI ​​model developer, and the software developer is performed. Various keys, certificates, and the like are used to perform secure communication between the camera 3, the AI ​​model developer terminal 2C, and the software developer terminal (not shown) and the cloud server 1. Furthermore, processing for updating or stopping the validity of a certificate is also performed in the license approval function F1.

[0508] Furthermore, in the permission approval function F1, when user registration (registration of account information with issuance of a user ID) is performed by the account service function F2 described below, processing is also performed to associate the camera 3 (the above-mentioned device ID) purchased by the user with the user ID.

[0509] The account service function F2 is a function for generating and managing the user's account information. The account service function F2 receives input of user information and generates account information based on the input user information (generates account information including at least user ID and password information).

[0510] Furthermore, in the account service function F2, registration processing (registration of account information) of AI model developers and AI application developers (hereinafter, also abbreviated as "software developers") is performed.

[0511] The device monitoring function F3 is a function that performs processing to monitor the usage status of the camera 3. The device monitoring function F3 monitors information such as the usage rates of the CPU and memory as various elements regarding the usage status of the camera 3 as described above, such as the usage location of the camera 3, the output frequency of output data of the AI ​​image processing, and the free space of the CPU and memory used for the AI ​​image processing.

[0512] The market function F4 is a function for selling AI models and AI applications. For example, through a sales website (sales website) provided by the market function F4, users can purchase AI applications and AI models used by AI applications. In addition, software developers can purchase AI models used to create AI applications via the above-mentioned sales site. Note that the market function F4 may have Fig.33 The same functional configuration as shown for the market function 104C.

[0513] The camera service function F5 is a function for providing the user with services related to the use of the camera 3. As an example of the camera service function F5, there is a function related to the generation of the above-mentioned analysis information. This function is a function for generating analysis information of the subject based on the processing result information of the image processing in the camera 3 and performing processing for allowing the user to view the generated analysis information via the user terminal 2.

[0514] Furthermore, the camera service function F5 includes an imaging setting search function. Specifically, the imaging setting search function is a function of acquiring recognition result information of AI image processing from the camera 3 and searching for imaging setting information of the camera 3 by using, for example, AI based on the acquired recognition result information.

[0515] Here, the imaging setting information broadly refers to setting information related to the imaging operation of acquiring the captured image. Specifically, the imaging setting information widely includes optical settings, settings related to the readout operation of the captured image signal, settings related to the image signal processing of the read captured image signal, and the like. For example, the optical settings include settings for focus, aperture, and the like. Settings related to the readout operation of the captured image signal include settings such as frame rate, exposure time, and gain. For example, settings related to the image signal processing for the read captured image signal include settings related to gamma correction processing, noise reduction processing, super-resolution processing, and the like.

[0516] In addition, the camera service function F5 includes, for example, an AI model search function. The AI ​​model search function is a function of searching for an optimal AI model used for AI image processing in the camera 3 by using AI based on the recognition result information of the AI ​​image processing of the camera 3. Here, the search for the AI ​​model includes, for example, a process of optimizing various processing parameters such as a weighting factor, setting information related to a neural network structure (for example, information including a kernel size), etc., in the case where the AI ​​image processing is implemented by a CNN including a convolution operation, etc.

[0517] In addition, the camera service function F5 includes, for example, a processing sharing determination function. In the processing sharing determination function, when the AI ​​application is deployed in the edge-side information processing device 100E, the camera service function F5 performs a process of determining the device of the deployment destination in units of SW components as the above-mentioned deployment preparation process. Note that it may be determined that some SW components are executed in the cloud-side information processing device 100C. In this case, assuming that the deployment has been performed on the cloud-side information processing device 100, the deployment process may not be performed.

[0518] For example, it is assumed that an AI application includes a SW component that detects a person's face, a SW component that extracts attribute information of the person, a SW component that aggregates the extraction results, and a SW component that visualizes the aggregated results.

[0519] In this case, the camera service function F5 uses the image sensor 303 of the camera 3 (see Fig.33 ) is determined as a device of the deployment destination for the SW component for detecting a person's face. The camera service function F5 determines the camera 3 as a device of the deployment destination for the SW component for extracting a person's attribute information. The camera service function F5 determines the fog server 4 as a device of the deployment destination for the SW component for aggregating the extraction results. The camera service function F5 determines that the SW component for visualizing the aggregated results is executed in the cloud server 1 without the need for new deployment to the device.

[0520] In this way, the camera service function F5 determines the processing sharing in each device by determining the deployment destination of each SW component.

[0521] Note that the determination is made in consideration of the specifications and performance of each device and a request from the user.

[0522] Since the camera service function F5 has the imaging setting search function and the AI ​​model search function as above, the imaging setting that makes the result of the AI ​​image processing good is performed. In addition, the camera service function F5 can perform AI image processing by using an appropriate AI model corresponding to the actual use environment.

[0523] Furthermore, since the camera service function F5 has a processing sharing determination function, the camera service function F5 can cause the AI ​​image processing and its analysis processing to be performed in an appropriate device.

[0524] Note that the camera service function F5 has an application setting function before deploying each SW component. The application setting function is a function to set an appropriate AI application according to the user's purpose.

[0525] For example, the camera service function F5 selects an appropriate AI application in response to a user selecting a use such as store monitoring or traffic monitoring. As a result, the SW components included in the AI ​​application are also automatically determined.

[0526] Note that, as described later, there may be a combination of multiple types of SW components for achieving the user's purpose by using an AI application. In this case, one combination is selected based on the information of the edge-side information processing device 100E and a request from the user.

[0527] For example, in the case where the user's purpose is store monitoring, the combination of SW components may vary between the case where the request from the user focuses on privacy and the case where the request from the user focuses on speed.

[0528] In the application setting function, the user performs the following operations on the user terminal 2 (corresponding to Fig.30 The processing of receiving an operation of selecting a destination (application) in the application user terminal 2B), the processing of selecting an appropriate AI application according to the selected application, etc.

[0529] Here, in the above, an example of a configuration in which the cloud-side information processing device 100 independently implements the license approval function F1, the account service function F2, the device monitoring function F3, the market function F4, and the camera service function F5 has been described. Note that these functions can be shared and implemented by multiple information processing devices. For example, it is conceivable that each of the above functions is performed by one information processing device. Alternatively, multiple information processing devices (such as the cloud server 1 and the management server 5) may share and perform a single function of the above functions.

[0530] AI model developer terminal 2C is Fig.29 The information processing device used by the developers of the AI ​​model in the software developer terminal is as described above. In addition, the software developer terminal is an information processing device used by the developers of the AI ​​application.

[0531] <6.1.4. Configuration of imaging device>

[0532] Fig.35 : is a block diagram showing an internal configuration example of the camera 3 according to an embodiment of the present disclosure.

[0533] like Fig.35 As shown, the camera 3 includes an imaging optical system 31, an optical system driving unit 32, an image sensor 303, a control unit 33, a storage unit 34, and a communication unit 35. The image sensor 303, the control unit 33, the storage unit 34, and the communication unit 35 are connected via a bus 36 and can communicate data with each other.

[0534] The imaging optical system 31 includes lenses such as a cover lens, a zoom lens, and a focus lens, and an aperture (aperture) mechanism. Light from a subject (incident light) is guided by the imaging optical system 31 and collected on a light receiving surface of the image sensor 303.

[0535] The optical system driving unit 32 comprehensively indicates a driving unit of the zoom lens, the focus lens, and the diaphragm mechanism included in the imaging optical system 31. Specifically, the optical system driving unit 32 includes an actuator for driving each of the zoom lens, the focus lens, and the diaphragm mechanism, and a driving circuit of the actuator.

[0536] The control unit 33 is constituted by, for example, a microcomputer having a CPU, a ROM, and a RAM. The control unit 33 performs overall control of the camera 3 by causing the CPU to execute various processes according to a program stored in the ROM or a program loaded in the RAM.

[0537] Furthermore, the control unit 33 instructs the optical system driving unit 32 to drive the zoom lens, focus lens, diaphragm mechanism, etc. The optical system driving unit 32 performs movement of the focus lens and zoom lens, opening and closing of diaphragm blades of the diaphragm mechanism, etc. according to these driving instructions.

[0538] Furthermore, the control unit 33 controls writing and reading of various data to and from the storage unit 34 .

[0539] The storage unit 34 is, for example, a nonvolatile storage device such as a hard disk drive (HDD) or a flash memory device, and serves as a storage destination (recording destination) of image data output from the image sensor 303 .

[0540] The communication unit 35 performs various types of data communications with external devices under the control of the control unit 33. The communication unit 35 is configured to be able to communicate with at least Fig.29 The fog server 4 (or cloud server 1) shown in the figure performs data communication.

[0541] The image sensor 303 is configured as, for example, a CCD type or a CMOS type image sensor.

[0542] The image sensor 303 includes an imaging unit 41, an image signal processing unit 42, an in-sensor control unit 43, an AI image processing unit 44, a memory unit 45, and a communication I / F 46. The imaging unit 41, the image signal processing unit 42, the in-sensor control unit 43, the AI ​​image processing unit 44, the memory unit 45, and the communication I / F 46 can communicate data with each other via a bus 47.

[0543] The imaging unit 41 includes: a pixel array unit in which pixels having photoelectric conversion elements such as photodiodes are two-dimensionally arranged; and a readout circuit that reads an electrical signal obtained by photoelectric conversion from each pixel included in the pixel array unit. The imaging unit 41 outputs the electrical signal as a captured image signal.

[0544] The readout circuit performs, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, and the like on the electric signal acquired by the photoelectric conversion, and further performs analog / digital (A / D) conversion processing.

[0545] The image signal processing unit 42 performs pre-processing, synchronization processing, YC generation processing, resolution conversion processing, codec processing, and the like on a captured image signal as digital data after the A / D conversion processing.

[0546] In the pre-processing, a clamping process of clamping the black levels of R, G, and B to predetermined levels, a correction process between the color channels of R, G, and B, and the like are performed on the captured image signal.

[0547] In the synchronization process, color separation processing is performed on the image data of each pixel in a manner including all color components of R, G, and B. For example, in the case of an imaging element using a color filter of a Bayer array, demosaic processing is performed as the color separation processing.

[0548] In the YC generation process, a luminance (Y) signal and a color (C) signal are generated (separated) from the image data of R, G, and B. In the resolution conversion process, a resolution conversion process is performed on the image data on which various signal processes are performed.

[0549] In the codec process, for example, the image data subjected to the above-mentioned various processes is subjected to encoding processing and file generation for recording or communication. In the codec process, a file format that can be executed as a moving image is generated in a file format such as Moving Picture Experts Group (MPEG)-2 or H.264. In addition, as a still image file, a file format such as Joint Photographic Experts Group (JPEG), Tagged Image File Format (TIFF), Graphics Interchange Format (GIF), etc. can be generated.

[0550] The in-sensor control unit 43 performs execution control of the imaging operation by giving instructions to the imaging unit 41. Similarly, the in-sensor control unit 43 also performs control of processing with respect to the image signal processing unit 42.

[0551] The AI ​​image processing unit 44 performs image recognition processing as AI image processing on the captured image.

[0552] The image recognition function using AI can be realized by using a programmable arithmetic processing device such as a CPU, a field programmable gate array (FPGA), or a digital signal processor (DSP).

[0553] The function of image recognition that can be implemented by the AI ​​image processing unit 44 can be switched by changing the algorithm of the AI ​​image processing. In other words, the function type of the AI ​​image processing is switched by switching the AI ​​model used for the AI ​​image processing. Various function types of the AI ​​image processing can be considered, and examples thereof include the types described below as examples.

[0554] - Category identification

[0555] -Semantic Segmentation

[0556] - Human detection

[0557] -Vehicle inspection

[0558] -Target tracking

[0559] -Optical Character Recognition (OCR)

[0560] Among the above-mentioned function types, category recognition is a function of identifying the category of a target. The "category" referred to herein is information indicating the category of an object, and for example, distinguishes "people", "cars", "planes", "ships", "trucks", "birds", "cats", "dogs", "deer", "frogs", "horses", etc.

[0561] Object tracking is a function of tracking a target object, and may be referred to as a function of acquiring historical information of the position of an object.

[0562] The storage unit 45 is used as a storage destination for various data such as the captured image data acquired by the image signal processing unit 42. In addition, the memory unit 45 can also be used to temporarily store data used by the AI ​​image processing unit 44 during AI image processing.

[0563] Furthermore, the storage unit 45 stores information of the AI ​​application and the AI ​​model used in the AI ​​image processing unit 44 .

[0564] Note that the information of the AI ​​application and the AI ​​model may be deployed in the memory unit 45 as a container or the like using container technology (described later), or may be deployed using microservice technology.

[0565] By deploying the AI ​​model for AI image processing in the memory unit 45 , the camera 3 can change the function type of AI image processing or change the AI ​​model to an AI model with improved performance through relearning.

[0566] Note that, as above, in this embodiment, it is assumed that the AI ​​model and the AI ​​application are used for image recognition. However, the AI ​​model and the AI ​​application may be used for a program or the like executed by utilizing AI technology.

[0567] In addition, in the case where the capacity of the storage unit 45 is small, the camera 3 can deploy the information of the AI ​​application and the AI ​​model as a container, etc. to a memory outside the image sensor 303, such as the storage unit 34, by using container technology. After configuration, the camera 3 can store the AI ​​model in the memory unit 45 in the image sensor 303 via the communication I / F 46 (described later).

[0568] The communication I / F 46 is an interface for communicating with the control unit 33, the storage unit 34, and the like outside the image sensor 303. The communication I / F 46 performs communication for acquiring the program executed by the image signal processing unit 42, the AI ​​application and the AI ​​model used by the AI ​​image processing unit 44, and the like from the outside, and performs communication for storing them in the memory unit 45 included in the image sensor 303.

[0569] As a result, the AI ​​model is stored in a part of the memory unit 45 included in the image sensor 303 , and the AI ​​image processing unit 44 becomes able to use the AI ​​model.

[0570] The AI ​​image processing unit 44 recognizes an object according to a purpose by performing predetermined image recognition processing using the AI ​​application or AI model acquired in this manner.

[0571] The recognition result information of the AI ​​image processing is output to the outside of the image sensor 303 via the communication I / F 46. That is, in addition to the image data output from the image signal processing unit 42, the recognition result information of the AI ​​image processing is output from the communication I / F 46 of the image sensor 303.

[0572] Note that image data or recognition result information may be output from the communication I / F 46 of the image sensor 303 .

[0573] For example, when the above-described AI model relearning function is used, captured image data used in the relearning function is uploaded from the image sensor 303 to the cloud-side information processing device 100C via the communication I / F 46 and the communication unit 35 .

[0574] Furthermore, when inference using an AI model is performed, recognition result information of the AI ​​image processing is output from the image sensor 303 to another information processing device outside the camera 3 via the communication I / F 46 and the communication unit 35 .

[0575] Various configurations can be considered for the image sensor 303. Here, an example will be described in which the image sensor 303 has a structure in which two layers are stacked.

[0576] Fig.36 3 is a diagram showing an example of the structure of the image sensor 303 according to an embodiment of the present disclosure. Fig.36 As shown in FIG. 3 , the image sensor 303 is configured as a monolithic semiconductor device of stacked two wafers.

[0577] The image sensor 303 is configured by stacking a chip D1 and a chip D2. The chip D1 has Fig.35 The functions of the imaging unit 41 are shown in FIG. The die D2 includes an image signal processing unit 42 , an in-sensor control unit 43 , an AI image processing unit 44 , a memory unit 45 , and a communication I / F 46 .

[0578] The chip D1 and the chip D2 are electrically connected by, for example, Cu-Cu bonding.

[0579] Various methods of deploying an AI model, an AI application, etc. in the camera 3 may be considered. As an example, an example using a container technology will be described.

[0580] Fig.37 is a view showing a deployment instance of an AI model and an AI application according to an embodiment of the present disclosure. Fig.37 FIG. 3 shows the situation where the AI ​​model and AI application are deployed in camera 3 .

[0581] like Fig.37 As shown in FIG. 1 , in the camera 3, an operating system 51 is installed on various hardware 50, for example, serving as a control unit 33 (see FIG. 1 ). Fig.35 )'s CPU or graphics processing unit (GPU), ROM, and RAM.

[0582] The operating system 51 is basic software that performs overall control of the camera 3 to implement various functions in the camera 3 .

[0583] The general-purpose middleware 52 is software that realizes basic operations such as a communication function using the communication unit 35 as the hardware 50 and a display function using a display unit such as a monitor as the hardware 50 , for example.

[0584] On the operating system 51 , not only the general middleware 52 but also the orchestration tool 53 and the container engine 54 are installed.

[0585] The orchestration tool 53 and the container engine 54 deploy and execute the container 55 by building a cluster 56 as an operating environment for the container 55 .

[0586] Notice, Fig.33 The edge runtime 301 shown in Fig.37 The orchestration tool 53 and container engine 54 shown in FIG.

[0587] The orchestration tool 53 has a function of enabling the container engine 54 to appropriately allocate the resources of the above-mentioned hardware 50 and the operating system 51. The container 55 is placed in a predetermined unit (described later) by the orchestration tool 53 and deployed on a working node (described later), in which the containers are respectively located in logically different areas.

[0588] The container engine 54 is a part of the middleware installed in the operating system 51, and is an engine that operates the container 55. Specifically, the container engine 54 has a function of allocating resources (such as memory and operating capabilities) of the hardware 50 and the operating system 51 to the container 55 based on a setting file or the like included in the middleware in the container 55.

[0589] Furthermore, the resources allocated here include not only resources such as the control unit 33 included in the camera 3 , but also resources such as the in-sensor control unit 43 , the memory unit 45 , and the communication I / F 46 included in the image sensor 303 .

[0590] The container 55 includes an application for realizing a predetermined function and middleware such as a library. By using resources of the hardware 50 and the operating system 51 allocated by the container engine 54, the container 55 operates to realize the predetermined function.

[0591] Fig.33 The AI ​​application / AI model 302 shown corresponds to one of the containers 55. That is, one of the various containers 55 deployed in the camera 3 implements a predetermined AI image processing function using the AI ​​application / AI model 302.

[0592] Reference Fig.38 A specific configuration example of a cluster 56 constructed by a container engine 54 and an orchestration tool 53 is described.

[0593] Fig.38is a diagram showing a configuration example of the cluster 56 according to an embodiment of the present disclosure.

[0594] Note that the cluster 56 may be constructed across a plurality of devices in such a manner as to realize functions by utilizing resources of not only the hardware 50 included in one camera 3 but also resources of other hardware included in other devices.

[0595] The orchestration tool 53 manages the execution environment of the container 55 in units of the worker node 57. In addition, the coordination tool 53 constructs a master node 58 that manages the entire worker node 57.

[0596] In the working node 57, multiple pods 59 are deployed. Each pod 59 is configured to include one or more containers 55 and implement a predetermined function. The pod 59 is a management unit for managing the container 55 through the orchestration tool 53.

[0597] The operation of the pod 59 in the working node 57 is controlled by the pod management library 60 .

[0598] The pod management library 60 includes a container runtime for enabling the pod 59 to use the logical allocation resources of the hardware 50 , an agent receiving control from the master node 58 , a network agent performing communication between pods 59 and with the master node 58 , and the like.

[0599] That is, each pod 59 can use each resource through the pod management library 60 to implement a predetermined function.

[0600] The master node 58 includes an application server 61, a manager 62, a scheduler 63 and a data sharing unit 64. The application server 61 deploys the pod 59. The manager 62 manages the deployment status of the container 55 through the application server 61. The scheduler 63 determines the working node 57 where the container 55 is deployed. The data sharing unit 64 performs data sharing.

[0601] By using Fig.37 and Fig.38 In the configuration shown in , the information processing system 100 can deploy the above-mentioned AI applications and AI models in the image sensor 303 of the camera 3 by using container technology.

[0602] It should be noted that, as above, AI models can be Fig.35 The communication I / F 46 in the image sensor 303 is stored in the memory unit 45 and used for AI image processing in the image sensor 303. Alternatively, Fig.37 and Fig.38 The configuration shown in may be deployed in the memory unit 45 and the in-sensor control unit 43 in the image sensor 303 , and the above-mentioned AI applications and AI models may be executed by utilizing the container technology in the image sensor 303 .

[0603] In addition, as described later, in the case where the AI ​​application and / or AI model is deployed in the fog server 4 or the cloud-side information processing device 100C, container technology may be used.

[0604] At this time, the information of the AI ​​application or AI model is stored in the memory (e.g. Fig.39 It is deployed and executed as a container or the like in the non-volatile storage unit 74, the storage unit 79, or the RAM 73 (described later)).

[0605] <6.1.5. Hardware Configuration of Information Processing Device>

[0606] Reference Fig.39 The hardware configuration of the information processing apparatus 1000 is described, and the information processing apparatus 1000 functions as the cloud server 1, the user terminal 2, the fog server 4, the management server 5, etc. included in the information processing system 100.

[0607] Fig.39 1 is a diagram showing an example of the hardware configuration of the information processing device 1000 according to an embodiment of the present disclosure.

[0608] The information processing apparatus 1000 has a CPU 71. The CPU 71 functions as an arithmetic processing unit that performs the above-described various processes, and performs various processes according to a program stored in a ROM 72 or a nonvolatile storage unit 74 such as an electrically erasable programmable read-only memory (EEP-ROM) or a program loaded from a storage unit 79 to a RAM 73. The RAM 73 also appropriately stores data and the like necessary for the CPU 71 to perform various processes.

[0609] Note that the CPU 71 included in the information processing device 1000 as the cloud server 1 functions as a license approval unit, an account service providing unit, a device monitoring unit, a market function providing unit, and a camera service providing unit to realize the above-described functions.

[0610] The CPU 71, the ROM 72, the RAM 73, and the nonvolatile memory unit 74 are connected to one another via a bus 83. An input / output interface (I / F) 75 is also connected to the bus 83.

[0611] An input section 76 including an operator and an operating device is connected to the input / output interface 75. For example, various operators and operating devices such as a keyboard, a mouse, a key, a dial, a touch panel, a touch pad, and a remote controller are assumed to be the input unit 76. The user's operation is detected by the input unit 76, and a signal corresponding to the input operation is interpreted by the CPU 71.

[0612] Furthermore, a display unit 77 including an LCD, an organic EL panel, or the like, and a sound output unit 78 including a speaker or the like are connected to the input / output interface 75 integrally or individually.

[0613] The display unit 77 is a display unit that performs various displays, and includes, for example, a display device provided in a housing of the computer device, a separate display device connected to the computer device, and the like.

[0614] The display unit 77 displays images for various image processing, moving images to be processed, etc. on the display screen based on the instructions from the CPU 71. In addition, the display unit 77 displays various operation menus, icons, messages, etc., that is, performs display as a graphical user interface (GUI) based on the instructions from the CPU 71.

[0615] In some cases, a storage unit 79 including a hard disk, a solid-state memory, or the like, and a communication unit 80 including a modem or the like are connected to the input / output interface 75 .

[0616] The communication unit 80 performs communication processing via a transmission path such as the Internet, wired / wireless communication with various devices, communication by bus communication, and the like.

[0617] Furthermore, a drive 81 is connected to the input / output interface 75 as necessary, and a removable storage medium 82 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted.

[0618] The driver 81 can read a data file such as a program used for each process from the removable storage medium 82. The read data file is stored in the storage unit 79, and the image and sound included in the data file are output through the display unit 77 and the sound output unit 78. In addition, the computer program and the like read from the removable storage medium 82 are installed in the storage unit 79 as needed.

[0619] In the computer apparatus, for example, software for the processing of the present embodiment can be installed via network communication through the communication unit 80 or the removable storage medium 82. Alternatively, the software may be stored in advance in the ROM 72, the storage unit 79, or the like.

[0620] Furthermore, a captured image captured by the camera 3 or a processing result of the AI ​​image processing may be received and stored in the storage unit 79 or the removable storage medium 82 via the drive 81 .

[0621] When the CPU 71 executes processing operations based on various programs, information processing and communication processing required for the cloud server 1 as the above-mentioned information processing device including the arithmetic processing unit are performed.

[0622] Note that the cloud server 1 may include Fig.34 The single computer device shown may be configured by being systematized by a plurality of computer devices. The plurality of computer devices may be systematized by a local area network (LAN) or the like, or may be arranged in a remote location by using a virtual private network (VPN) or the like using the Internet or the like. The plurality of computer devices may include a computer device as a server group (cloud) that can be used by a cloud computing service.

[0623] <6.2. Update Process>

[0624] After developing the SW components and AI models of AI applications, the relearning of AI models and the updating of AI models deployed in each camera 3, etc. (hereinafter referred to as "edge-side AI models") / AI applications are performed through the operation of the service provider or user (user) as a trigger. Fig.40 Describe in detail the process of relearning and updating processing.

[0625] Fig.40 is a view for describing an example of the flow of a relearning process / update process according to an embodiment of the present disclosure. Note that Fig.40 Focus is placed on one camera 3 among the plurality of cameras 3. In addition, although the edge-side AI model updated in the following description is deployed in the image sensor 303 included in the camera 3 as an example, the edge-side AI model may be deployed outside the image sensor 303 in the camera 3 for sure.

[0626] First, in processing step PS1, the service provider (or user) U instructs to relearn the AI ​​model. This instruction is performed by utilizing the application programming interface (API) function of the API module 110 included in the cloud-side information processing device 100. In addition, in this instruction, the amount of images used for learning (such as the number of pieces) is specified. In the following, the amount of images used for learning is also referred to as a "predetermined number of blocks".

[0627] In response to the instruction, the API module 110 sends a relearning request and image volume information to the center 120 (similar to the center 101C of 33) in process step PS2.

[0628] In processing step PS3 , the hub 120 transmits the update notification and the image amount information to the camera 3 which is the edge-side information processing device 100E.

[0629] In processing step PS4, the camera 3 transmits captured image data obtained by performing photographing to the image database (DB) 131 of the storage management unit 130. The photographing process and the transmitting process are performed until a predetermined number of blocks required for relearning are achieved.

[0630] Note that, in the case where the camera 3 acquires an inference result by performing inference processing on captured image data, the inference result may be stored in the image DB 131 as metadata of the captured image data in processing step PS4.

[0631] Since the inference result in the camera 3 is stored as metadata in the image DB 131, the cloud-side information processing device 100C can carefully select the data required to perform re-learning of the AI ​​model. Specifically, the cloud-side information processing device 100C can perform re-learning using image data whose inference result of the camera 3 is different from the inference result performed by the cloud-side information processing device 100C using abundant computer resources. As a result, the cloud-side information processing device 100C can reduce the time required for re-learning.

[0632] After completing the capture and transmission of the predetermined number of fragments, in process step PS5 the camera 3 notifies the hub 120 that transmission of the predetermined number of fragments of captured image data has been completed.

[0633] When receiving the notification, hub 120 notifies orchestration tool 140 that preparation of data for relearning is complete in process step PS6.

[0634] In processing step PS7 , the orchestration tool 140 transmits the execution instruction of the marking process to the marking module 150 .

[0635] In process step PS8 , the labeling module 150 acquires image data to be a target of labeling processing from the image DB 131 , and performs labeling processing.

[0636] The labeling process here may be a process for performing the above-mentioned category recognition, or may be a process for estimating the gender and age of the subject of the image and giving a label. Alternatively, the labeling process may be a process for estimating the posture of the subject and giving a label, or may be a process for estimating the action of the subject and giving a label.

[0637] The labeling process may be performed manually or automatically. In addition, the labeling process may be completed by the cloud-side information processing device 100C, or may be realized by utilizing a service provided by another server device (not shown).

[0638] The labeling module 150 that has completed the labeling process stores the labeling result information in the data set DB 132 in the processing step PS9. Here, the information stored in the data set DB 132 may be a collection of label information and image data, or may be image identification (ID) information for specifying image data instead of the image data itself. The data set DB 132 corresponds to, for example, the above-mentioned data set DB 12_3 (see Figure 6 ).

[0639] In process step PS10, Fig.40 The storage management unit 130 notifies the orchestration tool 140 of the result information of the detected storage mark.

[0640] When receiving the notification, the layout tool 140 confirms that the labeling process of the predetermined number of image data is completed, and sends a relearning instruction to the relearning module 160 in process step PS11.

[0641] The re-learning module 160 that receives the re-learning instruction obtains the data set for learning from the data set DB 132 in the processing step PS12, and obtains the AI ​​model to be updated from the learned AI model DB 133 in the processing step PS13. For example, the learned AI model DB 133 corresponds to the above-mentioned pre-compression model DB 12_1 (see Figure 6 ).

[0642] Fig.40 The re-learning module 160 in re-learns the AI ​​model by using the acquired data set and the AI ​​model. In the processing step PS14, the updated AI model acquired in this way is stored again in the learned AI model DB 133.

[0643] In processing step PS15 , the storage management unit 130 , which detects that the updated AI model is stored, gives a notification to the orchestration tool 140 .

[0644] In processing step PS16 , orchestration tool 140 that receives the notification sends a conversion instruction of the AI ​​model to conversion module 170 .

[0645] In processing step PS17 , the conversion module 170 that received the conversion instruction acquires the updated AI model from the learned AI model DB 133 , and performs conversion processing of the AI ​​model.

[0646] In the conversion process, a conversion process is performed based on the specification information of the camera 3 of the device as the deployment destination. In this process, miniaturization is performed in a manner that does not degrade the performance of the AI ​​model as much as possible, and file format conversion is performed in a manner that can operate the camera 3. This conversion process corresponds to the above-mentioned weight reduction process. In addition, the conversion process may include the above-mentioned equivalence evaluation process. That is, this conversion process may correspond to the reference Fig.28 The model generation process described in

[17] .

[0647] In other words, the conversion module 170 has Figure 6 The function of each unit of the control unit 13 shown in , and miniaturization is performed in consideration of equivalence with the updated AI model, and a converted AI model is generated.

[0648] Fig.40 The converted AI model on which the conversion module 170 in performs the conversion process is the above-mentioned edge-side AI model. In processing step PS18, the converted AI model is stored in the converted AI model DB 134.

[0649] At this time, for example, the conversion module 170 (or the orchestration tool 140) may store a converted AI model that satisfies a predetermined equivalence evaluation (such as a converted AI model having an equivalence equal to or greater than a predetermined threshold) into the converted AI model DB 134. That is, the cloud-side information processing device 100C may request that a predetermined equivalence evaluation be satisfied as a condition for the edge-side AI model deployed in the edge-side information processing device 100E.

[0650] In processing step PS19 , the storage management unit 130 , which detects that the converted AI model is stored in the converted AI model DB 134 , gives a notification to the orchestration tool 140 .

[0651] In process step PS20, orchestration tool 140 that receives the notification sends a notification for performing an AI model update to center 120. The notification includes information for specifying a location where an AI model for updating is stored.

[0652] In process step PS21, the hub 120 receiving the notification sends an update instruction of the AI ​​model to the camera 3. The update instruction also includes information for specifying a location where the AI ​​model is stored.

[0653] In the processing step PS22, the camera 3 performs a process of acquiring and deploying the AI ​​model after the object transformation from the transformed AI model DB 134. As a result, the AI ​​model used in the image sensor 303 of the camera 3 is updated.

[0654] In processing step PS23 , the camera 3 , which has completed the update of the AI ​​model by deploying the AI ​​model, sends an update completion notification to the hub 120 .

[0655] In process step PS24 , the hub 120 that receives the notification notifies the orchestration tool 140 that the AI ​​model update process of the camera 3 is complete.

[0656] It is noted that although the AI ​​model deployment has been described herein and used for the image sensor 303 of the camera 3 (e.g., Fig.35 303), but the AI ​​model can be deployed outside the image sensor 303. For example, even if the AI ​​model is in the image sensor in the camera 3 (e.g., Fig.35 storage unit 34) external or fog server 4 ( Fig.39In the case where the AI ​​model is deployed and used internally in the storage unit 79), the AI ​​model can be updated similarly.

[0657] In this case, the storage management unit 130 of the cloud-side information processing device 100C stores the device (location) where the AI ​​model is deployed when the AI ​​model is deployed. The hub 120 reads the device (location) where the AI ​​model is deployed from the storage management unit 130 and sends an update instruction of the AI ​​model to the device where the AI ​​model is deployed.

[0658] Specifically, the device that has received the update instruction performs processing and expansion in processing step PS22 to obtain the AI ​​model after the target transformation from the transformed AI model DB 134. Therefore, the AI ​​model of the device that has received the update instruction is updated.

[0659] Note that in the case where the information processing system 100 updates the AI ​​model, the processing is completed here.

[0660] In the case where the information processing system 100 uses the AI ​​model and the AI ​​model updates the AI ​​application, processing described later is further performed.

[0661] like Fig.40 As shown in , specifically, the orchestration tool 140 transmits a download instruction of an updated AI application (such as firmware) to the deployment control module 180 in processing step PS25.

[0662] In process step PS26, deployment control module 180 sends a deployment instruction of the AI ​​application to Hub 120. The instruction includes information for specifying the location where the updated AI application is stored.

[0663] In process step PS27 , the hub 120 sends a deployment instruction to the camera 3 .

[0664] In process step PS28 , the camera 3 downloads the updated AI application from the container DB 181 of the deployment control module 180 and performs deployment.

[0665] Note that in the above description, an example has been described in which the update of the AI ​​model operating on the image sensor 303 of the camera 3 and the update of the AI ​​application operating outside the image sensor 303 in the camera 3 are sequentially performed.

[0666] Furthermore, although the AI ​​application is described here for the sake of simple description, the AI ​​application is defined by a plurality of SW components as above (such as SW components B1, B2, B3, . . . , and Bn).

[0667] Thus, when deploying an AI application, the storage management unit 130 of the cloud-side information processing device 100C or the like stores where each SW component is deployed.

[0668] When processing step PS27 is processed, the hub 120 may read the deployment device (location) of each SW component from the storage management unit 130 and send a deployment instruction to the deployment device.

[0669] In processing step PS28, the device that receives the deployment instruction downloads the updated SW component from the container DB of the deployment control module and executes the deployment.

[0670] Note that the AI ​​applications mentioned here are SW components other than AI models.

[0671] In addition, in the case where both the AI ​​model and the AI ​​application operate in one device, the information processing system 100 can update both the AI ​​model and the AI ​​application together as one container. In this case, the update of the AI ​​model and the update of the AI ​​application can be performed simultaneously instead of sequentially. This can be achieved by performing the processing steps PS25, PS26, PS27, and PS28.

[0672] For example, it is assumed that containers of both the AI ​​model and the AI ​​application are deployable in the image sensor 303 of the camera 3. In this case, the information processing system 100 can update the AI ​​model and the AI ​​application by performing the processing of the above processing steps PS25, PS26, PS27, and PS28.

[0673] By performing the above-mentioned processing, the AI ​​model is relearned by using the captured image data captured in the user's usage environment. Thus, the information processing system 100 can generate an edge-side AI model that can output a highly accurate recognition result in the user's usage environment.

[0674] In addition, even if the user's usage environment changes, for example, if the layout in the store changes or if the installation position of the camera 3 changes, the information processing system 100 can relearn the AI ​​model appropriately each time. Thus, the information processing system 100 can maintain the service without reducing the recognition accuracy of the AI ​​model.

[0675] Note that the above-described respective processes may be performed not only when the AI ​​model is relearned but also when the system is operated for the first time in the user's usage environment.

[0676] <6.3. Market Screen Example>

[0677] An example of a screen presented to a user with respect to a market will be described with reference to the drawings.

[0678] Fig.41is a diagram showing an example of a login screen G1 of a marketplace according to an embodiment of the present disclosure. When a user uses various functions of the marketplace, the login screen G1 is presented to the user.

[0679] The login screen G1 includes an ID input area 91 for inputting a user ID and a password input area 92 for inputting a password.

[0680] Below the password input area 92, a login button 93 for logging in and a cancel button 94 for canceling the login are provided.

[0681] Furthermore, an operator for shifting to a page for a user who has forgotten a password, an operator for shifting to a page for new user registration, and the like are appropriately arranged at the lower part of the page.

[0682] When the user presses the login button 93 after entering the appropriate user ID and password, a process of switching to a user-specific page is performed in each of the cloud server 1 and the user terminal 2 (see Fig.29 ).

[0683] Fig.42 2A is a view illustrating an example of a market developer screen G2 according to an embodiment of the present disclosure. The developer screen G2 is presented to, for example, an AI application developer using the application developer terminal 2A or an AI model developer using the AI ​​model developer terminal 2C.

[0684] Each developer can purchase learning data sets, AI models, and AI applications through the market for development. In addition, each developer can register an AI application or AI model developed by him / herself in the market.

[0685] exist Fig.42 On the developer screen G2 shown in , purchasable learning data sets, AI models, AI applications, etc. (hereinafter collectively referred to as “data”) are displayed on the left.

[0686] Note that, although not shown, when purchasing a learning dataset, an image of the learning dataset is displayed on a display. When only a desired portion of the image is surrounded with a frame by using an input device such as a mouse and a name is input, learning can be prepared.

[0687] For example, in the case where it is desired to perform AI learning using an image of a cat, by surrounding only a portion of the cat on the image using a frame and inputting "cat" as a text input, an image with cat annotations added can be prepared for AI learning.

[0688] In addition, in order to easily find desired data, purposes such as "traffic monitoring", "flow line analysis" and "customer counting" may be selectable. That is, a display process of displaying data suitable for the selected purpose is performed in each of the cloud server 1 and the user terminal 2 (see Fig.29 ).

[0689] Note that the purchase price of each piece of data may be displayed on the developer screen G2.

[0690] Furthermore, on the right side of the developer screen G2, an input field 95 for registering a learning data set collected or created by a developer, and an AI model and an AI application developed by the developer is provided.

[0691] Furthermore, on the right side of the developer screen G2, an input field 95 for inputting a name and a data storage location is provided for each data. Furthermore, a check box 96 for setting necessity / unnecessity of retraining is provided for the AI ​​model.

[0692] Note that, on the right side of the developer screen G2, a price setting field (depicted as an input field 95 in the drawing) in which a price required to purchase data to be registered or the like can be provided.

[0693] Furthermore, in the upper portion of the developer screen G2, the user name, the last login date, etc. are displayed as part of the user information. Note that in addition to the above, the amount of currency and points that can be used when the user purchases data, etc. may be displayed.

[0694] Fig.43 1 is a view showing an example of a market user screen G3 according to an embodiment of the present disclosure. For example, by deploying AI applications and AI models in the camera 3 as an edge-side information processing device 100E managed by himself / herself (the above-mentioned application utilization user), the user screen G3 is presented to the user who performs various analyses, etc.

[0695] For example, via the market, the user can purchase the camera 3 arranged in the space to be monitored. Therefore, on the left side of the user screen G3, a radio button 97 is arranged in which the type and performance of the image sensor 303 mounted on the camera 3, the performance of the camera 3, etc. can be selected.

[0696] In addition, the user can purchase an information processing device as the fog server 4 through the market. Therefore, a radio button 97 for selecting the expression of the fog server 4 is arranged on the left side of the user screen G3.

[0697] Furthermore, a user who already has a fog server 4 can register the performance of the fog server 4 by inputting the performance information of the fog server 4 here.

[0698] The user can realize the desired function by installing the purchased camera 3 at any place such as a store managed by him / herself (optionally, a camera 3 not purchased through the market can be used). In the market, the user can register the information of the installation location of each camera 3 so as to maximize the function of the camera 3.

[0699] On the right side of the user screen G3, radio buttons 98 are arranged in which environmental information on the installation environment of the camera 3 can be selected. When the user appropriately selects environmental information on the installation environment of the camera 3, the above-described optimal imaging setting is set for the target camera 3.

[0700] Note that when purchasing Camera 3 and determining the installation location of the Camera 3 to be purchased, the user can purchase Camera 3 by selecting each item on the left side and each item on the right side of the user screen G3, where the optimal imaging settings corresponding to the planned installation location are pre-set.

[0701] An execution button 99 is provided on the user screen G3. When the user presses the execution button 99, the screen changes from the user screen G3 to a confirmation screen for confirming purchase or a confirmation screen for confirming setting of environmental information. Thus, the user can purchase a desired camera 3 or fog server 4 and set the environmental information of the camera 3.

[0702] In the market, for when the installation location of the camera 3 changes, the user can change the environmental information of each registered camera 3. The user can reset the optimal imaging setting of the camera 3 by re-entering the environmental information about the installation location of the camera 3 on the change screen (not shown).

[0703] <<7. Other Implementation Methods>>

[0704] The processing according to the above-mentioned embodiment (or variant) can be performed in a form (variant) different from the above-mentioned embodiment. For example, in each process described in the above-mentioned embodiment, all or part of the processing described as automatically performed can be performed manually, or all or part of the processing described as manually performed can be automatically performed by a known method. In addition, unless otherwise specified, the processing procedures, specific names and information including various data or parameters described in the above-mentioned documents or drawings can be arbitrarily changed. For example, the various information shown in each drawing is not limited to the information shown.

[0705] In addition, each component of each of the devices shown is a functional concept and does not need to be physically configured in the manner shown. That is, the specific form of allocation / integration of each device is not limited to the form shown in the drawings, and all or part of it can be functionally or physically allocated / integrated in any unit according to various loads and usage conditions.

[0706] <<8. Conclusion>>

[0707] Although the embodiments of the present disclosure have been described above, the technical scope of the present disclosure is not limited to the above embodiments themselves, and various modifications can be made within the spirit and scope of the present disclosure. In addition, components of different embodiments and modified examples can be arbitrarily combined.

[0708] Furthermore, the effects in each embodiment described in this specification are merely examples and are not limitations, and there may be different effects.

[0709] It should be noted that the present technology can also have the following configurations. (1)

[0711] An information processing device, comprising:

[0712] a control unit that evaluates equivalence between a first learning model before weight reduction by the weight reduction method and a second learning model after weight reduction by the weight reduction method, and

[0713] A second learning model is determined based on the evaluation results. (2)

[0715] The information processing device according to (1), wherein the control unit evaluates the equivalence by using an extensible AI (XAI) technique. (3)

[0717] The information processing device according to (1) or (2), wherein the control unit determines the second learning model based on an evaluation value obtained through evaluation of equivalence. (4)

[0719] The information processing device according to any one of (1) to (3), wherein the control unit evaluates the equivalence by using a feature amount used in a process using the first learning model and the second learning model. (5)

[0721] An information processing device according to any one of (1) to (4), wherein the control unit evaluates equivalence based on a first influence given by the data in the data set on the learning of the first learning model and a second influence given by the data in the data set on the learning of the second learning model. (6)

[0723] The information processing device according to any one of (1) to (5), wherein

[0724] The control unit evaluates equivalence between the first learning model and each of the first learning candidate model and the second learning candidate model having different parameters of the weight reduction method, and

[0725] The second learning model is determined from the first learning candidate model and the second learning candidate model based on the result of the evaluation. (7)

[0727] The information processing device according to any one of (1) to (6), wherein

[0728] The control unit evaluates equivalence between the first learning model and each of a first candidate learning model whose weight is reduced by a first weight reduction method and a second candidate learning model whose weight is reduced by a second weight reduction method, and

[0729] The second learning model is determined from the first learning candidate model and the second learning candidate model based on the result of the evaluation. (8)

[0731] The information processing device according to any one of (1) to (7), wherein

[0732] Control unit

[0733] acquiring a first evaluation result obtained by evaluating equivalence between a first pre-weight reduction model and a first learning candidate model, the first learning candidate model being obtained by weight reduction of the first pre-weight reduction model by a weight reduction method, and

[0734] a second evaluation result obtained by evaluating equivalence between the second pre-weight-reduction model and a second learning candidate model obtained by weight-reducing the second pre-weight-reduction model by the weight-reduction method, and

[0735] The second learning model is determined from the first learning candidate model and the second learning candidate model based on the first evaluation result and the second evaluation result. (9)

[0737] The information processing device according to any one of (1) to (5), wherein the control unit adjusts a parameter of the weight reduction method based on an evaluation result of the equivalence, and determines the second learning model. (10)

[0739] The information processing device according to any one of (1) to (5), wherein the control unit uses an evaluation of equivalence as a parameter of an evaluation function of the weight reduction method. (11)

[0741] The information processing device according to any one of (1) to (10), wherein the control unit presents the equivalent evaluation results to a user. (12)

[0743] The information processing device according to any one of (1) to (11), wherein the control unit receives at least one of being sold on the market and being deployed on another device regarding the second learning model, wherein an evaluation of equivalence is equal to or greater than a predetermined value. (13)

[0745] An information processing method, comprising:

[0746] evaluating equivalence between a first learning model before weight reduction by the weight reduction method and a second learning model after weight reduction by the weight reduction method; and

[0747] A second learning model is determined based on the evaluation results. (14)

[0749] A non-transitory computer-readable storage medium storing a program for causing a computer to implement:

[0750] evaluating equivalence between a first learning model before weight reduction by the weight reduction method and a second learning model after weight reduction by the weight reduction method; and

[0751] A second learning model is determined based on the evaluation results. (15)

[0753] A terminal device, comprising:

[0754] A control unit performs processing using a second learning model, wherein:

[0755] The second learning model is

[0756] A learning model is determined based on an evaluation result of equivalence between a first learning model before weight reduction by a weight reduction method and a second learning model after weight reduction by the weight reduction method.

[0757] Reference Numbers List

[0758] 10 Information processing device

[0759] 11 Communication unit

[0760] 12 Storage Units

[0761] 12_1 Pre-compressed model DB

[0762] 12_2 Post-compression model DB

[0763] 12_3 Dataset DB

[0764] 13Control Unit

[0765] 13_1 Compression processing unit

[0766] 13_2XAI Processing Unit

[0767] 13_3 Evaluation Processing Unit

[0768] 13_4 Model determination unit

[0769] 13_5 Output determination unit

[0770] 13_6 Input / output control unit

[0771] 14 Input / Output Units.

Claims

1. An information processing device, comprising: a control unit that evaluates equivalence between a first learning model before weight reduction by a weight reduction method and a second learning model after weight reduction by the weight reduction method, and The second learning model is determined based on a result of the evaluating.

2. The information processing device according to claim 1, wherein: The control unit evaluates the equivalence by using an extensible AI (XAI) technique.

3. The information processing device according to claim 1, wherein: The control unit determines the second learning model based on an evaluation value obtained through the evaluation of equivalence.

4. The information processing device according to claim 1, wherein: The control unit evaluates equivalence by using a feature amount used in a process using the first learning model and the second learning model.

5. The information processing device according to claim 1, wherein: The control unit evaluates equivalence according to a first degree of influence given by the data in the data set on the learning of the first learning model and a second degree of influence given by the data in the data set on the learning of the second learning model.

6. The information processing device according to claim 1, wherein: The control unit evaluates the equivalence between the first learning model and each of a first learning candidate model and a second learning candidate model, wherein the first learning candidate model and the second learning candidate model have different parameters of the weight reduction method, and determines the second learning model from the first learning candidate model and the second learning candidate model based on a result of the evaluation.

7. The information processing device according to claim 1, wherein: The control unit evaluates equivalence between the first learning model and each of a first candidate learning model with weight reduced by a first weight reduction method and a second candidate learning model with weight reduced by a second weight reduction method, and The second learning model is determined from the first learning candidate model and the second learning candidate model based on the result of the evaluation.

8. The information processing device according to claim 1, wherein: The control unit obtaining a first evaluation result obtained by evaluating equivalence between a first pre-weight-reduction model and a first learning candidate model, the first learning candidate model being obtained by weight-reducing the first pre-weight-reduction model by the weight-reduction method, and The second evaluation result is obtained by evaluating the equivalence between the second pre-weight reduction model and the second learning candidate model, wherein the second learning candidate model is obtained by weight reducing the second pre-weight reduction model by the weight reduction method, and the second learning model is determined from the first learning candidate model and the second learning candidate model based on the first evaluation result and the second evaluation result.

9. The information processing device according to claim 1, wherein: The control unit adjusts a parameter of the weight reduction method based on the evaluation result of the equivalence, and determines the second learning model.

10. The information processing device according to claim 1, wherein: The control unit uses the evaluation of the equivalence as a parameter of an evaluation function of the weight reduction method.

11. The information processing device according to claim 1, wherein: The control unit presents the evaluation result of the equivalence to a user.

12. The information processing device according to claim 1, wherein: The control unit receives at least one of being sold on the market and being deployed on another device with respect to the second learning model for which the evaluation of the equivalence is equal to or greater than a predetermined value.

13. An information processing method, comprising: evaluating equivalence between a first learned model before weight reduction by a weight reduction method and a second learned model after weight reduction by the weight reduction method; and A second learning model is determined based on a result of the evaluation.

14. A non-transitory computer-readable storage medium storing a program for causing a computer to implement: evaluating equivalence between a first learning model before weight reduction by a weight reduction method and a second learning model after weight reduction by the weight reduction method; and A second learning model is determined based on the evaluation results.

15. A terminal device, comprising: A control unit performs processing using a second learning model, wherein: The second learning model is A learning model determined based on a result of evaluating equivalence between a first learning model before weight reduction by a weight reduction method and a second learning model after weight reduction by the weight reduction method.

Images