Information processing device, information processing method, and learning system

Abstract

An information processing device according to the present invention comprises: an evaluation value calculation unit that calculates a variation evaluation value indicating the degree of fluctuation of the weight coefficients of the convolution filter when training data is input into an AI model during the training of the AI model; and an inference unit that infers, on the basis of the variation evaluation value calculated by the evaluation value calculation unit and on the basis of annotation information of the training data, an attribute of the training data which is insufficient or excessive in the training of the AI model.

Description

Information processing device, information processing method, and learning system

[0001] The present technology relates to an information processing device, an information processing method, and a learning system, and in particular to a technology related to learning of an AI (Artificial Intelligence) model.

[0002] There are various techniques for performing inference processing using AI (artificial intelligence) models. In training an AI model, using appropriate training data is considered important for improving performance. For example, Patent Document 1 below discloses a technique for estimating bias in training data based on annotation information in the training data and the inference results of the AI ​​model, and adjusting the training data based on information about the estimated bias. Specifically, Patent Document 1 below discloses that, for an AI model performing object detection processing targeting people, if there are many false detections of people in their 30s, it is estimated that there is insufficient training data with people in their 30s as subjects, and the missing data is added to the training data.

[0003] Japanese Patent Application Laid-Open No. 2021-111101

[0004] However, the accuracy of an inference result does not necessarily indicate the accuracy of the operation of an AI model. For example, in a well-known example, in an AI model that performs object detection processing targeting a specific subject, a correct detection result may be obtained even though an area in an input image different from the image area of ​​the subject is referenced. For this reason, if bias estimation of training data is performed based on the inference result as in Patent Document 1, specifically, if data that is insufficient for training (or data that is considered excessive) is performed, there is a risk that bias of the training data may not be accurately estimated.

[0005] This technology was developed in consideration of the above circumstances, and aims to improve the learning accuracy of AI models by improving the estimation accuracy of data that is insufficient or excessive for learning data used in AI model learning.

[0006] The information processing device according to the present technology includes an evaluation value calculation unit that calculates a degree of fluctuation in a weight coefficient of a convolution filter when training data is input to the AI ​​model during training of the AI ​​model, and an estimation unit that estimates attributes of the training data that are considered to be insufficient or excessive in training of the AI ​​model based on the degree of fluctuation in the weight coefficient of the convolution filter as described above, in other words, the evaluation value that indicates the responsiveness of the convolution filter.

[0007] Furthermore, a learning system according to the present technology includes an evaluation value calculation unit that calculates an intensity evaluation value indicating the degree of intensity fluctuation of a weight coefficient of a convolution filter when training data is input to the AI ​​model during training of the AI ​​model, an estimation unit that estimates attributes of the training data that are deemed to be insufficient or excessive in training of the AI ​​model based on the intensity evaluation value calculated by the evaluation value calculation unit and annotation information of the training data, a data adjustment unit that adjusts the training data by at least one of adding insufficient data identified from information on the attributes estimated by the estimation unit and removing excessive data identified from the information on the attributes, and a learning processing unit that performs training processing of the AI ​​model using the training data adjusted by the data adjustment unit. This learning system, like the information processing device according to the present technology described above, also enables appropriate evaluation of the correctness of the operation of the AI ​​model, improves the accuracy of estimation of data that is insufficient or excessive for training, and makes it possible to appropriately adjust the training data.

[0008] 1 is a block diagram showing an example of the configuration of a learning system as an embodiment. FIG. 2 is an explanatory diagram of an example of a CG image used in the relearning process. FIG. 3 is a diagram showing an example of the hardware configuration of an information processing device as an embodiment. FIG. 4 is a functional block diagram for explaining functions possessed by the information processing device and the data generating device in the first embodiment. FIG. 5 is a diagram showing an example of a first presentation screen. FIG. 6 is a diagram showing an example of a selection screen. FIG. 7 is a diagram showing an example of an imaging condition instruction screen. FIG. 8 is a flowchart showing an example of a specific processing procedure for realizing the learning method as the first embodiment. FIG. 9 is a flowchart showing an example of a specific processing procedure for estimating attributes of missing data and excess data based on an intensity evaluation value. FIG. 10 is an explanatory diagram of another example of a method for estimating attributes of missing data and excess data. FIG. 11 is an explanatory diagram of an example configuration in which an information processing device has a data generating unit. FIG. 12 is an explanatory diagram of an example configuration in which a device separate from the information processing device has a learning processing unit. FIG. 13 is a functional block diagram showing an example of the functional configuration of an information processing device as a second embodiment. FIG. 14 is an image diagram of learning when CG images are used as learning data in the second embodiment. FIG. 15 is a flowchart showing an example of a processing procedure for realizing the learning method as the second embodiment.

[0009] Hereinafter, with reference to the accompanying drawings, embodiments according to the present technology will be described in the following order: <1. First embodiment> (1-1. Example of configuration of learning system) (1-2. Example of hardware configuration of information processing device) (1-3. Learning method as an embodiment) (1-4. Processing procedure) <2. Second embodiment> <3. Modification> <4. Summary of embodiments> <5. Present technology>

[0010] 1. First Embodiment (1-1. Configuration Example of a Learning System) FIG. 1 is a block diagram showing a configuration example of a learning system according to an embodiment, which is configured to include an information processing device 1 according to a first embodiment of the present technology. The learning system according to the embodiment is a system for training an AI (Artificial Intelligence) model that performs inference processing for a predetermined task using a predetermined type of data, such as image data, sound data, or text data, as input data. As an example, the learning system according to the embodiment trains an AI model that uses image data as input data to achieve a predetermined image analysis task, such as object detection processing or object recognition processing. Note that the object detection processing here refers to a task of obtaining the position of an object in an image, and the inference result is output as region information, such as a bounding box indicating the area where the object exists. Furthermore, the object recognition processing refers to a task of recognizing what object appears in an input image, and the inference result is output as a likelihood for each object class, such as a person, dog, or cat. Here, the image analysis task may be a task that combines object detection processing and object recognition processing, specifically, a task of recognizing (classifying) an object while also detecting its area. Furthermore, a task such as semantic segmentation, which classifies objects in predetermined block units such as pixel units, may also be considered.

[0011] In this embodiment, it is assumed that inference processing using a trained AI model is executed within a camera. Specifically, it is assumed that an AI processing unit that performs inference processing using an AI model is provided within an image sensor in the camera. Examples of uses of such cameras include use as surveillance cameras in stores, on roads, parking lots, etc. The AI ​​processing unit performs inference processing targeting, for example, people or automobiles, and the inference results can be used to analyze customer trends in a store, the amount of automobile traffic, the number of parked cars, etc.

[0012] As shown in the figure, the learning system of the embodiment includes at least an information processing device 1, a data generating device 2, and a user terminal 3. In the learning system, the information processing device 1, the data generating device 2, and the user terminal 3 are each configured as a computer device, and are capable of mutual data communication via a network NT, which is a communication network such as the Internet.

[0013] The information processing device 1 performs a learning process for an AI model and creates a trained AI model. The data generating device 2 generates learning data to be used in the learning process for the AI ​​model in the information processing device 1.

[0014] The user terminal 3 is a computer device used by a user who receives at least one of the inference results of the trained AI model and the analysis results based on the inference results. For example, in the case of the above-mentioned surveillance camera application in a store, the user would be an employee of the company that operates the store.

[0015] Here, in the embodiment, the AI ​​model is trained as follows: first, initial training is performed using real-life images associated with annotation information such as those provided by ImageNet, and then CG (Computer Graphics) images are used to train the AI ​​model so that it can achieve performance appropriate for the inference environment.

[0016] In training an AI model, it would be ideal to create a highly versatile AI model that can handle a variety of situations, but realizing such a versatile AI model requires training using a huge amount of training data, which increases the training cost.Furthermore, realizing a highly versatile AI model requires the use of a relatively large-scale neural network, which also leads to an increase in hardware resources.

[0017] In the embodiments, it is assumed that inference processing using a trained AI model is performed on an edge device with relatively limited hardware resources, such as a camera. For this reason, a method is adopted in which, after initial training as described above, training data is generated using CG and then retrained. To improve adaptability to the actual inference environment, it would be ideal to use real-life images captured in the actual camera installation environment, but this would require a lot of effort from the user. Training using CG images eliminates the need to capture real-life images, while enabling the generation of appropriate training data tailored to the inference environment, allowing for efficient retraining of the AI ​​model after initial training. Furthermore, using CG images prevents images of actual people from being included in the training data, thereby avoiding the risk of privacy violations.

[0018] FIG. 2 is an explanatory diagram of an example of a CG image used in the re-learning process. Here, an example of a CG image is described in which a camera having an AI processing unit is used as a surveillance camera in a store. In this case, to generate the CG image, as shown in FIG. 2A , a 3D (three-dimensional) model is generated that reproduces the store structure and a person as a target customer. At this time, the 3D model of the person and the 3D models of store components such as shelves and walls can be variably set in terms of their position, orientation, shape, size, color, etc. Furthermore, lighting conditions for the 3D model (e.g., lighting conditions for indoor lighting and natural light entering through a window) can be variably set. Furthermore, the position and angle of view of the camera's viewpoint Pv relative to the 3D model can also be set.

[0019] By rendering a two-dimensional image of a 3D model captured from a specified viewpoint Pv at a specified angle of view, learning data can be generated using CG images such as those shown in Figure 2B. By being able to variably set the position, color, shape, and lighting conditions of objects such as people, it is possible to generate images that reproduce a variety of scenes. Furthermore, by being able to specify the viewpoint Pv and angle of view for the 3D model, it is possible to reproduce images based on the position and angle of view of a camera actually installed in a store.

[0020] In FIG. 1, a data generating device 2 is a device that generates learning data using such CG images.

[0021] (1-2. Example of hardware configuration of information processing device) Fig. 3 is a block diagram showing an example of the hardware configuration of the information processing device 1. Note that the data generation device 2 and the computer device serving as the user terminal 3 shown in Fig. 1 can also have the same hardware configuration as shown in Fig. 3, and therefore a graphical description of the hardware configuration of these data generation device 2 and user terminal 3 will be omitted.

[0022] As shown in the figure, the information processing device 1 includes a CPU 11. The CPU 11 executes various processes in accordance with programs stored in a ROM 12 or programs loaded from a storage unit 19 into a RAM 13. The RAM 13 also stores data necessary for the CPU 11 to execute various processes as appropriate.

[0023] The CPU 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output interface (I / F) 15 is also connected to this bus 14.

[0024] An input unit 16 consisting of operators and operation devices is connected to the input / output interface 15. For example, the input unit 16 may be various operators and operation devices such as a keyboard, a mouse, keys, a dial, a touch panel, a touch pad, a remote controller, etc. An input operation is detected by the input unit 16, and a signal corresponding to the input operation is interpreted by the CPU 11.

[0025] A display unit 17, such as an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) panel, and an audio output unit 18, such as a speaker, are connected integrally or separately to the input / output interface 15. The display unit 17 is used to display various types of information, and may be, for example, a display device provided in the housing of the computer device, or a separate display device connected to the computer device.

[0026] The display unit 17 displays images for various image processing, moving images to be processed, etc. on the display screen based on instructions from the CPU 11. The display unit 17 also displays various operation menus, icons, messages, etc., i.e., a GUI (Graphical User Interface), based on instructions from the CPU 11.

[0027] The input / output interface 15 may be connected to a storage unit 19 configured with a hard disk drive (HDD) or solid-state memory, or a communication unit 20 configured with a modem or the like.

[0028] The communication unit 20 performs communication processing via a transmission path such as the Internet, and communication with various devices via wired / wireless communication, bus communication, and the like.

[0029] A drive 21 is also connected to the input / output interface 15 as required, and a removable recording medium 22 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is appropriately loaded therein.

[0030] The drive 21 can read data files such as programs used for various processes from the removable recording medium 22. The read data files are stored in the storage unit 19, and images and sounds contained in the data files are output on the display unit 17 and the audio output unit 18. Furthermore, the computer programs and the like read from the removable recording medium 22 are installed in the storage unit 19 as needed.

[0031] In a computer device having the above-described hardware configuration, for example, software for the processing of this embodiment can be installed via network communication by the communication unit 20 or via the removable recording medium 22. Alternatively, the software may be stored in advance in the ROM 12, the storage unit 19, etc. The CPU 11 performs processing operations based on various programs, thereby executing the information processing and communication processing required by the information processing device 1.

[0032] The information processing device 1 is not limited to being configured as a single computer device as shown in Fig. 3, but may be configured as a system of multiple computer devices. The multiple computer devices may be systemized using a LAN (Local Area Network) or the like, or may be located in a remote location using a VPN (Virtual Private Network) or the like using the Internet or the like. The multiple computer devices may include computer devices as a server group (cloud) available through a cloud computing service.

[0033] (1-3. Learning Method as First Embodiment) A learning method as the first embodiment will be described with reference to Fig. 4 to Fig. 7. Fig. 4 is a functional block diagram for explaining the functions of the information processing device 1 and the data generating device 2 in the first embodiment.

[0034] As shown in the figure, the data generating device 2 functions as a data generating unit F20. The data generating unit F20 generates learning data using CG images as described above. Specifically, the data generating unit F20 generates CG images based on image content in accordance with instructions from the information processing device 1.

[0035] The information processing device 1 has functions as a learning processing unit F1, an evaluation value calculation unit F2, an estimation unit F3, a first presentation processing unit F4, a second presentation processing unit F5, a data adjustment unit F6, and a learning control unit F7. In this example, the functions of the learning processing unit F1 to the learning control unit F7 are realized by processing of the CPU 11, but it is not essential to adopt a configuration in which all of these functions are realized by processing of the CPU 11, and it is also possible to adopt a configuration in which at least some of the functions are realized by processing other than that of the CPU 11.

[0036] The learning processing unit F1 has a learning device F11 of the AI ​​model, and performs a learning process of the AI ​​model by inputting learning data to the learning device F11. The learning processing unit F1 in this embodiment performs the learning process as the initial learning described above, and also performs the learning process as the relearning described above. In the learning process as relearning, the learning processing unit F1 performs the learning process of the AI ​​model using learning data adjusted by the data adjustment unit F6 described below.

[0037] The evaluation value calculation unit F2 calculates a fluctuation evaluation value that indicates the degree of fluctuation in the weight coefficient of the convolution filter when learning data is input to the AI ​​model during learning of the AI ​​model.

[0038] The estimation unit F3 estimates attributes of the learning data that are considered to be insufficient or excessive in learning the AI ​​model based on the violence evaluation value calculated by the evaluation value calculation unit F2 and annotation information of the learning data.

[0039] The data adjustment unit F6 adjusts the learning data by at least one of adding missing data identified from the attribute information estimated by the estimation unit F3 and removing excess data identified from the attribute information.

[0040] The learning control unit F7 controls the learning processing unit F1 to perform learning processing of the AI ​​model using the learning data adjusted by the data adjustment unit F6.

[0041] In this embodiment, the addition of training data to be used for re-learning is performed by expanding the amount and distribution of the training data by generating new data, such as by transforming or combining the features of the training data (also called "clean data") used in the initial training, as in so-called DA (Data Augmentation). Regarding the addition of training data as DA and the adjustment of training data as the deletion of excessive training data, the aforementioned Patent Document 1 estimates bias in the training data based on annotation information of the training data and the inference results of an AI model, and adjusts the training data based on information about the estimated bias.

[0042] However, as mentioned above, the correctness of the inference result does not necessarily indicate the correctness of the operation of the AI ​​model. Therefore, if bias estimation of training data is performed based on the inference result as in Patent Document 1, the bias of the training data cannot be correctly estimated, and there is a risk that the training data cannot be adjusted appropriately.

[0043] In contrast, in this embodiment, a violence evaluation value indicating the degree of fluctuation of the weight coefficient of the convolution filter is calculated as described above, and attributes of learning data that are considered to be insufficient or excessive in learning of the AI ​​model are estimated based on this violence evaluation value. The violence evaluation value can be said to be an evaluation value indicating the reactivity of the convolution filter, and if the "degree of fluctuation" indicated by the violence evaluation value is large, it means that there is a lot of learning remaining.

[0044] By using the above-described intensity evaluation value, it becomes possible to appropriately evaluate the correctness of the operation of the AI ​​model. Therefore, it is possible to improve the estimation accuracy of data that is insufficient or excessive for learning data used in learning the AI ​​model, and to improve the learning accuracy of the AI ​​model.

[0045] Here, currently, adjustment of training data for re-learning is generally performed manually, and while experienced experts tend to perform appropriate adjustments (adjustments that achieve the intended inference performance), it is considered difficult for inexperienced and unfamiliar people to perform appropriate adjustments. Conventional training data adjustments rely heavily on human intuition and do not necessarily improve learning accuracy.

[0046] In this embodiment, the evaluation value calculation unit F2, estimation unit F3, and data adjustment unit F6 described above can perform an appropriate quantitative evaluation of the operation of the AI ​​model and an appropriate estimation of the excess or deficiency of learning data based on the quantitative evaluation, so that learning data can be adjusted appropriately without relying on human intuition.

[0047] A specific example of a method for estimating attributes of training data that are insufficient for training and attributes of training data that are considered to be excessive will be described.

[0048] In this example, the evaluation value calculation unit F2 calculates a value based on saliency as the fluctuation evaluation value. In this specification, "saliency" refers to a value obtained by averaging the weighting coefficients in the convolution filter of the AI ​​model. This saliency allows the degree of fluctuation in the weighting coefficients of the convolution filter to be appropriately evaluated, improving the estimation accuracy of the estimation unit F3 and thereby improving the learning accuracy of the AI ​​model.

[0049] For saliency, we use the same one as described in Reference 1 below. Reference 1: "Where do Models go Wrong? Parameter-Space Saliency Maps for Explainability" Roman Levin, Manli Shu, Eitan Borgnia, Furong Huang, Micah Goldblum, Tom Goldstein: arXiv:2108.01335

[0050] This saliency can be calculated for each convolution filter in an AI model, allowing for more diverse evaluations than a loss value (error function) that can only be calculated as a single value for the entire AI model. Furthermore, since it is a value that can be calculated for each convolution filter, saliency can be calculated in various units, such as for each pixel of the input image (learning data) or for each layer. When an AI model performs object recognition processing, saliency calculated for each pixel can also be used as a value indicating the magnitude of the influence that pixel has on the score of the predicted class.

[0051] Note that the number of convolution filters in an AI model may reach several thousand, and it may not be appropriate to use the saliency calculated for each filter as an evaluation value as is. Taking this into consideration, the intensity evaluation value may be calculated as a moving average value of the saliency for each filter. For example, when the number of convolution filters is several thousand, the number of sections may be set to 100, and the moving average value of the saliency for each filter may be calculated.

[0052] In this example, to estimate the attributes of the training data that are insufficient for training or the attributes of the training data that are deemed to be excessive, not only the intensity evaluation value such as Saliency as described above but also the evaluation values ​​such as Diversity and Affinity are used.

[0053] For more information on diversity and affinity, please refer to Reference 2 below. Reference 2: "Affinity and Diversity: Quantifying Mechanisms of Data Augmentation" Raphael Gontijo-Lopes, Sylvia J. Smullin, Ekin D. Cubuk, Ethan Dyer: arXiv:2002.08973

[0054] As disclosed in Reference 2, diversity is an evaluation value that quantifies the degree of complexity (diversity) of data expanded in relation to an AI model and a learning procedure. By using such an evaluation value as diversity together with an evaluation value such as saliency, it becomes possible to estimate the attributes of data that are insufficient for learning or data that is considered excessive in terms of data diversity. In other words, it becomes possible to adjust the training data to be used for re-learning so that the distribution of the training data is appropriate in terms of diversity.

[0055] Furthermore, affinity is an evaluation value that quantifies the degree of similarity (affinity) between the distribution of the training data used to train the AI ​​model and the distribution of the augmented data. By using such an evaluation value as affinity together with an evaluation value such as saliency, it becomes possible to estimate the attributes of data that are insufficient for training or data that is considered excessive in terms of data affinity. In other words, it becomes possible to adjust the training data so that the distribution of the training data used for re-training is appropriate in terms of affinity.

[0056] A specific method for estimating the attributes of excess or deficiency data using these Diversity and Affinity and the intensity evaluation value will be described below.

[0057] In this example, the estimation unit F3 estimates that attributes of the training data for which the violence evaluation value is equal to or greater than a threshold and the diversity value is equal to or less than a threshold are attributes that are missing in the training. Here, for example, assume that the AI ​​model performs object detection processing targeting automobile objects, and that the AI ​​model has insufficient object detection accuracy for automobiles, such as light cars and red cars. In this case, if the training data to be evaluated includes light cars and red cars as subjects, the violence evaluation value will be large and the diversity value will be small. Therefore, by setting the estimation condition that the violence evaluation value is equal to or greater than a threshold and the diversity value is equal to or less than a threshold, as described above, it is possible to appropriately estimate data that is missing in the training.

[0058] In this case, for vehicle models with little real-world test data, the lack of data is difficult to detect, but by using Diversity, it is possible to properly detect the lack of data for such vehicle models.

[0059] Furthermore, in this example, the estimation unit F3 estimates that attributes of the training data for which the violence evaluation value is equal to or greater than a threshold and the affinity value is equal to or less than a threshold are attributes that are missing in the training. For example, assume that the AI ​​model performs object detection processing targeting automobiles, and that the object detection accuracy in a particular scene involving a red car is insufficient, so that it is desired to increase the scene variety of the training data for red cars. In this case, if the training data to be evaluated includes a red car as a subject, the violence evaluation value will be large and the affinity value will be small. Therefore, by setting the estimation condition that the violence evaluation value is equal to or greater than a threshold and the affinity value is equal to or less than a threshold, as described above, it is possible to appropriately estimate the data that is missing in the training. Specifically, it can be estimated that, for red cars, training data for scenes other than the training data to be evaluated should be increased.

[0060] In this case, a small affinity for the training data of "red car" means that there is insufficient affinity for "red car" and there is insufficient scene variation for "red car," and training data should be generated to increase the scene variation for the training data of "red car." For example, if the scene attribute of the training data to be evaluated is "daytime," additional training data with a scene attribute other than "evening" or "night" should be generated as training data for "red car."

[0061] Furthermore, in this example, the estimation unit F3 estimates an attribute of the learning data for which the intensity evaluation value is equal to or less than a threshold and the affinity is equal to or greater than a threshold as an attribute that is deemed excessive in learning. When the intensity evaluation value is equal to or less than a threshold (low degree of intensity), it can be estimated that learning is sufficient for the attribute of the learning data to be evaluated. When the affinity is equal to or greater than a threshold (excessive affinity), it can be estimated that there is an excess of learning data with attributes similar to the learning data to be evaluated. Therefore, by setting the estimation condition that the intensity evaluation value is equal to or less than a threshold and the affinity value is equal to or greater than a threshold as described above, it is possible to appropriately estimate data that is deemed excessive in learning. For example, if the learning data for white commercial vehicles accounts for a considerable amount, such as 70%, of the learning data for automobiles, it can be estimated that the attribute of "white car" is an attribute of excessive learning data.

[0062] Here, the estimation unit F3 uses annotation information associated with the learning data as extended data input to the AI ​​model to estimate the attributes of the excess and deficient data, but in this example, the association of annotation information with the learning data as extended data is performed by the data generation unit F20 in the data generation device 2.

[0063] For clarity, the term "annotation information" used here is not limited to information used as training data in learning, but broadly refers to information associated with learning data. In this example, the learning data (CG images) as augmented data may be generated as images containing multiple target objects for inference processing by the AI ​​model. Therefore, as annotation information to be associated with the learning data, the data generation unit F20 generates coordinate information of the bounding boxes of the target objects and, for each target object included in the image, generates information indicating the attributes of the target objects and associates the information with the generated learning data.

[0064] For example, if the target object is a person, the attribute information of the target object may include any information indicating attributes of a person, such as age, gender, skin color, hairstyle, body type, clothing color, and orientation (sideways, forward, backward, etc.). If the target object is a vehicle, any information indicating attributes of the vehicle, such as the vehicle model (large, medium, compact, sedan, wagon, van, etc.), color, door open / close status, orientation, etc. The attribute information of the target object may also include information on the lighting conditions of the target object, specifically, information indicating the illumination conditions of, for example, indoor lighting or natural light irradiated on the target object. Because lighting condition information indicates how the target object is illuminated in the image, it falls under the category of information indicating the attributes of the target object.

[0065] Here, when multiple target objects are included in the image serving as training data as described above, the evaluation value calculation unit F2 calculates an intensity evaluation value for each target object. Then, the estimation unit F3 estimates attributes of training data that are insufficient and attributes of training data that are considered excessive in training the AI ​​model based on the intensity evaluation value for each target object calculated by the evaluation value calculation unit F2 and annotation information associated with the training data. The specific method for estimating attributes of excess and insufficient data (in this example, a method that also uses diversity and affinity) is as described above, and a duplicate description will be avoided.

[0066] 4, the first presentation processing unit F4 performs a process of presenting information indicating the attribute estimated by the estimation unit F3 to the user. Specifically, the first presentation processing unit F4 performs a process of displaying the information indicating the attribute estimated by the estimation unit F3 on the display unit (display unit 17) of the user terminal 3.

[0067] 5 is a diagram showing an example of a first presentation screen presented to a user by the first presentation processing unit F4, which shows an example of a screen corresponding to a case where attributes of excess or deficiency data are estimated for an AI model that performs object detection processing with a person as the target object.

[0068] As shown in the figure, the first presentation screen may display example images of scenes where detection was not possible and text information indicating the reasons for the failure. The images of scenes where detection was not possible are training data in which the AI ​​model was unable to detect a target object, and are selected from the training data of attributes that the estimation unit F3 has estimated to be lacking in training.

[0069] The figure gives examples of reasons for failure to detect the object, such as "insufficient training data for the camera facing forward at a 30-degree depression angle," "insufficient training data for a white shirt," "insufficient training data for a man in his 20s," and "insufficient training data for the back view," but these are merely examples.

[0070] In this example, in response to the estimation of attributes by the estimation unit F3, the first presentation processing unit F4 controls so that the above-described first presentation screen is displayed on the display unit of the user terminal 3. For example, in response to the completion of the estimation process by the estimation unit F3, the first presentation processing unit F4 performs a process of notifying the user terminal 3 of the URI (Uniform Resource Identifier) ​​of a web page that will present information as the first presentation screen. The user can access the corresponding web page using the URI and be presented with the first presentation screen via the display unit of the user terminal 3.

[0071] By presenting the first presentation screen as described above, the user can understand the attributes of the missing data (which may include excess data) in learning. This allows the user to perform case studies on what attributes of learning data are missing and what attributes are excess in AI model learning, which can help increase the user's knowledge about adjusting learning data when learning an AI model.

[0072] 4, the second presentation processing unit F5 performs a process of presenting a selection screen that allows the user to select whether to obtain data based on the attributes estimated by the estimation unit F3 through an automatic generation process by the data generation unit F20 or by sensing. In this example, since the learning data is image data, the selection screen allows the user to select whether to obtain data based on the estimated attributes through an automatic generation process or by capturing an image (see FIG. 6).

[0073] When "obtaining by imaging" is selected on the selection screen, the second presentation processing unit F5 performs processing to present to the user an imaging condition instruction screen as shown in FIG. 7. As shown, the imaging condition instruction screen displays information indicating the conditions of the scene to be added and information indicating the data conditions. As the information indicating the conditions of the scene to be added, it is conceivable to display information indicating the lighting conditions, camera conditions, and conditions of the detection target, as shown in the figure. Furthermore, as the information indicating the data conditions, it is conceivable to display information indicating the resolution, format, and required number of captured images.

[0074] The information indicating the conditions for the scene to be added shown in the figure is merely an example for explanatory purposes, and the information indicating the conditions for the scene to be added will be information according to the attributes estimated by the estimation unit F3. As for the information indicating the data conditions, it is possible to display predetermined information.

[0075] Furthermore, when automatic generation is selected on the selection screen shown in Figure 6, the CPU 11 notifies the data generation unit F20 in the data generation device 2 of information on attributes that were estimated to be missing in the estimation process of the estimation unit F3, and generates learning data as the missing data.

[0076] When the learning data is automatically generated by the data generation unit F20 in this way, the data adjustment unit F6 shown in FIG. 4 performs a process of adding the data generated by the data generation unit F20 to the learning data to be used in the re-learning process. Furthermore, when the estimation unit F3 estimates an attribute of the learning data to be excessive, the data adjustment unit F6 performs a process of deleting the learning data with the corresponding attribute from the learning data to be used in the re-learning process (excluding it from use).

[0077] Furthermore, when imaging is selected on the selection screen and the user images the learning data, the data adjustment unit F6 performs a process of adding the captured learning data as learning data to be used in the re-learning process. In this case, it is considered that the association of annotation information with the learning data obtained by imaging is performed manually by a user or the like. Alternatively, it is considered that the association of annotation information with the learning data obtained by imaging is performed automatically rather than manually by preparing an AI model that infers annotation information in the information processing device 1, inferring annotation information for each learning data using the AI ​​model, and performing a process of associating the inferred annotation information with each learning data.

[0078] 6 and the imaging condition instruction screen shown in Fig. 7 may also be displayed to the user on a web page, for example, and in this case, the second presentation processing unit F5 notifies the user terminal 3 of a URI indicating a link to the web page of the selection screen or the imaging condition instruction screen. At this time, in transitioning from the selection screen to the imaging condition instruction screen, it may be possible to configure the operation of the button for selecting imaging on the selection screen to be a link operation to the display page of the imaging condition instruction screen.

[0079] (1-4. Processing Procedure) A specific example of a processing procedure for realizing the learning method according to the first embodiment described above will be described with reference to the flowchart in Fig. 8. In this example, the processing shown in Fig. 8 is executed by the CPU 11 of the information processing device 1 based on a program stored in a predetermined storage device such as the ROM 12 or the storage unit 19.

[0080] In step S101, the CPU 11 performs an AI model learning process. In this embodiment, it is assumed that adjustment of learning data based on the intensity evaluation value, etc. is performed on learning data during relearning, and it is assumed that initial learning using clean data has already been performed on the AI ​​model. Therefore, the AI ​​model learning process in step S101 is performed on an AI model that has already undergone initial learning. Specifically, the learning process of the initially trained AI model is performed using learning data as clean data and learning data as augmented data obtained by data generation as DA using the data generation unit F20.

[0081] In step S102 following step S101, the CPU 11 performs a performance evaluation process. That is, a process of calculating an evaluation value of the inference performance of the AI ​​model that underwent the learning process in step S101. Examples of the evaluation value of the inference performance (hereinafter referred to as the "performance evaluation value") include the accuracy rate (the proportion of correct predictions to all predictions), precision (the proportion of correct predictions by the model that were actually correct), recall (the proportion of correct predictions by the model that were actually correct), and F-measure. However, the specific type of evaluation value is not particularly limited as long as it can evaluate the inference performance of the trained AI model. The performance evaluation value is calculated based on the inference results when verification data is provided as input data to the trained AI model.

[0082] In step S103 following step S102, the CPU 11 determines whether the performance evaluation value is equal to or greater than a threshold value, that is, whether the inference performance is equal to or greater than a certain standard.

[0083] In step S103, if the performance evaluation value is equal to or greater than the threshold value, the CPU 11 ends the series of processes shown in FIG.

[0084] On the other hand, if the performance evaluation value is not equal to or greater than the threshold in step S103, the CPU 11 proceeds to step S104, where it performs a process of estimating the attributes of the missing data and the excess data based on the annotation information, the degree of randomness evaluation value, diversity, and affinity. Here, in this example, it is assumed that the degree of randomness evaluation value, diversity, and affinity are calculated for each piece of training data as augmented data.

[0085] 9 is a flowchart illustrating the detailed processing of step S104. As shown in the figure, the processing of step S104 in this example is made up of processing of steps S201 to S206.

[0086] In step S201, the CPU 11 calculates a violence evaluation value for each pixel. Specifically, in this example, saliency is calculated for each pixel. In step S202 following step S201, the CPU 11 calculates a violence evaluation value for each bounding box. That is, for each bounding box indicated by the annotation information associated with the target learning data, the violence evaluation values ​​calculated in step S201 are aggregated to calculate a violence evaluation value for each bounding box.

[0087] In step S203, the CPU 11 searches for attributes of bounding boxes whose intensity evaluation values ​​are equal to or greater than the threshold, and for attributes whose diversity and affinity are equal to or less than the threshold. Then, in step S204, the CPU 11 determines the corresponding attributes as attributes of missing data.

[0088] In step S205 following step S204, the CPU 11 searches for an attribute whose affinity is equal to or greater than the threshold among the attributes of the bounding box whose intensity evaluation value is equal to or less than the threshold, and in step S206 determines the corresponding attribute as an attribute of excess data.

[0089] In response to execution of the process of step S104 shown in Fig. 9, the CPU 11 proceeds to the process of step S105 shown in Fig. 8. In step S105, the CPU 11 performs a first presentation process. This is a process performed by the first presentation processing unit F4 described above, and is a process of presenting information indicating the attributes estimated in the process of step S104 to the user. A specific example of this process has already been described, so a duplicate description will be avoided.

[0090] In step S106 following step S105, the CPU 11 performs a second presentation process. This is a process by the second presentation processing unit F5 described above, and involves presenting a selection screen that allows the user to select whether data based on the attributes estimated in the process of step S104 is obtained by automatic generation or by imaging. A specific example of this second presentation process has already been described, so a duplicate description will be avoided.

[0091] In step S107 following step S106, the CPU 11 determines whether or not to automatically generate the missing data. Specifically, in this example, it determines whether or not an operation to select automatic generation has been performed on the selection screen.

[0092] If it is determined in step S107 that the missing data is to be automatically generated, the CPU 11 proceeds to step S108 and issues an instruction to generate the missing data. That is, the CPU 11 instructs the data generating device 2 to generate learning data as the missing data by instructing the data generating device 2 to use information indicating the attributes of the missing data estimated in the processing of step S104. Then, in the following step S109, the CPU 11 performs a process of receiving the missing data generated by the data generating device 2 as a process of receiving the missing data.

[0093] On the other hand, if it is determined in step S107 that the missing data will not be automatically generated (i.e., in this example, if an operation to obtain the missing data by capturing an image has been performed), the CPU 11 proceeds to step S110 and performs a process of accepting the missing data. That is, this is a process of accepting the missing data captured by the user. The captured missing data may be uploaded to the information processing device 1 from the user terminal 3 via the network NT, for example. In this case, the acceptance process in step S110 may involve presenting an upload screen to the user, or the like.

[0094] 7, the process for displaying the imaging condition instruction screen may be executed after a negative result is obtained in step S107 and before the acceptance process in step S110. Furthermore, the association of annotation information with the captured missing data may be performed as exemplified above.

[0095] In response to execution of either step S109 or S110, the CPU 11 proceeds to step S111 to perform adjustment processing of the learning data, i.e., processing to add the generated or captured missing data to the learning data for re-learning, and processing to delete the excess data from the learning data for re-learning.

[0096] In step S112 following step S111, the CPU 11 performs an AI model learning process using the adjusted learning data. Then, upon completion of the learning process in step S112, the CPU 11 returns to step S102. This causes a series of processes, from the calculation of the intensity evaluation value to the learning process using the adjusted learning data, to be performed with the termination condition being that the performance evaluation value of the AI ​​model is equal to or greater than a threshold. Since the learning data is adjusted each time based on attribute information of the missing data or excess data estimated based on an appropriate evaluation value, it is possible to improve learning efficiency and learning accuracy.

[0097] In the above example, the condition for terminating learning is set to be that the performance evaluation value of the AI ​​model is equal to or greater than a threshold value. However, the condition for terminating learning can also be set to a condition other than that based on the performance evaluation value, such as a condition based on the number of times learning has been performed.

[0098] While the above describes an example in which an evaluation value other than the violence evaluation value is used to estimate the attributes of missing data and excess data, the estimation can also be performed using only the violence evaluation value. An example of this will be described with reference to FIG. 10 . FIGS. 10A and 10B are explanatory diagrams of an example of estimating the attributes of missing data, and FIG. 10C is an explanatory diagram of an example of estimating the attributes of excess data. For example, as shown in FIG. 10A , a violence evaluation value is calculated for each age attribute, and the attribute of an age with a higher violence evaluation value than other ages is estimated as the attribute of the missing data. The example in the figure shows a case in which the violence evaluation value for teenagers is higher than for other age groups, and in this case, it can be estimated that data for teenagers is lacking in learning.

[0099] As shown in FIG. 10B, it is also possible to estimate, as the missing data attribute value, an attribute value of the target object such as age that is far from the statistical average and has a larger violence evaluation value than other attribute values.

[0100] Furthermore, for the attributes of excess data, as illustrated in Figure 10C, it is possible to calculate an intensity evaluation value for each attribute value as a camera angle, and estimate the attribute value with a smaller intensity evaluation value than other attribute values ​​as the attribute value of excess data.

[0101] In addition, although the above example shows the calculation of the intensity evaluation value, diversity, and affinity for each piece of learning data input to the AI ​​model during relearning, and the estimation of attributes based on these calculated evaluation values, it is also possible to group multiple pieces of learning data with the same or similar attributes and calculate the intensity evaluation value, diversity, and affinity for each group. This can improve the efficiency of the process of estimating the attributes of excess or insufficient data.

[0102] Furthermore, the attribute estimation method described above is merely an example for illustrative purposes, and estimation using more complex methods is also possible. In such cases, the attributes of the missing data and the excess data may be estimated by multivariate analysis using the intensity evaluation value and attribute information indicated by the annotation information associated with the training data.

[0103] In the above example, the generation of the learning data to be added is performed in a data generating device 2 separate from the information processing device that calculates the violence evaluation value and estimates the attributes. However, the generation of the learning data may also be performed in the information processing device. In that case, it may be configured such that the CPU 11 functions as a data generating unit F20, as in the information processing device 1A illustrated in FIG.

[0104] Furthermore, in the above example, an information processing device that calculates the violence evaluation value and estimates the attributes performs the learning process of the AI ​​model. However, it is also possible to adopt a configuration in which the learning process is performed by a device separate from the information processing device. FIG. 12 illustrates an example of the configuration of a learning system in this case. As shown in the figure, in this learning system, a learning device 5 having a learning processing unit F1 is added. Furthermore, an information processing device 1B having a configuration in which the learning processing unit F1 is omitted from the information processing device 1 is used as the information processing device that calculates the violence evaluation value and estimates the attributes. Note that the information processing device 1B can also be configured to include a data generation unit F20, similar to the information processing device 1A shown in FIG. 11.

[0105] 2. Second Embodiment Next, a second embodiment will be described. In the second embodiment, when CG images are used as learning data, the image quality of the CG images to be input to the AI ​​model is adjusted to improve the learning accuracy.

[0106] Fig. 13 is a functional block diagram showing an example of the functional configuration of an information processing device 1C according to the second embodiment. The hardware configuration of the information processing device 1C is similar to that shown in Fig. 3, and therefore a duplicated description will be avoided. Note that while the example shown here assumes that the data generating device 2 has the data generating unit F20, as in the example of Fig. 1, the second embodiment may also adopt a configuration in which the information processing device 1C has the data generating unit F20, as in the example of Fig. 11.

[0107] As shown in the figure, the CPU 11 in the information processing device 1C has a learning processing unit F1, an evaluation value calculation unit F2, an estimation unit F3, a first presentation processing unit F4, a second presentation processing unit F5, a data adjustment unit F6, and an image signal processing unit F8. In addition, the CPU 11 in this case has a learning control unit F7C instead of the learning control unit F7.

[0108] The image signal processing unit F8 adjusts the image quality of the learning data using CG images. In this embodiment, the image signal processing unit F8 is configured to adjust the image quality using AI.

[0109] In this case, when CG images are used as learning data, the learning processing unit F1 inputs the learning data whose image quality has been adjusted by the image signal processing unit F8 to the learning device F11 and performs learning processing on the AI ​​model.

[0110] 14 is a conceptual diagram of learning when CG images are used as learning data in the second embodiment. As shown in the figure, the learning data based on CG images is input to an image signal processing unit F8, where it is subjected to image quality adjustment, and then provided as input data to a learning device F11.

[0111] Here, since the image signal processing unit F8 is configured with AI, in the second embodiment, learning is also performed on the image signal processing unit F8. The purpose of performing image quality adjustment by the image signal processing unit F8 is to improve inference performance when the finally obtained trained AI model is actually used in an inference environment. For this reason, in learning by the image signal processing unit F8, the loss value calculated by the learner F11 is used as a learning index.

[0112] In this case, performing the learning of the image signal processing unit F8 and the learning of the AI ​​model simultaneously in parallel involves difficulties in terms of the convergence of the learning, etc. For this reason, in this embodiment, the learning of the image signal processing unit F8 and the learning of the AI ​​model are performed alternately. In other words, the learning of the image signal processing unit F8 with the parameters of the AI ​​model fixed and the learning of the AI ​​model with the parameters of the image signal processing unit F8 fixed are performed alternately.

[0113] The learning control unit F7C shown in Fig. 13 performs control to realize such alternate learning. Specifically, the learning control unit F7C controls so that a learning process of the image signal processing unit F8 based on the loss value of the AI ​​model with the parameters of the AI ​​model fixed and a learning process of the AI ​​model with the parameters of the image signal processing unit F8 fixed are alternately performed.

[0114] Here, it is conceivable that the image signal processing unit F8 can be trained to reduce the loss value. For example, even in an inference environment, when the image signal processing unit F8 is provided in the preceding stage of the AI ​​model, it can be said that training the image signal processing unit F8 to reduce the loss value in this way is effective.

[0115] However, in this embodiment, it is assumed that the image signal processing unit F8 is not used in the inference environment, i.e., in the inference environment, a captured real image is input to the AI ​​model without undergoing image quality adjustment by the image signal processing unit F8.

[0116] In this way, when the image signal processing unit F8 is not used in an inference environment, there is a concern that training the image signal processing unit F8 to reduce the loss value will simply make learning easier and will not contribute to improving performance.

[0117] Therefore, in this embodiment, the image signal processing unit F8 learns so as to increase the loss value. Specifically, in learning by the image signal processing unit F8, the loss value calculated by the learning unit F11 is not used as is as an index for learning, but the inverted value of the loss value is used.

[0118] By training the image signal processing unit F8 to increase the loss value as described above, the AI ​​model is trained by adjusting the CG image to make it more difficult for the AI ​​model to perform inference processing. This makes it possible to create a highly versatile AI model that can handle a wider variety of images, thereby improving the inference performance of the AI ​​model.

[0119] 15 is a flowchart showing an example of a processing procedure for realizing the learning method according to the second embodiment described above. In this example, the processing shown in FIG. 15 is executed by the CPU 11 of the information processing device 1C based on a program stored in a predetermined storage device such as the ROM 12 or the storage unit 19.

[0120] First, in step S301, the CPU 11 executes a learning process for the image signal processing unit F8 with the parameters of the initially trained AI model fixed. As described above, a loss value is used for learning the image signal processing unit F8. Therefore, in the process of step S301, a CG image after image quality adjustment by the image signal processing unit F8 is provided as input data to the initially trained AI model with fixed parameters, and the loss value is calculated using the function of the learning device F11. Also, as described above, learning of the image signal processing unit F8 is performed so as to increase the loss value.

[0121] In step S302 following step S301, the CPU 11 executes a learning process for the AI ​​model while fixing the parameters of the image signal processing unit F8.

[0122] In step S303 following step S302, the CPU 11 determines whether the learning termination condition is met. In this case as well, the learning termination condition may be based on the performance evaluation value of the AI ​​model, but the learning termination condition may also be based on a condition other than the performance evaluation value, such as the number of times learning is performed.

[0123] If it is determined in step S303 that the learning end condition is met, the CPU 11 ends the series of processes shown in FIG.

[0124] On the other hand, if it is determined that the learning termination condition is not satisfied, the CPU 11 proceeds to step S304, where it executes learning processing of the image signal processing unit F8 with the parameters of the AI ​​model fixed. That is, with the parameters of the AI ​​model learned in step S302 fixed, it executes learning processing of the image signal processing unit F8 based on the loss value of the AI ​​model (learning processing to increase the loss value).

[0125] After the learning process in step S304 is completed, the CPU 11 returns to step S302. This allows the process of alternately learning the image signal processing unit F8 and learning the AI ​​model to be repeated until a predetermined learning end condition is met.

[0126] In addition, in order to prevent extreme changes in image quality from occurring in the learning process of the image signal processing unit F8, it is also possible to add a correction to the loss value (in this example, the inverted value) input as a learning index according to the image difference before and after image quality adjustment.

[0127] Furthermore, although the above explanation is based on the assumption that the image signal processing unit F8 is not used in the inference environment, if the image signal processing unit F8 is used in the inference environment, the image signal processing unit F8 can be trained to reduce the loss value.

[0128] 3. Modifications Note that the embodiment is not limited to the specific examples described above, and various modified configurations may be employed. For example, although the above example illustrates the generation of a CG image as an example of automatic generation of additional data, automatic generation of additional data may also be performed using a generation AI. In this case, information on missing attributes is input as a prompt for LLMs (Large Language Models), and the generation AI generates the missing data. In this case, association of annotation information with the generated additional data may be performed manually by a user or the like, or may be performed as an automatic process using an AI model that infers annotation information.

[0129] Furthermore, although the above provides an example of applying the learning method according to the present technology when an AI model performs inference processing on image data, the present technology can be widely and suitably applied to the learning of AI models that perform inference processing on data other than images, such as sound data.

[0130] 4. Summary of the Embodiments As described above, the information processing device (1, 1A, 1B, 1C) according to the embodiments includes an evaluation value calculation unit (F2) that calculates a fluctuation evaluation value indicating the degree of fluctuation in the weight coefficients of a convolution filter when training data is input to the AI ​​model during training of the AI ​​model, and an estimation unit that estimates attributes of training data that are considered to be insufficient or excessive in training of the AI ​​model based on the fluctuation evaluation value calculated by the evaluation value calculation unit and annotation information of the training data. By using the evaluation value indicating the fluctuation degree of the weight coefficients of the convolution filter, in other words, the responsiveness of the convolution filter, it is possible to appropriately evaluate the correctness of the operation of the AI ​​model. Therefore, it is possible to improve the accuracy of estimating data that is insufficient or excessive for training in the training data used for AI model training, thereby improving the training accuracy of the AI ​​model.

[0131] Furthermore, the information processing device according to the embodiment includes a data adjustment unit (F6) that adjusts the training data by adding missing data identified from the attribute information estimated by the estimation unit or by removing excess data identified from the attribute information. This allows the AI ​​model to be retrained using a training data set in which the excess or deficiency has been adjusted. This improves the training accuracy of the AI ​​model and the inference performance of the AI ​​model.

[0132] Furthermore, in the information processing device according to the embodiment, the training data is image data, which can improve the training accuracy of an AI model that performs inference processing related to images, such as various object detection processing and object recognition processing, thereby improving the inference performance of the AI ​​model.

[0133] Furthermore, in an information processing device as an embodiment, image data as learning data is an image in which a plurality of target objects to be inferred are arranged, the annotation information includes attribute information for each target object, the evaluation value calculation unit calculates an intensity evaluation value for each target object, and the estimation unit estimates attributes of the target objects that are deemed to be insufficient or excessive in learning of the AI ​​model based on the intensity evaluation value for each target object and the annotation information. This makes it possible to improve the learning accuracy of the AI ​​model when the AI ​​model performs inference processing for a plurality of target objects in an image, and to improve the inference performance of the AI ​​model.

[0134] Furthermore, the information processing device according to the embodiment includes a first presentation processing unit (F4) that performs processing to present information indicating the attributes estimated by the estimation unit to the user. This allows the user to understand the data that is insufficient or excessive for learning. The user can perform case studies on what attributes of learning data are insufficient or excessive in AI model learning, which can help increase the user's knowledge about adjusting learning data during AI model learning. Furthermore, when the user uses a sensing device to prepare learning data for re-learning, such as when capturing images of the learning data for re-learning, the user can be shown what kind of learning data they should prepare.

[0135] Furthermore, the information processing device according to the embodiment includes a second presentation processing unit (F5) that presents a selection screen that allows the user to select whether to obtain data based on the attributes estimated by the estimation unit through automatic generation processing or through sensing, thereby improving the degree of freedom in the method for obtaining learning data for re-learning.

[0136] Furthermore, in the information processing device according to the embodiment, the evaluation value calculation unit calculates a value based on saliency as the fluctuation evaluation value. The saliency allows the degree of fluctuation of the weight coefficients of the convolution filter to be appropriately evaluated, improving the estimation accuracy of the estimation unit and thereby improving the learning accuracy of the AI ​​model.

[0137] Furthermore, in the information processing device according to the embodiment, the estimation unit uses an intensity evaluation value and an evaluation value as diversity to estimate attributes. By using the evaluation value as diversity together with an intensity evaluation value such as saliency, it becomes possible to estimate attributes of data that are insufficient for learning or that are considered excessive in terms of data diversity. Therefore, it becomes possible to adjust the learning data to be used for re-learning so that the distribution of the learning data is appropriate in terms of diversity, thereby improving the learning accuracy of the AI ​​model and, ultimately, improving the inference performance of the AI ​​model.

[0138] Furthermore, in the information processing device according to the embodiment, the estimation unit estimates that an attribute of the training data for which the violence evaluation value is equal to or greater than a threshold and the diversity value is equal to or less than a threshold is an attribute that is missing in the training. For example, assume that the AI ​​model performs object detection processing targeting automobile objects, and that the AI ​​model has insufficient object detection accuracy for automobiles, such as light cars and red cars. In this case, if the training data to be evaluated includes light cars and red cars as subjects, the violence evaluation value will be large and the diversity value will be small. Therefore, by setting the estimation condition that the violence evaluation value is equal to or greater than a threshold and the diversity value is equal to or less than a threshold, as described above, it is possible to appropriately estimate data that is missing in the training.

[0139] Furthermore, in the information processing device according to the embodiment, the estimation unit uses an intensity evaluation value and an evaluation value for affinity to estimate attributes. By using the evaluation value for affinity together with an intensity evaluation value such as saliency, it becomes possible to estimate attributes of data that are insufficient for learning or that are considered excessive in terms of data affinity. Therefore, it becomes possible to adjust the learning data to be used for re-learning so that the distribution of the learning data is appropriate in terms of affinity, thereby improving the learning accuracy of the AI ​​model and, ultimately, the inference performance of the AI ​​model.

[0140] Furthermore, in the information processing device according to the embodiment, the estimation unit estimates that attributes of the training data for which the violence evaluation value is equal to or greater than a threshold and the affinity value is equal to or less than a threshold are attributes that are lacking in the training. For example, assume that the AI ​​model performs object detection processing targeting automobiles, and that the object detection accuracy in a particular scene involving a red car is insufficient, and that it is desired to increase the scene variety of the training data for the red car. In this case, if the training data to be evaluated includes a red car as a subject, the violence evaluation value will be large and the affinity value will be small. Therefore, by setting the estimation condition that the violence evaluation value is equal to or greater than a threshold and the affinity value is equal to or less than a threshold, as described above, it is possible to appropriately estimate the data that is lacking in the training (it is possible to estimate that more training data for other scenes involving red cars should be added).

[0141] Furthermore, in an information processing device (1C) according to an embodiment, the training data is CG image data, and the device includes an image signal processing unit (F8) that adjusts the image quality of the training data, and a learning processing unit (F1) that has an AI model learning device and inputs the training data image quality adjusted by the image signal processing unit to the learning device to perform AI model training. This makes it possible to adjust the image quality of the CG image so that AI model training suited to the actual inference environment is realized. Therefore, the inference performance of the AI ​​model can be improved.

[0142] Furthermore, in the information processing device according to the embodiment, the image signal processing unit is provided with a learning control unit (F7C) that performs image quality adjustment using AI and controls the image signal processing unit to alternate between learning processing of the image signal processing unit based on the loss value of the AI ​​model with the parameters of the AI ​​model fixed and learning processing of the AI ​​model with the parameters of the image signal processing unit fixed. This makes it possible to use CG images for learning the AI ​​model, while creating an AI model that is suitable for inputting real-life images in an actual inference environment.

[0143] Furthermore, in the information processing device according to the embodiment, the learning process of the image signal processing unit is performed so as to increase the loss value. By performing learning of the image signal processing unit so as to increase the loss value, it becomes possible to create an appropriate AI model in response to a case where the image signal processing unit is not used in the inference environment. Therefore, the inference performance of the AI ​​model can be improved.

[0144] An information processing method as an embodiment is an information processing method in which an information processing device performs an evaluation value calculation process that calculates an intensity evaluation value that indicates the degree of intensity of the weight coefficient of a convolution filter when learning data is input to an AI model during learning of the AI ​​model, and an estimation process that estimates attributes of learning data that are considered to be insufficient or excessive in learning of the AI ​​model based on the intensity evaluation value calculated in the evaluation value calculation process and annotation information of the learning data. With such an information processing method, it is possible to obtain the same functions and effects as the information processing device as the above-mentioned embodiment.

[0145] A learning system according to an embodiment includes an evaluation value calculation unit that calculates a fluctuation evaluation value indicating the degree of fluctuation in the weight coefficients of a convolution filter when training data is input to the AI ​​model during training of the AI ​​model; an estimation unit that estimates attributes of the training data that are deemed insufficient or excessive in training of the AI ​​model based on the fluctuation evaluation value calculated by the evaluation value calculation unit and annotation information of the training data; a data adjustment unit that adjusts the training data by at least one of adding missing data identified from the attribute information estimated by the estimation unit and removing excessive data identified from the attribute information; and a learning processing unit that performs training processing of the AI ​​model using the training data adjusted by the data adjustment unit. Similar to the information processing device according to the embodiment described above, this learning system also enables appropriate evaluation of the correctness of the operation of the AI ​​model, improves the accuracy of estimation of data that is insufficient or excessive for training, and enables appropriate adjustment of the training data. Therefore, the learning accuracy of the AI ​​model can be improved.

[0146] Furthermore, the learning system according to the embodiment includes a learning control unit that controls a series of processes, from the calculation of the intensity evaluation value by the evaluation value calculation unit to the learning process using the adjusted learning data by the learning processing unit, to be performed until a predetermined termination condition is met. This makes it possible to repeat the series of processes, from the calculation of the intensity evaluation value to the learning process using the adjusted learning data, until a predetermined learning termination condition is met, such as the performance evaluation value of the AI ​​model being equal to or greater than a threshold. Since the learning data is adjusted each time based on attribute information of missing data or excess data estimated based on an appropriate evaluation value, it is possible to improve learning efficiency and learning accuracy.

[0147] Here, as an embodiment, a program that causes, for example, a CPU, a DSP, or a device including these, to realize the processes described with reference to Figures 8, 9, 15, etc., can be considered. That is, the program of the embodiment is a program readable by a computer device, and causes the computer device to realize a function of calculating a fluctuation evaluation value indicating the degree of fluctuation in the weight coefficient of a convolution filter when training data is input to an AI model during training of the AI ​​model, and estimating attributes of training data that are considered to be insufficient or excessive in training of the AI ​​model based on the calculated fluctuation evaluation value and annotation information of the training data. Such a program allows the functions of the CPU 11 of the above-described embodiment to be realized in a computer device such as the information processing device 1, 1A, 1B, 1C, etc.

[0148] The above-described programs can be pre-recorded on a hard disk drive (HDD) or solid state drive (SSD) as a recording medium built into a computer or other device, or on a ROM within a microcomputer having a CPU. Alternatively, the programs can be temporarily or permanently stored (recorded) on a removable recording medium such as a flexible disk, a CD-ROM (Compact Disc Read Only Memory), a Magneto Optical (MO) disc, a Digital Versatile Disc (DVD), a Blu-ray Disc (Blu-ray Disc (registered trademark)), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a LAN or the Internet.

[0149] Furthermore, such a program is suitable for providing a wide range of learning methods according to embodiments, and can cause various types of information processing devices to function as devices that implement the learning methods of the present disclosure.

[0150] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0151] <5. The Present Technology> The present technology may also have the following configuration. (1) An information processing device comprising: an evaluation value calculation unit that calculates a degree of fluctuation in a weight coefficient of a convolution filter when training data is input to an AI model during training of the AI ​​model; and an estimation unit that estimates attributes of the training data that are deemed to be insufficient or excessive in training of the AI ​​model based on the degree of fluctuation calculated by the evaluation value calculation unit and annotation information of the training data. (2) The information processing device described in (1) above, including a data adjustment unit that adjusts the training data by at least one of adding insufficient data identified from information on the attributes estimated by the estimation unit and removing excessive data identified from the information on the attributes. (3) The information processing device described in (1) or (2) above, wherein the training data is image data. (4) The information processing device according to (3), wherein the image data as the learning data is an image in which a plurality of target objects to be inferred are arranged, the annotation information includes attribute information for each of the target objects, the evaluation value calculation unit calculates the violence evaluation value for each of the target objects, and the estimation unit estimates attributes of the target objects that are deemed to be insufficient or excessive in learning the AI ​​model based on the violence evaluation value for each of the target objects and the annotation information. (5) The information processing device according to any of (1) to (4), further comprising a first presentation processing unit that performs processing to present to a user information indicating the attributes estimated by the estimation unit. (6) The information processing device according to any of (1) to (5), further comprising a second presentation processing unit that performs processing to present a selection screen that allows a user to select whether data based on the attributes estimated by the estimation unit is obtained by automatic generation processing or by sensing. (7) The information processing device according to any of (1) to (6), wherein the evaluation value calculation unit calculates a value based on saliency as the violence evaluation value. (8) The information processing device according to any one of (1) to (7), wherein the estimation unit uses the violence evaluation value and an evaluation value as diversity to estimate the attribute.(9) The information processing device according to (8), wherein the estimation unit estimates, as an attribute lacking in learning, an attribute of the learning data for which the violence evaluation value is equal to or greater than a threshold and the Diversity value is equal to or less than a threshold. (10) The information processing device according to any of (1) to (9), wherein the estimation unit uses the violence evaluation value and an evaluation value as Affinity to estimate the attribute. (11) The information processing device according to (10), wherein the estimation unit estimates, as an attribute lacking in learning, an attribute of the learning data for which the violence evaluation value is equal to or greater than a threshold and the Affinity value is equal to or less than a threshold. (12) The information processing device according to any of (1) to (11), wherein the learning data is data based on CG images, and wherein the information processing device comprises: an image signal processing unit that adjusts image quality of the learning data; and a learning processing unit that has a learning device for the AI ​​model, and inputs the learning data image quality adjusted by the image signal processing unit to the learning device and performs learning processing for the AI ​​model. (13) The information processing device according to (12), wherein the image signal processing unit performs the image quality adjustment using AI, and includes a learning control unit that controls alternating learning processing of the image signal processing unit based on a loss value of the AI ​​model with parameters of the AI ​​model fixed, and learning processing of the AI ​​model with parameters of the image signal processing unit fixed. (14) The information processing device according to (13), wherein the learning processing of the image signal processing unit is performed so as to increase the loss value. (15) An information processing method, wherein an information processing device performs: an evaluation value calculation process that calculates a fluctuation evaluation value that indicates a degree of fluctuation in a weight coefficient of a convolution filter when learning data is input to the AI ​​model during learning of the AI ​​model; and an estimation process that estimates attributes of the learning data that are considered to be insufficient or excessive in learning of the AI ​​model, based on the fluctuation evaluation value calculated in the evaluation value calculation process and annotation information of the learning data.(16) A learning system comprising: an evaluation value calculation unit that calculates a degree of fluctuation in a weight coefficient of a convolution filter when learning data is input to an AI model during learning of the AI ​​model, an estimation unit that estimates attributes of the learning data that are considered to be insufficient or excessive in learning of the AI ​​model based on the degree of fluctuation evaluation value calculated by the evaluation value calculation unit and annotation information of the learning data, a data adjustment unit that adjusts the learning data by at least one of adding missing data identified from the information of the attribute estimated by the estimation unit and removing excessive data identified from the information of the attribute, and a learning processing unit that performs learning processing of the AI ​​model using the learning data adjusted by the data adjustment unit. (17) The learning system according to (16), comprising a learning control unit that controls a series of processes from the calculation of the degree of fluctuation evaluation value by the evaluation value calculation unit to the learning processing using the adjusted learning data by the learning processing unit so that they are performed until a predetermined termination condition is satisfied.

[0152] 1, 1A, 1B, 1C Information processing device 2 Data generation device 3 User terminal NT Network Pv Viewpoint 11 CPU 12 ROM 13 RAM 14 Bus 15 Input / output interface 16 Input unit 17 Display unit 18 Audio output unit 19 Storage unit 20 Communication unit 21 Drive 22 Removable recording medium F1 Learning processing unit F11 Learning device F2 Evaluation value calculation unit F3 Estimation unit F4 First presentation processing unit F5 Second presentation processing unit F6 Data adjustment unit F7, F7C Learning control unit F20 Data generation unit 5 Learning device F8 Image signal processing unit

Claims

1. An information processing device comprising: an evaluation value calculation unit that calculates a degree of fluctuation in the weight coefficients of a convolution filter when training data is input to an AI model during training of the AI ​​model; and an estimation unit that estimates attributes of the training data that are considered to be insufficient or excessive in training of the AI ​​model based on the degree of fluctuation in the training data calculated by the evaluation value calculation unit and annotation information of the training data.

2. The information processing device according to claim 1, further comprising a data adjustment unit that adjusts the learning data by at least one of adding missing data identified from the attribute information estimated by the estimation unit and removing excess data identified from the attribute information.

3. The information processing device according to claim 1, wherein the learning data is image data.

4. The information processing device according to claim 3, wherein the image data as learning data is an image in which a plurality of target objects to be inferred are arranged, the annotation information includes attribute information for each of the target objects, the evaluation value calculation unit calculates the violence evaluation value for each of the target objects, and the estimation unit estimates attributes of the target objects that are deemed to be insufficient or excessive in learning the AI ​​model based on the violence evaluation value for each of the target objects and the annotation information.

5. The information processing device according to claim 1, further comprising a first presentation processing unit that performs processing to present information indicating the attribute estimated by the estimation unit to a user.

6. The information processing device according to claim 1, further comprising a second presentation processing unit that performs a process of presenting a selection screen that allows a user to select whether data based on the attributes estimated by the estimation unit is obtained by automatic generation processing or by sensing.

7. The information processing device according to claim 1, wherein the evaluation value calculation unit calculates a value based on saliency as the intensity evaluation value.

8. The information processing device according to claim 1, wherein the estimation unit uses the violence evaluation value and the diversity evaluation value to estimate the attribute.

9. The information processing device according to claim 8, wherein the estimation unit estimates that an attribute of the training data for which the intensity evaluation value is equal to or greater than a threshold and the diversity value is equal to or less than a threshold is an attribute that is lacking in training.

10. The information processing device according to claim 1, wherein the estimation unit uses the violence evaluation value and an evaluation value as affinity to estimate the attribute.

11. The information processing device according to claim 10, wherein the estimation unit estimates that an attribute of the learning data for which the intensity evaluation value is equal to or greater than a threshold and the affinity value is equal to or less than a threshold is an attribute that is lacking in learning.

12. An information processing device according to claim 1, wherein the learning data is CG image data, and the information processing device comprises: an image signal processing unit that adjusts the image quality of the learning data; and a learning processing unit that has a learning device for the AI ​​model, and inputs the learning data whose image quality has been adjusted by the image signal processing unit into the learning device to perform learning processing for the AI ​​model.

13. The information processing device according to claim 12, wherein the image signal processing unit performs the image quality adjustment using AI, and includes a learning control unit that controls the image signal processing unit to alternate between a learning process of the image signal processing unit based on a loss value of the AI ​​model with the parameters of the AI ​​model fixed and a learning process of the AI ​​model with the parameters of the image signal processing unit fixed.

14. The information processing device according to claim 13, wherein the learning process of the image signal processing unit is performed so as to increase the loss value.

15. An information processing method in which an information processing device performs: an evaluation value calculation process that calculates an intensity evaluation value that indicates the degree of intensity fluctuation of a weight coefficient of a convolution filter when training data is input to an AI model during training of the AI ​​model; and an estimation process that estimates attributes of the training data that are considered to be insufficient or excessive in training of the AI ​​model based on the intensity evaluation value calculated in the evaluation value calculation process and annotation information of the training data.

16. A learning system comprising: an evaluation value calculation unit that calculates a fluctuation evaluation value indicating the degree of fluctuation in the weight coefficient of a convolution filter when learning data is input to an AI model during learning of the AI ​​model; an estimation unit that estimates attributes of the learning data that are deemed to be insufficient or excessive in learning of the AI ​​model based on the fluctuation evaluation value calculated by the evaluation value calculation unit and annotation information of the learning data; a data adjustment unit that adjusts the learning data by at least one of adding missing data identified from information on the attributes estimated by the estimation unit and removing excessive data identified from information on the attributes; and a learning processing unit that performs learning processing of the AI ​​model using the learning data adjusted by the data adjustment unit.

17. A learning system as described in claim 16, comprising a learning control unit that controls a series of processes from the calculation of the intensity evaluation value by the evaluation value calculation unit to the learning process using the adjusted learning data by the learning processing unit so that they are carried out until a predetermined termination condition is met.

Images