Information processing device, learning method, and learning program

Abstract

This technology is suitable for generating AI models that produce output tailored to the user's preferences. [Solution] An exemplary information processing device is an information processing device that retrains an AI model trained with first training data, and includes a controller. The controller performs training on the AI ​​model using the first training data and second training data which is training data based on feedback information for the AI ​​model. After the training, it performs a judgment process using a first target value which is a target value for the accuracy rate of the first training data, and a judgment process using a second target value which is a target value for the accuracy rate of the second training data, respectively. Based on the result of each judgment process, it determines whether additional training is necessary and generates a trained AI model through retraining.

Description

Technical Field

[0001] The present invention relates to a technique for relearning a learned AI (Artificial Intelligence) model.

Background Art

[0002] Conventionally, a notification system that gives notifications to a driver (user) for purposes such as driving support has been installed in a vehicle (see, for example, Patent Document 1). As disclosed in Patent Document 1, the means of information notification in a vehicle are diverse. For example, display, sound, and vibration are used.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In a vehicle, information notification has diverse notification means (display, sound, vibration, etc.) as described above, and the types of information to be notified are also diverse. For this reason, the control of information notification is not easy, and it is even more difficult to perform notification control that suits the preferences of users such as drivers. In view of such circumstances, it is conceivable to apply an AI model to the control of information notification.

[0005] When applying an AI model to a notification system installed in a vehicle, it is assumed that the AI model is initially a learned model learned using initially prepared learning data. Then, when a user (such as a driver) starts using the AI model, feedback data from the user is obtained, and relearning based on the feedback data is performed, so that the AI model is assumed to approach an AI model that outputs according to the preferences of the user.

[0006] However, a large amount of feedback data is required to get an AI model to produce output that matches the user's preferences. Furthermore, even if retraining is performed using feedback data, the AI ​​model may not produce output that is desirable to the user due to reasons such as overfitting.

[0007] In view of the above, the present invention aims to provide a technology suitable for generating AI models that produce output tailored to the user's preferences. [Means for solving the problem]

[0008] An exemplary information processing device of the present invention is an information processing device that retrains an AI model trained with first training data, and comprises a controller. The controller performs training on the AI ​​model using the first training data and second training data which is training data based on feedback information for the AI ​​model. After the training, the controller performs a determination process using a first target value which is a target value for the accuracy rate of the first training data, and a determination process using a second target value which is a target value for the accuracy rate of the second training data, respectively. The controller determines whether additional training is necessary according to the result of each determination process and generates a trained AI model through retraining. [Effects of the Invention]

[0009] In this exemplary invention, an AI model trained with first training data (a typical example being initial training data) is trained (retrained) using the first training data and second training data, which is training data based on feedback information for the AI ​​model. The AI ​​model trained through retraining is generated by separately determining target accuracy values ​​for the first training data and target accuracy values ​​for the second training data, and determining whether additional training is necessary based on the degree of achievement of each target value. In this configuration, by adjusting the target accuracy values ​​for the first and second training data, it is possible to adjust whether the AI ​​model trained through retraining is a model that emphasizes the reflection of user feedback information or a model that emphasizes the suppression of overfitting. In other words, by appropriately adjusting the target accuracy values ​​for the first and second training data, it is possible to generate an AI model that appropriately reflects feedback information and suppresses overfitting. Since the feedback information reflects the user's preferences, according to this exemplary invention, it is possible to generate an AI model that produces output that matches the user's preferences. [Brief explanation of the drawing]

[0010] [Figure 1] Diagram showing the general configuration of the driver assistance system. [Figure 2] Block diagram showing the configuration of the functional unit of the controller of the information notification control device. [Figure 3] Schematic diagram to explain the HMI control model. [Figure 4] Block diagram showing the general configuration of the information processing device. [Figure 5] A flowchart illustrating the flow of the training method for a pre-trained AI model using the first training data, as executed by an information processing device. [Figure 6] A schematic diagram showing the data changes during the execution of the process shown in Figure 5. [Figure 7] A diagram showing an example of data with correct labels. [Modes for carrying out the invention]

[0011] Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts will be denoted by the same reference numerals and will not be repeated in the description.

[0012] <1. Driver assistance systems> [1-1. Overview] Figure 1 shows a schematic configuration of a driver assistance system 100 according to an embodiment of the present invention. The driver assistance system 100 is a system that assists in the driving of a vehicle VE. The driver assistance system 100 can also be described as an information notification system that notifies the driver of the vehicle VE of information related to driving. The information notified may include not only information related to driving, but also entertainment information such as music and movies, and in such a configuration, the system indicated by reference numeral 100 may be understood as an information provision system. When configured as an information provision system, the provision (notification) of information related to driving is not essential, and the information provision system is not limited to vehicles, but can be applied to various uses such as homes, factories, or offices.

[0013] As shown in Figure 1, the driver assistance system 100 comprises an information notification control device 1, an information source device 2, an information notification device 3, an FB (feedback) information acquisition device 4, and a server device 5. In this embodiment, the information notification control device 1, the information source device 2, the information notification device 3, and the FB information acquisition device 4 are mounted on the vehicle VE. The server device 5 is located outside the vehicle VE and is configured to communicate with the information notification control device 1 via a communication network such as the Internet (not shown).

[0014] In this embodiment, the driver assistance system 100 includes a server device 5, but this is merely an example. The driver assistance system 100 may also be configured without the server device 5.

[0015] The information notification control device 1 performs control related to information notification. The information notification control device 1 determines in what notification mode to notify the information to be notified to the notification target (a detailed example is a driver) received from the information source device 2. Further, the information notification control device 1 controls the information notification device 3 so that the notification is performed in the determined notification mode. The determination of the notification mode includes, for example, the determination of notification parameters such as notification means, notification location, notification intensity, etc. Details of the information notification control device 1 will be described later.

[0016] Note that in this embodiment, the information notification control device 1 is an in-vehicle device mounted on the vehicle VE, but this is an example. The information notification control device 1 may be composed of an in-vehicle device and a server device provided communicably via a communication network such as the Internet with the in-vehicle device. Further, the information notification control device 1 mounted on the vehicle VE may be configured as a part of an in-vehicle device such as a navigation device or a display audio.

[0017] The information source device 2 is a device that serves as an information source of the information to be notified to the notification target and provides notification information to the information notification control device 1. Specifically, the information source device 2 is a device that serves as an information source of the information to be notified to the driver. The information source device 2 outputs the information to be notified to the driver toward the information notification control device 1. The information source device 2 may be configured, for example, to read information stored in a storage medium (not shown) and output it toward the information notification control device 1. Further, the information source device 2 may be configured to generate the information to be notified by itself and output it toward the information notification control device 1. Further, the information source device 2 may be configured to receive the information to be notified from another device such as a server device and output it toward the information notification control device 1.

[0018] In this embodiment, the information (notification information) output from the information source device 2 is information related to driving. The information related to driving may widely include information indicating the state of the vehicle VE, information indicating the state of the driver, information informing about the surrounding environment of the vehicle VE, information related to route guidance, attention - attracting information during driving, and the like. The information source device 2 that outputs such notification information may be constituted by, for example, a navigation device, a safety monitoring device, etc. mounted on the vehicle VE.

[0019] Note that the information source device 2 may be constituted by only one device or by a plurality of devices. Also, in this embodiment, the information source device 2 is an in - vehicle device, but at least a part of its functions may be constituted by a server device communicable with the in - vehicle device. Further, when the notification information may also include information other than information related to driving, the notification information may include entertainment information such as music and videos (movies, etc.) in a TV, radio, disk player, etc., and Internet information such as news.

[0020] The information notification device 3 performs information notification according to a command from the information notification control device 1. Specifically, the information notification device 3 performs information notification by using at least one of the driver's vision, hearing, and touch according to a command from the information notification control device 1. In this embodiment, the information notification device 3 is constituted by a plurality of types of devices. The plurality of types of devices are a display device 31, an audio output device 32, and a vibration generating device 33.

[0021] The display device 31 is a means of notifying information using the driver's vision. The display device 31 is positioned in a location where the driver of the vehicle VE to which the driver assistance system 100 is applied can easily see the displayed content. For example, the display device 31 is a liquid crystal display or an organic EL display placed on the dashboard of the vehicle VE. Alternatively, the display device 31 is a head-up display (HUD) that projects information onto the windshield of the vehicle VE. Alternatively, the display device 31 is an indicator light placed on the meter panel, rearview mirror, or side mirror. The display device 31 may also be part of a stationary or portable navigation system or safety monitoring device mounted in the vehicle. The display device 31 may also be a display operation device with an operation function such as a touch panel. There may be one or more display devices 31. If there are multiple, there may be multiple types of display devices 31.

[0022] The voice output device 32 is a means of notifying information using the driver's hearing. The voice output device 32 is, for example, a speaker positioned in a location where the driver of a vehicle VE to which the driver assistance system 100 is applied can easily hear the voice. The voice output device 32 may also be part of a stationary or portable navigation system or safety monitoring system installed in the vehicle. There may be one or more voice output devices 32.

[0023] The vibration generator 33 is a means of notifying the driver of information using their sense of touch. The vibration generator 33 includes, for example, a vibrator positioned in a location where the driver of the vehicle can feel the vibration. The vibrator is positioned, for example, in the driver's seat. More specifically, the vibrator is positioned in the seat that supports the driver's buttocks, or in the backrest that supports the driver's back when their buttocks are on the seat. The vibrator may be, for example, a vibrator with an electrical-magnetic circuit configuration in which the diaphragm of a speaker (having a structure suitable for acoustic conversion) is replaced with a diaphragm suitable for vibration transmission, or a vibrator with a configuration utilizing a piezoelectric element.

[0024] In this embodiment, the system utilizes three senses—sight, hearing, and touch—when notifying information, but this is merely an example. The types of senses that can be used when notifying information may be one or two, for example. For instance, the types of senses that can be used when notifying information may be two, such as sight and hearing. Furthermore, the information notification device 3 may consist of one type of device or multiple devices other than three.

[0025] The FB information acquisition device 4 is a means for acquiring feedback information from a user (in this embodiment, the driver) who has received a notification from the information notification device 3. The feedback information is the driver's impression, feelings, or opinion regarding the information notification, or information that allows for the inference of such impression, feelings, or opinion. The FB information acquisition device 4 outputs the acquired feedback information to the information notification control device 1. The information notification control device 1 processes the acquired feedback information and reflects it in the control of the notification. The information notification control device 1 also performs processing such as storing the acquired feedback information so that it can be used as data for learning the HMI (Human Machine Interface) control model described later.

[0026] The FB information acquisition device 4 may be, for example, a camera, a microphone, or an input device such as a touch panel. The camera is, in detail, a camera that photographs the driver, such as a camera provided in a drive recorder. From the driver's facial expressions captured in the camera's images, the driver's feelings regarding the information notification can be inferred, and this information can serve as feedback information. The microphone is, in detail, a microphone placed near the driver's seat of the vehicle VE. From the driver's voice input via the microphone, the driver's feelings regarding the information notification can be inferred, and this information can serve as feedback information. In a configuration using a voice recognition device, the driver can directly input their thoughts and feelings regarding the information notification using the microphone. The input device such as a touch panel is, in detail, a function provided in a navigation system, and the driver can directly input their thoughts and feelings regarding the information notification using the input device.

[0027] The input device for inputting information may consist of a terminal device such as a smartphone owned by the driver. However, in this case, there may be cases where the terminal device cannot communicate directly with the information notification control device 1, for example, if the vehicle VE's power is turned off. For this reason, the feedback information transmitted from the terminal device may be configured to be sent to a cloud server first, and the information notification control device 1 may be configured to acquire the feedback information from the cloud server.

[0028] Server device 5 is, in detail, a cloud server. Server device 5 is an information processing device (learning device) that performs training on the HMI control model used by the information notification control device 1. The HMI control model is an AI model, and training refers to machine learning. Server device 5 provides the trained AI model (in detail, model information of the AI ​​model) to the information notification control device 1 via a communication network such as the internet. Server device 5 also obtains the above feedback information from the information notification control device 1 and uses the feedback information to retrain the HMI control model. Details of the HMI control model and the retraining of the HMI control model will be described later.

[0029] In this embodiment, the information processing device that performs the learning (including retraining) of the HMI control model is configured as a server device 5, but this is merely an example. The information processing device that performs the learning of the HMI control model may be configured to be mounted on the vehicle VE (i.e., an in-vehicle device) and does not necessarily have to be configured as a server device.

[0030] [1-2. Information Notification Control Device] As shown in Figure 1, the information notification control device 1 comprises a controller 11 and a memory 12. The information notification control device 1 is a so-called computer device and, in addition to the controller 11 and memory 12, includes an input / output unit (not shown). In this embodiment, the information notification control device 1 also includes a communication unit (not shown) that enables communication using a communication network such as the Internet.

[0031] The controller 11 is configured to include an arithmetic circuit that performs calculations. The arithmetic circuit is more specifically composed of a processor. The processor is composed of, for example, a CPU (Central Processing Unit). The controller 11 may consist of one processor or multiple processors. If it consists of multiple processors, those processors should be provided to communicate with each other.

[0032] Memory 12 consists of volatile memory and non-volatile memory. Volatile memory is specifically RAM (Random Access Memory). Non-volatile memory is specifically ROM (Read Only Memory). Non-volatile memory may also be flash memory or a hard disk drive, etc. Non-volatile memory stores programs and data that can be read by the computer.

[0033] The functions of controller 11 are realized by the processor executing arithmetic processing according to a program stored in memory 12. The number of programs that realize the functions of controller 11 may be one or more. The functions of controller 11 may be realized by software, i.e., by an arithmetic circuit executing arithmetic processing according to a program, but may also be realized by other methods. At least some of the functions of controller 11 may be realized using, for example, an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). In other words, at least some of the functions of controller 11 may be realized by hardware using a dedicated IC or the like. Furthermore, at least some of the functions of controller 11 may be realized by using both software and hardware.

[0034] Figure 2 is a block diagram showing the configuration of the functional units of the controller 11 of the information notification control device 1. As shown in Figure 2, the controller 11 comprises an acquisition unit 110, a preprocessing unit 111, an HMI control model execution unit 112, an output control unit 113, and a feedback information processing unit 114 as its functional units. Note that each functional unit 110 to 114 is a conceptual component. The function performed by one component may be distributed among multiple components. Alternatively, the functions of multiple components may be integrated into a single component.

[0035] The acquisition unit 110 acquires information from the memory 12 and from devices (in-vehicle or external) and sensors that are configured to communicate with the information notification control device 1. The information acquired by the acquisition unit 110 includes notification information output from the information source device 2 and feedback information output from the FB information acquisition device 4. In addition, the information acquired by the acquisition unit 110 includes various types of information for input into the HMI control model, which will be described in detail later.

[0036] The preprocessing unit 111 performs various preprocessing steps necessary for providing information notification when it becomes necessary to notify the driver, such as by obtaining notification information from the information source device 2. These preprocessing steps include, for example, determining the notification priority and deciding on the notification format.

[0037] The priority of a notification is determined based on the urgency of the information to be notified. For example, if the information to be notified is information that will eliminate danger, the urgency will be high and the priority will be determined to be high. For example, if the information to be notified is directions such as turning left or right, the urgency will not necessarily be high and the priority will be determined to be low. The priority of a notification may be determined by creating a table in advance that associates the notification content (notification information) with the priority and storing it in memory 12, and then using that table. Alternatively, as another example, the priority of a notification may be determined using an AI model that takes the notification content and various information such as the occupant (driver) profile as input and outputs the priority. Such an AI model may be generated by supervised learning performed on a large amount of data with correct labels that associate the notification content and occupant profile with the correct value of the priority.

[0038] The determination of the notification format includes determining the means and location of information notification. This determination process determines, for example, which of text, graphics, sound, and vibration will be used as the means of information notification. This determination process also determines where the display, sound generation, or vibration generation will occur. Note that if there is only one notification means for each category such as display, the notification location is automatically determined by the determination of the notification means, so in such a configuration, it is not necessary to perform a determination process. The means of information notification, etc., may be determined by a pre-prepared rule base. This rule base may use, for example, notification priority information or occupant (driver) profiles. The occupant profile is, for example, stored in memory 12 (see Figure 1) in advance. For example, in the case of a driver with poor hearing, it is decided that voice will not be used as a notification means, and other means (display or vibration) will be used instead. Also, if the notification priority is high, it is decided that multiple notification means will be used.

[0039] The HMI control model execution unit 112 reads the model information 121 (see Figure 1) stored in the memory 12 and executes processing by the HMI control model. The model information 121 is the model information of the HMI control model, and in detail includes the structure and parameters of the HMI control model, as well as code instructions for executing processing by the HMI control model.

[0040] Figure 3 is a schematic diagram illustrating the HMI control model 6. As described above, the HMI control model 6 is a trained AI model that performs inference processing in response to information input and outputs the inference results. In detail, the HMI control model 6 receives priority information and notification format obtained by the preprocessing unit 111, as well as various other information, and outputs notification parameters for providing information notifications that are estimated to be comfortable for the driver, as a result of inference on said input. The HMI control model 6 with such functions may consist of a single AI model or multiple AI models. For example, the HMI control model 6 may include a visual model for notification control by display, an auditory model for notification control by sound, and a tactile model for notification control by vibration.

[0041] Other types of information input to the HMI control model 6 may include, for example, notification information, scene information, DMS (Driver Monitoring System) information, occupant profiles, and environmental information. This information is obtained, for example, from devices (in-vehicle or external devices) or sensors that are communicably connected to the information notification control device 1, or from the memory 12 provided by the information notification control device 1.

[0042] Notification information includes, for example, "Merge Warning" and "Blind Spot Warning." Scene information includes, for example, ADAS (Advanced Driver-Assistance Systems) activation scenes and surrounding vehicle approach scenes. ADAS activation scenes include automatic braking scenes. DMS information is driver monitoring information from in-vehicle cameras, such as driver drowsiness, concentration level, or posture. Occupant profiles are occupant information such as the driver, such as age, gender, or driving history. Environmental information is environmental information inside and outside the vehicle, such as brightness information inside and outside the vehicle, noise information inside the vehicle, vibration information inside the vehicle, or weather information.

[0043] The notification parameters output by the HMI control model 6 are, in detail, notification parameters corresponding to the notification format determined by the preprocessor 111. For example, if it is determined that text, graphics, voice, and vibration will be used as notification formats, parameters corresponding to these will be output. For example, for text and graphics, parameter information such as size, placement, and color scheme will be output. For voice, parameter information such as volume and sound quality will be output. For vibration, parameter information such as vibration intensity and vibration frequency will be output.

[0044] The output control unit 113 (see Figure 2) controls the information notification device 3 according to the notification signal indicating the content of the notification information and the notification parameters output by the HMI control model 6. In detail, the output control unit 113 controls the display device 31, the audio output device 32, and the vibration generator 33 (all of which are shown in Figure 1). As a result of this control, the driver receives information notification from at least one of the display device 31, the audio output device 32, and the vibration generator 33.

[0045] The feedback information processing unit 114 processes the feedback information acquired from the FB information acquisition device 4. For example, the feedback information processing unit 114 processes the input information to be input to the HMI control model 6 so that notification parameters reflecting the feedback information are output from the HMI control model 6. The feedback information processing unit 114 also stores the feedback information as information for updating (retraining) the HMI control model 6. The stored information is used, for example, by being transmitted to the server device 5 (see Figure 1).

[0046] <2. Information Processing Device (Learning Device)> Next, we will describe the information processing device (learning device) 5 that performs the learning of the HMI control model 6 (AI model) described above. In this embodiment, the information processing device 5 is a server device, but as mentioned above, it may be an in-vehicle device mounted on the vehicle VE. Also, in this embodiment, the target of learning performed using the information processing device 5 is the HMI control model 6, but this is an example, and it may be other AI models. For this reason, below, the target of learning performed using the information processing device 5 will simply be referred to as the AI ​​model.

[0047] The information processing device 5 trains an AI model using pre-prepared initial training data. The AI ​​model training is, in detail, supervised learning. Any known supervised learning method may be used. The initial training data is, in detail, a dataset composed of multiple data points. Each data point in the initial training data is labeled with a correct answer. If the AI ​​model is an HMI control model 6, the correct answer labels are notification parameters that the user (in a detailed example, the driver) finds comfortable.

[0048] In this embodiment, the information processing device 5 is configured to perform initial training of the AI ​​model using initial training data, but this is merely an example. The information processing device 5 may also be configured to perform only retraining of an already trained AI model, without performing initial training. In this embodiment, the retraining method is distinctive, so the configuration related to retraining will be described in detail below.

[0049] Figure 4 is a block diagram illustrating the schematic configuration of an information processing device 5 according to an embodiment of the present invention. As described above, the information processing device 5 retrains a trained AI model using first training data. The information processing device 5 is characterized by its retraining method. For this reason, Figure 4 shows the components involved in retraining, while descriptions of other components are omitted. The first training data is the training data (dataset) used to train the trained AI model, and a typical example is the initial training data.

[0050] As shown in Figure 4, the information processing device 5 comprises a controller 51, a memory 52, and a communication unit 53. The information processing device 5 is a so-called computer device.

[0051] The controller 51 is configured to include an arithmetic circuit that performs calculations. The arithmetic circuit is more specifically composed of a processor. The processor is composed of, for example, a CPU. The controller 51 may be composed of one processor or multiple processors. If it is composed of multiple processors, those processors should be provided to be able to communicate with each other.

[0052] Memory 52 comprises volatile memory and non-volatile memory. The volatile memory is specifically RAM. The non-volatile memory is specifically ROM. The non-volatile memory may also be flash memory or a hard disk drive, etc. The non-volatile memory stores programs and data that can be read by the computer. The programs include a learning program 521 that causes the computer to execute a learning method for retraining a trained AI model.

[0053] The program, including the learning program 521 stored in memory 52, may be provided, for example, on a computer-readable non-volatile recording medium. The non-volatile recording medium may be, for example, an optical recording medium (e.g., an optical disc), a magneto-optical recording medium (e.g., a magneto-optical disc), a USB memory, or an SD card, in addition to the non-volatile memory described above. As another example, the learning program 521, etc., may be provided from a program provision server via a communication line such as the Internet (a configuration provided by so-called download).

[0054] The communication unit 53 is configured as a communication interface having an interface circuit for connecting to a communication network (not shown), such as the Internet. However, if the information processing device 5 is an in-vehicle device rather than a server device, the communication unit 53 does not need to be provided.

[0055] The functions of the controller 51 are realized by the processor executing arithmetic processing according to a program stored in memory 52. ​​The number of programs that realize the functions of the controller 51 may be one or more. The functions of the controller 51 may be realized by software, but may also be realized by other methods. At least some of the functions of the controller 51 may be realized by hardware, for example, by using an ASIC or FPGA. Furthermore, at least some of the functions of the controller 51 may be realized by using a combination of software and hardware.

[0056] As shown in Figure 4, the controller 51 comprises, as its functional units, a data acquisition unit 511, a data separation unit 512, a data merging unit 513, a learning unit 514, a determination unit 515, and an additional data extraction unit 516. Note that each functional unit 511 to 516 is a conceptual component. The function performed by one component may be distributed among multiple components. Alternatively, the functions of multiple components may be integrated into a single component.

[0057] The data acquisition unit 511 acquires training data to be used for retraining the AI ​​model that has been trained using the first training data. The training data acquired by the data acquisition unit 511 includes the first training data (a typical example being the initial training data) and the second training data. The data acquisition unit 511 acquires the first training data and the second training data from, for example, memory 52. ​​Instead of acquiring from memory 52, acquisition using a communication network such as the internet or acquisition from a non-volatile recording medium such as an optical disc may be used.

[0058] The second training data is training data based on feedback information for the AI ​​model. More specifically, the feedback information for the AI ​​model is feedback information from users who have used the AI ​​model, and in this embodiment, it is information acquired by the FB information acquisition device 4 (see Figure 1) described above. The second training data is a dataset that collects multiple data with correct labels, each with a correct label obtained based on the feedback information. The generation of the second training data may be performed by a person using the information accumulated by the feedback information processing unit 114 (see Figure 2), or it may be performed by a device. Alternatively, the generation of the second training data may be performed by both a person and a device.

[0059] The data separation unit 512 separates the training data acquired by the data acquisition unit 511. Specifically, the data separation unit 512 divides the multiple ground truth labeled data included in the training dataset into two sets: training data used to generate an AI model through learning, and evaluation data used to evaluate the AI ​​model generated using the training data. This data division may be configured such that the data is randomly distributed to each set, for example, so that the proportion of data in each set is predetermined. The predetermined proportions may be, for example, 90% training data and 10% evaluation data.

[0060] In this embodiment, the data separation unit 512 (i.e., the information processing device 5) separates the first learning data into training data and evaluation data before performing the learning process. The data separation unit 512 (i.e., the information processing device 5) also separates the second learning data into training data and evaluation data before performing the learning process. Note that the separation of learning data is not mandatory; for example, all learning data may be used as either training data or evaluation data. That is, data separation processing may not be performed on at least one of the first or second learning data.

[0061] The data merging unit 513 combines data from multiple datasets (data with correct labels) to form a single dataset (training dataset) used for generating an AI model through learning. In this embodiment, the data merging unit 513 combines the training data separated from the first training data with the training data separated from the second training data. That is, the data merging unit 513 combines the data from the first training data that has been assigned to training and the data from the second training data that has been assigned to training to form a single training dataset. The data merging unit 513 also combines data extracted by the additional data extraction unit 516, which will be described in detail later, to add to an existing training dataset.

[0062] The learning unit 514 uses the training dataset obtained after the data merging process by the data merging unit 513 to perform training on the already trained AI model. In detail, the training process involves retraining the already trained AI model using the training dataset to generate a new AI model. As described above, the training dataset includes data extracted from the first training data and data extracted from the second training data. Therefore, the information processing device 5 is configured to perform training on the AI ​​model trained using the first training data, using the first training data and the second training data.

[0063] The learning process by the learning unit 514 is performed using supervised learning with data that has correct labels. Any known supervised learning method may be used. The learning process by the learning unit 514 yields a retrained AI model. However, in this embodiment, the AI ​​model obtained by the learning unit 514 will not become a retrained AI model unless it clears the judgment criteria set in the judgment unit 515, which will be described in detail later. For this reason, the AI ​​model after the learning process by the learning unit 514 may be referred to as a provisional AI model below. The learning unit 514 may also retrain the provisional AI model.

[0064] The determination unit 515 determines whether the provisional AI model after the learning process in the learning unit 514 has cleared the target value. A provisional AI model that has cleared the target value is recognized as a fully trained AI model through relearning. The AI ​​model recognized as a fully trained AI model through relearning becomes the AI ​​model that replaces the fully trained AI model using the first training data. On the other hand, a provisional AI model that has not cleared the target value will undergo additional learning (additional training) until it clears the target value.

[0065] In this embodiment, the determination unit 515 (i.e., the information processing device 5) executes a determination process using a first target value, which is the target value of the accuracy rate for the first training data, and a determination process using a second target value, which is the target value of the accuracy rate for the second training data, after the learning process. In other words, in this embodiment, a target value based on the first training data previously used to generate the AI ​​model and a target value based on the second training data based on feedback information are set separately, and the provisional AI model is evaluated using the two target values. The provisional AI model then becomes a trained AI model through retraining when it clears both of the two target values. Details of the determination process in the determination unit 515 will be described later.

[0066] The additional data extraction unit 516 extracts additional data to be used for further training of the provisional AI model, which is performed when the determination unit 515 fails to meet the target value. In this embodiment, as described above, there are two determination processes performed after the training process. The additional data extraction process differs depending on which of the two determination processes determines that the target value was not met. Details of the processing by the additional data extraction unit 516 will be described later.

[0067] As can be seen from the above explanation, the information processing device 5 of this embodiment is configured to determine whether additional learning is necessary according to the results of each of the two judgment processes and to generate a trained AI model through retraining. In this configuration, where two types of target values ​​are set and a trained AI model is generated through retraining, it is possible to adjust whether the final AI model is a model that emphasizes the reflection of user feedback information or a model that emphasizes the suppression of overfitting by adjusting the target values ​​of the accuracy rates for the first training data and the second training data. In other words, by appropriately adjusting the target values ​​of the accuracy rates for the first training data and the second training data, it is possible to generate an AI model that appropriately reflects feedback information and suppresses overfitting. The learning method that can achieve such effects will be described in more detail below.

[0068] <3. Learning Methods> Figure 5 is a flowchart illustrating the flow of a learning method (specifically, a retraining method) for a pre-trained AI model using first training data, as executed by the information processing device 5. This flowchart shows the technical content of a computer program (learning program 521) that enables a computer to implement the AI ​​model learning method of this embodiment. This computer program can be stored on various non-volatile recording media readable by a computer and provided (sold, distributed, etc.). This computer program may consist of only one program, or it may consist of multiple programs working together. The process shown in Figure 5 is executed when a predetermined amount of learning data (data with correct labels) based on feedback information has been accumulated.

[0069] In Figure 5, the first training data is the initial training data used for the initial training of the AI ​​model, and in the explanation of Figure 5, the first training data will be referred to as the initial training data. Also, in the explanation of Figure 5, the second training data will be referred to as the FB (feedback) training data. Furthermore, in Figure 5, it is assumed that the controller 51 (data acquisition unit 511) has already acquired the initial training data and the FB training data.

[0070] Furthermore, in explaining Figure 5, we will refer to Figure 6 as appropriate. Figure 6 is a schematic diagram showing the data changes during the execution of the process shown in Figure 5. As shown in Figure 6, the initial training data acquired by the data acquisition unit 511 includes n (integer, n>1) data points with correct labels (data X1 to data Xn). In addition, the FB training data acquired by the data acquisition unit 511 includes m (integer, m>1) data points with correct labels (data Y1 to data Yn).

[0071] Figure 7 shows an example of data with correct labels. The example shown in Figure 7 is data when the AI ​​model is HMI control model 6 (see Figure 3), and in detail, it is data with correct labels related to vibration-based notifications. As shown in Figure 7, data with correct labels has correct labels that are used to compare the input values ​​input to the AI ​​model with the output values ​​output from the AI ​​model. The numerical form of each item (priority, gender, age, vibration intensity, vibration frequency, etc.) is defined using numerical conversion tables, etc., so that it is a numerical form suitable for handling in the AI ​​model. For example, for gender, male is defined as "1", female as "2", and others as "3". Also, for example, for age, under 20 is defined as "1", 21 to 25 is defined as "2", 26 to 35 is defined as "3", 36 to 59 is defined as "4", 60 to 69 is defined as "5", 70 to 79 is defined as "6", and 80 and over is defined as "7".

[0072] As described above, the feedback information processing unit 114 (see Figure 2) performs the process of accumulating feedback information, but because it is necessary to generate training data as shown in Figure 7, it performs the following processing, for example. When accumulating data, the feedback information processing unit 114 stores the values ​​actually used when the information was notified as input values ​​for each item in the training data. In this storage process, the feedback information processing unit 114 associates the feedback information, which is the user's response to the information notification, as the correct label and stores it in the form of training data. The process of obtaining the correct label varies depending on what kind of feedback the user provides. For example, if the user directly inputs the parameter values ​​they deem appropriate (vibration intensity, etc.), the user's input values ​​become the correct labels. Also, if the user makes a relative evaluation of the information notification (for example, strong, slightly strong, appropriate, slightly weak, weak, etc.), the values ​​used when the information was notified (actual values) are corrected based on that evaluation, and the corrected values ​​become the correct labels. For example, if the relative evaluation is "appropriate," the actual values ​​become the correct labels. Furthermore, for example, if the relative evaluation is "strong," the correct label may be a value obtained by correcting the actual value so that it becomes weaker than the actual value. Also, for example, if the feedback information is emotion, the system may be configured to obtain a corrected value by applying the acquired emotion information to a pre-prepared correction value table, correct the actual value, and obtain the value obtained through this correction as the correct label.

[0073] In step S1 shown in Figure 5, the controller 51 (data separation unit 512) separates the initial training data and the FB training data. As shown in Figure 6, the initial training data and the FB training data are divided into training data and evaluation data. Both the training data and the evaluation data contain data with correct labels. The ratio (separation ratio) of the number of data points in the training data to the number of data points in the evaluation data is, for example, 9:1. Note that the separation ratio shown here is an example and may be changed as appropriate. Also, the separation ratio may differ between the initial training data and the FB training data. Once the separation of the training data is complete, the process proceeds to the next step S2.

[0074] In step S2, the controller 51 (data merging unit 513) merges the training data separated from the initial training data with the training data separated from the FB training data. As a result, as shown in Figure 6, a training dataset is generated in which the data with correct labels contained in the training data separated from the initial training data and the data with correct labels contained in the training data separated from the FB training data are combined into one. Note that, as described above, the data used for the learning process may not be a configuration that combines the data separated from the initial training data and the FB training data, but rather a configuration that combines all of the initial training data and the FB training data. Once the data merging is complete, the process proceeds to the next step S3.

[0075] In step S3, the controller 51 (learning unit 514) performs a learning process on the AI ​​model that has been trained using the initial training data. This learning process is so-called supervised learning. As shown in Figure 6, this learning process is performed using the training dataset obtained by data merging. In other words, this learning process is performed using the training data separated from the initial training data and the training data separated from the FB training data. That is, the initial training data (first training data) used in the learning process is the training data separated from the initial training data. Also, the FB training data (second training data) used in the learning process is the training data separated from the FB training data.

[0076] As described above, the AI ​​model obtained through the learning process in step S3 is a provisional AI model whose retraining is not yet complete. In the following, the provisional AI model will be referred to as follows: The provisional AI model obtained through the learning process using a training dataset obtained by combining the training data separated from the initial learning data and the training data separated from the FB learning data will be referred to as the first provisional AI model. Furthermore, the provisional AI model obtained through additional learning using a training dataset with additional data, which is constructed by adding the additional data extracted by the additional data extraction unit 516 (see Figure 4) to the training dataset described above, will be referred to as the second provisional AI model. Once the learning process in step S3 is completed, the process proceeds to the next step, S4.

[0077] In step S4, the controller 51 (decision unit 515) calculates the accuracy rate using the first provisional AI model obtained through the learning process and the evaluation data (FB evaluation data) separated from the FB learning data (see also Figure 6). In other words, the accuracy rate for the FB learning data (second learning data) in this embodiment is the accuracy rate for the evaluation data separated from the FB learning data. With this configuration, the evaluation of the generated AI model can be performed separately from the data used to generate the AI ​​model.

[0078] In detail, the determination unit 515 performs inference processing using the first provisional AI model for each data point with a correct label included in the FB evaluation data, compares the obtained inference result with the correct label, and determines whether it is correct or not. If the inference result is different from the correct label, it is determined to be incorrect. Note that if the AI ​​model is the HMI control model 6 (see Figure 3), the inference result is a notification parameter. The determination unit 515 obtains the accuracy rate by calculating the percentage of data points for which a correct answer was obtained out of all data points included in the FB evaluation data. Once the accuracy rate is obtained, the process proceeds to the next step S5.

[0079] In step S5, the controller 51 (determination unit 515) calculates whether the accuracy rate obtained in step S4 is equal to or greater than a predetermined target value. The target value is a value determined by a person such as a designer and is stored in memory 52 in advance. The target value in step S5 is a target value set for FB evaluation data and is an example of the second target value of the present invention. If the accuracy rate is equal to or greater than the target value (Yes in step S5), the process proceeds to step S8. If the accuracy rate is less than the target value (No in step S5), the process proceeds to step S6.

[0080] In step S6, the controller 51 (additional data extraction unit 516) calculates the error between the expected value based on the correct label and the output value of the first provisional AI model for each of the FB evaluation data that was determined to be incorrect in step S4. The expected value based on the correct label is the correct label itself if the AI ​​model has only one output item, and one correct value selected from among the multiple correct values ​​contained in the correct label if the AI ​​model has multiple output items. If the AI ​​model has multiple output items, for example, an error may be calculated for each item, and the sum of these calculated errors may be used as the error for each incorrect data. Once the error has been calculated for all the incorrect data, the process proceeds to the next step S7.

[0081] In step S7, the controller 51 (additional data extraction unit 516) extracts additional data to be used for additional learning from the incorrect data, according to the result of error calculation for the incorrect data. Specifically, from the incorrect data, the data up to the top M (an integer M≧1: the designer etc. sets an appropriate value based on experiments etc.) with the smallest error from the correct answer is extracted as additional data. In the example shown in Figure 6, only one additional data is extracted, "Data Yex". That is, in the example shown in Figure 6, M=1, and only the incorrect data with the smallest error is extracted as additional data. When step S7 is completed, the process returns to step S2 above.

[0082] The processing from step 2 onward when returning from step S7 to step S2 is largely as described above, but there are some slight differences, which will be explained here.

[0083] The data merging in step S2 involves adding (merging) the additional data extracted in step S7 to the training dataset (the previously obtained training dataset), which consists of training data separated from the initial training data and the FB training data, respectively. In the example shown in Figure 6, "Data Yex" is added to the previously obtained training dataset. The subsequent learning (step S3) is additional learning using the training dataset to which the additional data has been merged. Hereafter, the following terms will be used to describe additional learning. Additional learning performed by adding additional data extracted from the FB evaluation data to the training dataset will be referred to as the first additional learning. Additional learning performed by adding additional data extracted from the initial evaluation data (described later) to the training dataset will be referred to as the second additional learning.

[0084] As can be seen from the above, the information processing device 5 compares the accuracy rate of the first provisional AI model obtained through the learning process on the FB learning data (second learning data) with a predetermined target value (second target value) to determine whether or not to perform additional learning. Furthermore, if the accuracy rate on the FB learning data is smaller than the second target value, the information processing device 5 adds the Mth most incorrect data from the FB learning data, starting with the data with the smallest error from the correct answer, to the learning data and performs additional learning (see Figure 6). With this configuration, additional learning is performed by adding data with a small error from the correct answer, so that the AI ​​model can be modified so that the incorrect data becomes correct with minimal impact on the previously performed learning.

[0085] If the process returns to step S2 from step S7, it is possible that the accuracy rate in step S5 may again be lower than the target value. In such cases, the processes in steps S6 and S7 will be performed again, followed by the processes from step S2 onward. Also, the diagram in Figure 6 is written on the premise that the first additional learning is performed, but the first additional learning may not be performed. In this case, the error calculation using FB evaluation data (step S6) and the extraction of additional data (step S7) will not be performed. Furthermore, in the example shown in Figure 6, the first additional learning is assumed to be performed only once, and the processes such as the calculation of the accuracy rate using FB evaluation data performed after the first additional learning are omitted.

[0086] In step S8, the controller 51 (decision unit 515) calculates the accuracy rate using the first provisional AI model or the second provisional model and evaluation data (initial evaluation data) separated from the initial training data (see also Figure 6). If no additional training has been performed beforehand, the first provisional AI model is used to calculate the accuracy rate, and if additional training has been performed beforehand, the second provisional AI model is used to calculate the accuracy rate. In this embodiment, the accuracy rate for the initial training data (first training data) is the accuracy rate for the evaluation data separated from the initial training data. With this configuration, the generated AI model can be evaluated separately from the data used to generate the AI ​​model.

[0087] In detail, the determination unit 515 performs inference processing using either the first provisional AI model or the second provisional AI model for each of the data with correct labels included in the initial evaluation data, compares the obtained inference result with the correct label, and determines whether it is correct or not. If the inference result is different from the correct label, it is determined to be incorrect. Note that in Figure 6, since it is assumed that the first additional learning will be performed, the AI ​​model used to calculate the accuracy rate is the second provisional AI model. The determination unit 515 obtains the accuracy rate by calculating the percentage of data for which correct answers were obtained out of all the data included in the initial evaluation data. Once the accuracy rate is obtained, the process proceeds to the next step S9.

[0088] In step S9, the controller 51 (determination unit 515) calculates whether the accuracy rate obtained in step S8 is equal to or greater than a predetermined target value. The target value is a value determined by a person such as a designer and is stored in memory 52 in advance. The target value in step S9 is a target value set for initial evaluation data and is an example of the first target value of the present invention. The target value in step S5 and the target value in step S9 may be the same or different. If the accuracy rate is equal to or greater than the target value (Yes in step S9), the learning method shown in Figure 5 is completed, and a trained AI model is obtained through retraining. The trained AI model obtained through retraining is, in detail, the provisional AI model used to calculate the accuracy rate in step S8. This provisional AI model may be the first provisional AI model or the second provisional AI model. If the accuracy rate is less than the target value (No in step S9), the process proceeds to step S10.

[0089] In step S10, the controller 51 (additional data extraction unit 516) calculates the error between the expected value obtained from the correct label and the output value of the first or second provisional AI model for each of the data points from the initial evaluation data that were determined to be incorrect in step S8. The method for calculating the error is the same as in step S6. When calculating the error, if additional learning has not been performed beforehand, the output value of the first provisional AI model is used, and if additional learning has been performed beforehand, the output value of the second provisional AI model is used. In the example shown in Figure 6, since additional learning has been performed beforehand, the output value of the second provisional AI model is used when calculating the error. Once the error has been calculated for all of the incorrect data points, the process proceeds to the next step S11.

[0090] In step S11, the controller 51 (additional data extraction unit 516) extracts additional data from the incorrect data to be used for additional learning (second additional learning) according to the result of error calculation for the incorrect data. Specifically, the top N (an integer N≧1: the designer etc. sets an appropriate value based on experiments etc.) data with the smallest error from the correct answer are extracted as additional data from the incorrect data. In the example shown in Figure 6, only one additional data is extracted, "Data Xex". That is, in the example shown in Figure 6, N=1, and only the incorrect data with the smallest error is extracted as additional data. Note that the additional data extraction process in steps S7 and S11 is the same, but the number of data extracted M in step S7 and the number of data extracted N in step S11 may be different. When step S11 is completed, the process returns to step S2 above.

[0091] The processing from step 2 onward when returning from step S11 to step S2 is largely as described above, but there are some slight differences, which will be explained here.

[0092] The data merging in step S2, as shown in Figure 6, involves adding (merging) the additional data extracted in step S11 to the training dataset used to generate the provisional AI model to be subjected to further training. In the example shown in Figure 6, the provisional AI model to be subjected to further training is the second provisional AI model because it has already undergone further training (specifically, the first further training), but it could also be the first provisional AI model. Also, in the example shown in Figure 6, the training dataset to which the additional data is added is the training dataset to which the additional data (data Yex) has already been added. In the example shown in Figure 6, "data Xex" has been added to the training dataset. Furthermore, the training performed after this (step S3) is further training (second further training) using the training dataset to which the additional data has been merged.

[0093] If the process returns from step S11 to step S2, it is possible that the accuracy rate in step S5 may again be lower than the target value. In such cases, after the processes in steps S6 and S7 are repeated, the first additional learning will be performed. Also, if the process returns from step S11 to step S2, it is possible that the accuracy rate in step S9 may again be lower than the target value. In such cases, after the processes in steps S10 and S11 are repeated, the second additional learning will be performed. Furthermore, the diagram in Figure 6 is written assuming that the second additional learning is performed, but it may not be performed. In this case, the error calculation using the initial evaluation data (step S10) and the extraction of additional data (step S11) are not performed. In addition, in the example shown in Figure 6, the second additional learning is assumed to be performed only once, and the description of the judgment process using evaluation data performed after the second additional learning is omitted.

[0094] As can be seen from the above, the information processing device 5 compares the accuracy rate of the second provisional AI model obtained through additional learning, or the first provisional AI model obtained without additional learning, on the initial learning data (first learning data) with a predetermined target value (first target value) to determine whether or not to perform additional learning. Thus, in this embodiment, the target value for the accuracy rate in the FB learning data and the target value for the accuracy rate in the initial learning data are set separately, and the necessity of additional learning is determined for each to generate a trained AI model through retraining. For this reason, when generating a trained AI model through retraining, by adjusting each target value, it is possible to adjust whether to prioritize the reflection of feedback information or to prioritize the suppression of overfitting due to feedback information.

[0095] Furthermore, the information processing device 5 performs the judgment process using the target accuracy value (first target value) for the initial training data after the judgment process using the target accuracy value (second target value) for the FB training data. This configuration makes it possible to perform additional training using the initial training data if the provisional AI model is excessively influenced by the feedback information. As a result, it is possible to suppress the AI ​​model obtained after retraining from becoming an AI model that excessively reflects the feedback information.

[0096] Furthermore, if the accuracy rate in the initial training data is lower than the first target value, the information processing device 5 adds the top N data points from the initial training data that were incorrect but had the smallest error with the correct answer to the training data and performs additional training (see Figure 6). With this configuration, additional training is performed by adding data with a small error with the correct answer, so that the AI ​​model can be modified so that the incorrect data becomes correct with minimal impact on the previously performed training.

[0097] In this embodiment, for example, by setting a high target accuracy value for the FB training data, it is possible to generate an AI model that prioritizes the reflection of feedback information. This configuration is advantageous when there is little accumulated feedback information and little FB training data. Also, for example, by setting a high target accuracy value for the initial training data, it is possible to generate an AI model while suppressing overfitting due to feedback information. With this configuration, for items for which there is no feedback information, it is possible to generate an AI model that reflects FB information while retaining the functions obtained from learning with the initial training data. By setting a good balance between the target accuracy value for the FB training data and the target accuracy value for the initial training data, it is possible to obtain an AI model that reflects feedback information while suppressing overfitting.

[0098] In the above configuration, a trained AI model is obtained through retraining when the accuracy of the provisional AI model exceeds two target values. However, this is an example. For example, in the processing in steps S5 and S9 in Figure 5, the accuracy may not reach the target value. To account for such cases, an upper limit on the number of processing iterations may be set for the loops that perform steps S6 and S7, and for the loops that perform steps S10 and S11. In a configuration where an upper limit on the number of processing iterations is set, when the number of processing iterations reaches the upper limit, the provisional AI model with the highest accuracy among the provisional AI models obtained in the processing performed up to that point may be considered as the AI ​​model that has cleared the target value. This makes it possible to generate a trained AI model through retraining even if the accuracy does not reach the target value.

[0099] <4. Things to keep in mind> The various technical features disclosed in the embodiments for carrying out the invention as described herein can be modified in various ways without departing from the spirit of the technical creation. Furthermore, the multiple embodiments and modifications disclosed in the embodiments for carrying out the invention as described herein may be combined to the extent possible. [Explanation of symbols]

[0100] 5. Server equipment, information processing equipment 6. HMI control model (AI model) 521...Learning Program

Claims

1. An information processing device for retraining a trained AI model using first training data, comprising a controller, The aforementioned controller, The AI ​​model is trained using the first training data and the second training data, which is training data based on feedback information for the AI ​​model. After the learning process, a judgment process is performed using a first target value, which is the target value of the accuracy rate for the first learning data, and a judgment process is performed using a second target value, which is the target value of the accuracy rate for the second learning data. An information processing device that determines whether additional training is necessary according to each of the aforementioned judgment processing results and generates a trained AI model through retraining.

2. The information processing apparatus according to claim 1, wherein the determination process using the first target value is performed after the determination process using the second target value.

3. The aforementioned controller, The accuracy rate of the first provisional AI model obtained through the learning process for the second learning data is compared with the second target value to determine whether or not to perform the additional learning. The information processing device according to claim 2, which compares the accuracy rate of the second provisional AI model obtained by the additional learning, or the first provisional AI model obtained without the additional learning, on the first learning data with the first target value to determine whether or not to perform the additional learning.

4. The aforementioned controller, If the accuracy rate for the second training data is smaller than the second target value, the data from the second training data that were incorrect, up to the top M (an integer M ≥ 1) data with the smallest error from the correct answer, are added to the training data, and the additional training is performed. The information processing device according to claim 3, wherein if the accuracy rate for the first training data is smaller than the first target value, the additional training is performed by adding the top N (an integer N ≥ 1) data from the first training data that were incorrect, in order of the smallest error from the correct answer, to the training data.

5. Before performing the learning process, the controller separates the first learning data into training data and evaluation data. The first training data used in the learning process is training data separated from the first training data, The information processing apparatus according to any one of claims 1 to 4, wherein the accuracy rate for the first training data is the accuracy rate for evaluation data separated from the first training data.

6. Before performing the learning process, the controller separates the second learning data into training data and evaluation data. The second training data used in the learning process is training data separated from the second training data, The information processing apparatus according to claim 5, wherein the accuracy rate for the second training data is the accuracy rate for evaluation data separated from the second training data.

7. The AI ​​model trained using the first training data is subjected to a training process using the first training data and the second training data, which is training data based on feedback information for the AI ​​model. After the learning process, a judgment process is performed using a first target value, which is the target value of the accuracy rate for the first learning data, and a judgment process is performed using a second target value, which is the target value of the accuracy rate for the second learning data. A computer-based method for training an AI model, comprising determining whether additional training is necessary or not according to each of the aforementioned judgment processing results, and generating a trained AI model through retraining.

8. The AI ​​model, which has been trained using the first training data, is subjected to a training process using the first training data and the second training data, which is training data based on feedback information for the AI ​​model. After the learning process, a judgment process is performed using a first target value, which is the target value of the accuracy rate for the first learning data, and a judgment process is performed using a second target value, which is the target value of the accuracy rate for the second learning data. The process involves determining whether additional training is necessary based on the results of each of the aforementioned judgment processes, and generating a trained AI model through retraining. A learning program that instructs a computer to execute a command.

Images