Learning device

Abstract

This learning device includes: an extraction unit that, in response to the input of a state action string including at least one combination of a state and an action, extracts a feature vector using a mechanism configured to distribute the feature vector on a feature space; and an update unit that, on the basis of the feature vector extracted by the extraction unit, updates a prototype that is a typical example of the state action string.

Description

Learning Device

[0001] The present disclosure relates to a learning device, a learning method, a recording medium, and an inference device.

[0002] There are known techniques that are used to infer the behavior of an object to be imitated depending on the state.

[0003] For example, Patent Document 1 describes a device that infers an action in response to a state input and outputs a typical example (prototype) used during the inference. For example, in the case of the technology described in Patent Document 1, the device learns a sequence of actions that will serve as a typical example (prototype) during learning. Furthermore, during inference, the device acquires a prototype corresponding to the state to be inferred. The device then infers an action based on the state and the prototype. As a result, the device can output the inferred action and also output a prototype that is a typical example used during inference.

[0004] International Publication No. 2021 / 242585

[0005] In the case of the technology described in Patent Document 1, only a specific prototype is selected during inference, and only the same prototype is output as the basis for inference. As such, there is a problem in that it may be difficult to output more appropriate evidence.

[0006] Therefore, one of the objects of the present disclosure is to provide a learning device, a learning method, a recording medium, and an inference device that can solve the above-mentioned problems.

[0007] In order to achieve this object, the learning device in the present disclosure has a configuration including: an extraction unit that extracts a feature vector using a mechanism that distributes feature vectors in a feature space in response to an input of a state-action sequence that includes at least one combination of a state and an action; and an update unit that updates a prototype, which is a typical example of the state-action sequence, based on the feature vector extracted by the extraction unit.

[0008] Furthermore, the learning method in the present disclosure is configured such that an information processing device extracts a feature vector using a mechanism that distributes feature vectors in a feature space in response to an input of a state-action sequence including at least one combination of a state and an action, and updates a prototype, which is a typical example of the state-action sequence, based on the extracted feature vector.

[0009] Furthermore, the recording medium in the present disclosure is a computer-readable recording medium having recorded thereon a program for causing an information processing device to execute the process of: extracting a feature vector using a mechanism that distributes feature vectors in a feature space in response to input of a state-action sequence including at least one combination of a state and an action; and updating a prototype, which is a typical example of the state-action sequence, based on the extracted feature vector.

[0010] Furthermore, the inference device of the present disclosure has a configuration including: an acquisition unit that acquires a learned prototype using a mechanism that distributes feature vectors in a feature space according to the state to be inferred; an inference unit that infers an action based on the state to be inferred and the prototype acquired by the acquisition unit; and an output unit that outputs the action inferred by the inference unit and the prototype used during the inference.

[0011] According to the above-described configurations, it is possible to realize the output of more appropriate grounds.

[0012] FIG. 1 is a diagram illustrating an overview of an inference device according to the present disclosure. FIG. 2 is a block diagram illustrating an example configuration of an inference device according to the present disclosure. FIG. 3 is a diagram illustrating an example of a state-action sequence. FIG. 4 is a diagram illustrating an example of option selection. FIG. 5 is a diagram illustrating an example of splitting a state-action sequence. FIG. 6 is a diagram illustrating an example of feature extraction processing. FIG. 7 is a diagram illustrating an example of feature extraction processing. FIG. 8 is a diagram illustrating an example of feature extraction processing. FIG. 9 is a diagram illustrating an example of prototype update processing. FIG. 10 is a diagram illustrating an example of prototype update processing. FIG. 11 is a diagram illustrating an example of output. A flowchart illustrating an example operation during learning. A flowchart illustrating an example operation during inference. A diagram illustrating an example of feature vectors. A diagram illustrating an example of feature vectors. A diagram illustrating an example hardware configuration of a learning device according to the present disclosure. A block diagram illustrating an example configuration of a learning device. A flowchart illustrating an example operation of a learning device. A block diagram illustrating a configuration of a second inference device according to the present disclosure.

[0013] [First Embodiment] A configuration example of a first inference device 100 according to the present disclosure will be described with reference to FIGS. 1 to 17. FIG. 1 is a diagram illustrating an overview of the inference device 100. FIG. 2 is a block diagram illustrating an example of the configuration of the inference device 100. FIG. 3 is a diagram illustrating an example of a state-action sequence. FIG. 4 is a diagram illustrating an example of option selection. FIG. 5 is a diagram illustrating an example of splitting a state-action sequence. FIGS. 6 to 9 are diagrams illustrating an example of feature extraction processing. FIGS. 10 to 12 are diagrams illustrating an example of prototype update processing. FIG. 13 is a diagram illustrating an example of output. FIG. 14 is a flowchart illustrating an example of operation during learning. FIG. 15 is a flowchart illustrating an example of operation during inference. FIGS. 16 and 17 are diagrams illustrating examples of feature vectors. Note that in the present disclosure, the drawings may be associated with one or more embodiments.

[0014] In a first embodiment of the present disclosure, an inference device 100 (learning device) that is an information processing device that infers an action in response to an input of a state will be described. For example, as shown in FIG. 1 , during learning, the inference device 100 learns (by machine learning) a prototype that serves as a typical example based on a state-action sequence that indicates transitions between states and actions. Furthermore, during inference, the inference device 100 acquires a prototype corresponding to the state to be inferred. Then, the inference device 100 infers an action based on the state and the prototype. Thereafter, the inference device 100 can output the inferred action and the prototype that is the typical example used during inference.

[0015] Furthermore, when learning a prototype, the inference device 100 described in this disclosure extracts feature vectors using an architecture with a mechanism for distributing feature vectors in a feature space to perform the learning. For example, the inference device 100 performs the extraction using an autoencoder architecture that includes an encoder that receives a state-action sequence and obtains a feature vector, and a decoder that receives the feature vector and reconstructs the state-action sequence. With this configuration, feature vectors are distributed in the feature space. Therefore, the feature vectors of prototypes that are updated according to the extracted feature vectors can also be distributed. This allows the inference device 100 to select an appropriate prototype during inference without selecting only a specific prototype.

[0016] The inference device 100 described in the present disclosure can be used, for example, to infer and imitate the behavior of an imitation target, such as a specialist or other expert. For example, the inference device 100 can be used for retail-related inference and decision-making support, such as inferring the behavior that the inference target should take based on the state and behavior of a good customer and recommending the necessary behavior to perform the inferred behavior. The inference device 100 may also be used in healthcare-related applications, such as supporting decision-making by inferring and presenting the behavior of a user who wants to improve their health based on the state and behavior of a user who has shown health improvement effects. In addition to the above-mentioned examples, the inference device 100 may be used in any situation where it infers behavior according to a state using machine learning and supports various decision-making.

[0017] In the present disclosure, a state refers to data indicating the attributes, appearance, situation, etc., of an object at a certain point in time. A state can include one or more values. For example, in a healthcare-related context, a state can include at least some values ​​of age, gender, BMI (Body Mass Index), body fat percentage, blood pressure, calorie intake, and values ​​corresponding to dietary content. A state may include values ​​other than those exemplified above. Furthermore, an action refers to the content or type of behavior performed by an object when the object is in a corresponding state. For example, in a healthcare-related context, an action may include values ​​corresponding to the type and content of the behavior, such as the type, duration, and intensity of exercise. Similar to a state, an action may include values ​​other than those exemplified above. Furthermore, a state-action sequence refers to a transition between a state and an action, including one or more combinations of a state at a certain point in time and an action corresponding to the state. For example, a state-action sequence may be data indicating the transition between a state and an action of a person in a time series. A state-action sequence may indicate the transition between a state and an action of one or more people in a time series.

[0018] In addition, in the present disclosure, a prototype refers to a typical example of a transition between a state and an action learned from a state-action sequence. A prototype can include one or more combinations of a state and an action corresponding to the state. In addition, in the present disclosure, a subgoal selected by the option selection unit 120 based on a state or the like is called an option. For example, an option can be associated with a prototype. Therefore, an option can also be considered as identification information for identifying a prototype. For example, the number of options may be preset to k (k is a natural number).

[0019] FIG. 2 shows an example configuration of the inference device 100. Referring to FIG. 2, the inference device 100 includes an acquisition unit 110, an option selection unit 120, a state-action sequence division unit 130, a feature extraction unit 140, a prototype update unit 150, a prototype acquisition unit 170, a behavior inference unit 180, and an output unit 190. For example, the inference device 100 includes a calculation device such as a central processing unit (CPU) and a recording device that stores a program. For example, the calculation device reads and executes a program from a storage device, thereby enabling the hardware and program to cooperate to realize the processing unit. The inference device 100 may also include a state history database 111, a prototype database 160, and the like.

[0020] 2 illustrates an example in which the functions of the inference device 100 are realized using a single information processing device. However, at least some of the functions of the inference device 100 may be realized using multiple information processing devices, for example, on the cloud. Furthermore, the inference device 100 may have configurations other than those illustrated, such as an operation input unit such as a keyboard or a mouse, or a screen display device such as a display.

[0021] In addition, instead of the above-mentioned CPU, the inference device 100 may have a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), an MPU (Micro Processing Unit), an FPU (Floating point number Processing Unit), a PPU (Physics Processing Unit), a TPU (Tensor Processing Unit), a quantum processor, a microcontroller, or a combination of these.

[0022] The acquisition unit 110 acquires a state-action sequence to be learned, a state to be inferred, etc. For example, the acquisition unit 110 acquires a state-action sequence to be learned, a state to be inferred, etc. by referring to a state history database 111 that stores state-action sequences.

[0023] For example, as illustrated in Fig. 3, the acquiring unit 110 can acquire a state-action sequence including a plurality of states. As an example, in the case illustrated in Fig. 3, the acquiring unit 110 acquires a state-action sequence including 11 states from states s1 to s11. Furthermore, each of states s1 to s11 acquired by the acquiring unit 110 includes a plurality of values ​​such as age, gender, and weight. Although not illustrated in Fig. 3, the state-action sequence may include values ​​of actions corresponding to each state.

[0024] The acquiring unit 110 may acquire the state-action sequence to be learned, the state to be inferred, etc., by a method other than those exemplified above. For example, the acquiring unit 110 may acquire the state-action sequence to be learned, the state to be inferred, etc., by any method, such as accepting input using an operation input unit such as a keyboard or a mouse, or by acquiring from another external device.

[0025] The state history database 111 is a storage device that stores state-action sequences to be learned. For example, the state history database 111 may include one or more state-action sequences. As described above, a state-action sequence may include one or more combinations of a state at a certain point in time and an action corresponding to the state. The state history database 111 may also include states to be inferred. The various pieces of information included in the state history database 111 are acquired by any means, such as accepting input using an operation input unit such as a keyboard or mouse, or by acquiring information from an external device, and are stored in the storage device.

[0026] 2, the state history database 111 may be external to the inference device 100, or may be included in the inference device 100. In the present disclosure, there are no particular limitations on the device that includes the state history database 111.

[0027] The option selection unit 120 selects an option for a state-action sequence to be learned, a state to be inferred, etc. For example, the option selection unit 120 can select an option for each state included in a state-action sequence or a state to be inferred based on the content of the state to be selected, etc.

[0028] 4 shows an example of option selection by the option selection unit 120. For example, referring to FIG. 4, the option selection unit 120 selects an option for each state included in the state-action sequence. As an example, in the example shown in FIG. 4, the option selection unit 120 selects option 3 for states s1 and s2. Furthermore, the option selection unit 120 selects option 1 for states s3 to s7, and option 3 for states s8 to s11.

[0029] In the present disclosure, the option selection unit 120 can select an option by using a model that has been trained in advance to output an option in response to a state input. In other words, the option selection unit 120 may be an option selector that has been trained in advance to output an option in response to a state input. Furthermore, the option selection unit 120's option selection policy may be learned by reinforcement learning or the like. As an example, the option selection unit 120 may perform reinforcement learning such that the closer the inferred value of the behavior by the behavior inference unit 180 (described later) is to the correct value, the higher the reward in policy learning. Furthermore, the option selection unit 120 may learn the policy using a neural network or the like. The option selection unit 120 may also perform learning using a model other than the example shown.

[0030] The state-action sequence division unit 130 divides the state-action sequence into a plurality of segments based on the options selected by the option selection unit 120. Here, a segment refers to a subsequence consisting of at least a part of the state-action sequence. For example, the state-action sequence division unit 130 can divide the state-action sequence into each sequence in which the options selected by the option selection unit 120 are consecutive.

[0031] 4, the option selection unit 120 selects option 3 for states s1 and s2, selects option 1 for states s3 to s7, and selects option 11 for states s8 to s11. Therefore, the state-action sequence division unit 130 divides the state-action sequence into three segments, as shown in FIG. 5, namely, states s1 and s2 where option 3 is selected, states s3 to s7 where option 1 is selected, and states s8 to s11 where option 3 is selected.

[0032] The feature extraction unit 140 extracts the state-action sequence divided by the state-action sequence division unit 130 into successive options as a feature vector in a feature space. In other words, the feature extraction unit 140 extracts a feature vector for each segment divided by the state-action sequence division unit 130 according to the features, such as the states included in the segment. For example, in the example shown in FIG. 5, the state-action sequence is divided into three segments by the state-action sequence division unit 130. Therefore, the feature extraction unit 140 extracts a feature vector for each of the three segments divided by the state-action sequence division unit 130, as shown in the example shown in FIG. 6.

[0033] In the present disclosure, the feature extraction unit 140 extracts feature vectors using an architecture with a mechanism for distributing feature vectors in a feature space. For example, as illustrated in FIG. 7 , the feature extraction unit 140 performs the extraction using an autoencoder architecture including an encoder that inputs a segment that is at least a part of a state-action sequence to obtain a feature vector, and a decoder that inputs the feature vector to reconstruct segment data. In other words, the feature extraction unit 140 can calculate a feature vector in response to the input of a segment. Note that when using the autoencoder architecture, the feature extraction unit 140 performs learning of weight parameters and the like so that the input segment data and the reconstructed segment data are closer to each other. For example, the feature extraction unit 140 performs the above extraction using a loss function L shown in Equation 1. AE The parameters of the encoder and decoder can be learned so as to reduce . In addition, the encoder and decoder may employ, for example, a long short-term memory (LSTM) architecture.

[0034] In addition, in the equation 2 showing the state in FIG. 7, k is a number given to each state-action sequence, and t is a step number. Furthermore, the equation 3 shows the reconstructed state. Furthermore, in the equation 4 showing the feature vector, v ｍ indicates the boundary number of the divided step.

[0035] More preferably, the feature extraction unit 140 uses a variational autoencoder (VAE) as an autoencoder, as exemplified in Fig. 8. When a variational autoencoder is used, the feature extraction unit 140 extracts a mean μ and a variance σ in response to the input of a segment, as exemplified in Fig. 8. 2 By calculating the latent variables by calculating the feature vectors, it is possible to calculate the feature vectors.

[0036] Figure 9 and Equation 5 show the loss function L when using VAE. VAE The first half of Equation 5 is the same as in the case of an autoencoder, and the second half is unique to VAE.

[0037] When using VAE, the standard normal distribution N(0,I) is assumed as the prior distribution of the feature vector shown in Equation 4. For example, the feature vector (Equation 4) when the state shown in Equation 6 is input can be expressed by Equation 7. Here, Equations 8 and 9 are the mean and variance parameters output by the encoder, respectively.

[0038] The feature vector (Equation 4) is sampled from the above distribution, but can be calculated using a technique called the Reparameterization Trick using the following equation (Equation 10): where ε is random noise sampled from ε ~ N(0,I).

[0039] For example, as described above, the feature extraction unit 140 extracts latent variables as feature vectors by using an encoder in an autoencoder, etc. Furthermore, the feature extraction unit 140 can perform learning of weight parameters, etc., so that the data of the input segment and the data of the reconstructed segment become closer to each other.

[0040] The prototype update unit 150 updates the prototype using the feature vector extracted by the feature extraction unit 140. For example, as shown in FIG. 10 , the prototype update unit 150 can update the feature vector of the prototype based on an evaluation result using the feature vector extracted by the feature extraction unit 140 and the feature vector of the prototype before the update. In addition to updating the feature vector of the prototype, the prototype update unit 150 can also update the state-action sequence of the corresponding prototype, as shown in FIG. 11 . The state-action sequence of the prototype may include data indicating states and state-action transitions in time series. For example, FIG. 11 illustrates data indicating states and state-action transitions corresponding to three prototypes in time series. The prototype update unit 150 then stores the state-action sequence, feature vector, and the like of the updated prototype in the prototype database 160.

[0041] As an example, the prototype updating unit 150 calculates an update value from the feature vector extracted by the feature extracting unit 140. Then, the prototype updating unit 150 can update the feature vector of the prototype according to the calculated update value. Furthermore, the prototype updating unit 150 updates the state-action sequence of the prototype based on the updated feature vector. For example, the prototype updating unit 150 can identify the state-action sequence of the segment from which the feature vector was extracted, and update the state-action sequence of the prototype using states included in the identified state-action sequence.

[0042] In addition, when the feature extraction unit 140 uses an autoencoder architecture, the prototype update unit 150 may update the feature vector and state-action sequence of the prototype using a method similar to the method described in Patent Document 1, as shown in the following equations 11 and 12. In addition, k denotes the feature vector of the prototype, and k is the prototype number.

[0043] Furthermore, when the VAE architecture is used, the prototype update unit 150 can update the prototype using the equations shown in FIG. 12 and the following equations 13 and 14. In addition, μk denotes the mean of the prototype feature vectors, and o σk denotes the variance of the prototype feature vectors.

[0044] For example, by using the methods described above, cluster structure can be promoted by training the feature vectors of segments to be closer to the prototypes. Also, by training each prototype to be closer to the features of the nearest segment, the interpretability of the prototype can be improved. Furthermore, by training each prototype to be farther apart when the distance between them is close, diversity can be improved.

[0045] The prototype database 160 is a storage device that stores the feature vectors and state-action sequences of prototypes updated by the prototype update unit 150. In the prototype database 160, the feature vectors and state-action sequences of prototypes may be associated with options. For example, the prototype database 160 is updated in response to the prototype update unit 150 updating the prototypes.

[0046] Prototype database 160 may be included in inference device 100, as illustrated in Figure 2, or may be located outside of inference device 100. In the present disclosure, there are no particular limitations on the device that includes prototype database 160.

[0047] The prototype acquisition unit 170 acquires a prototype corresponding to the state to be inferred, in accordance with the selection result by the option selection unit 120 for the state to be inferred. For example, the prototype acquisition unit 170 acquires a feature vector of a prototype associated with the option selected by the option selection unit 120, by referring to the prototype database 160. The prototype acquisition unit 170 may acquire a state-action sequence of the prototype together with the feature vector of the prototype, by referring to the prototype database 160.

[0048] The behavior inference unit 180 infers behavior based on the state to be inferred and the feature vector of the prototype acquired by the prototype acquisition unit 170 .

[0049] For example, the behavior inference unit 180 may infer behavior by using a model that has been trained in advance to output behavior in response to input of a state to be inferred and a prototype feature vector. The processing by the behavior inference unit 180 may be similar to the technology described in Patent Document 1.

[0050] The output unit 190 outputs the result of the inference by the behavior inference unit 180. For example, as shown in FIG. 13 , the output unit 190 can output the behavior inferred by the behavior inference unit 180, the prototype used for the inference, and the like. The output unit 190 may output the state-action sequence of the prototype as well as the feature vector of the prototype. For example, the output unit 190 may display the inferred behavior on a screen display device or transmit it to an external device, etc. For example, by the output unit 190 outputting the prototype used for the inference along with the result of the inference, it is possible to present the basis for judgment and realize more accurate support for decision-making, etc.

[0051] The above is an example of the configuration of the inference device 100. Next, an example of the operation of the inference device 100 will be described with reference to Figs.

[0052] 14 shows an example of the operation of the inference device 100 when updating a prototype. Referring to FIG. 14, the option selection unit 120 selects an option for a state-action sequence to be learned (step S101). For example, the option selection unit 120 can select an option for each state included in the state-action sequence based on the content of the state to be selected.

[0053] The state-action sequence division unit 130 divides the state-action sequence into a plurality of segments based on the options selected by the option selection unit 120 (step S102). For example, the state-action sequence division unit 130 divides the state-action sequence into each sequence in which the options selected by the option selection unit 120 are consecutive.

[0054] The feature extraction unit 140 extracts the state-action sequence divided by the state-action sequence division unit 130 into consecutive options as feature vectors in a feature space (step S103). In the present disclosure, the feature extraction unit 140 extracts the feature vectors using architecture having a mechanism for distributing the feature vectors in the feature space. For example, the feature extraction unit 140 performs the extraction using an autoencoder architecture that includes an encoder that inputs a segment that is part of the state-action sequence to obtain a feature vector, and a decoder that inputs the feature vector to reconstruct the data of the segment.

[0055] The prototype update unit 150 updates the prototype using the feature vector extracted by the feature extraction unit 140 (step S104). For example, the prototype update unit 150 can update the feature vector of the prototype and also update the state-action sequence of the corresponding prototype.

[0056] Furthermore, the prototype update unit 150 stores the feature vectors and state-action sequences of the updated prototype in the storage device as the prototype database 160 (step S105).

[0057] The above is an example of the operation of inference device 100 when updating a prototype. Next, an example of the operation of inference device 100 when inferring behavior will be described with reference to FIG.

[0058] 15 shows an example of the operation of the inference device 100 during behavior inference. Referring to Fig. 15, in response to acquisition of a state to be inferred by the acquisition unit 110, the option selection unit 120 selects an option for the state to be inferred (step S201).

[0059] The prototype acquisition unit 170 acquires a prototype corresponding to the state to be inferred, in accordance with the selection result by the option selection unit 120 for the state to be inferred (step S202). For example, the prototype acquisition unit 170 refers to the prototype database 160 to acquire a feature vector of a prototype associated with the option selected by the option selection unit 120.

[0060] The behavior inference unit 180 infers behavior based on the state to be inferred and the feature vector of the prototype acquired by the prototype acquisition unit 170 (step S203). For example, the behavior inference unit 180 may infer behavior by using a model that has been trained in advance to output behavior in response to input of the state to be inferred and the feature vector of the prototype.

[0061] The output unit 190 outputs the result of the inference by the behavior inference unit 180 (step S204). For example, the output unit 190 can output the behavior inferred by the behavior inference unit 180, the prototype used for the inference, and the like.

[0062] The above is an example of the operation of the inference device 100 during behavior inference.

[0063] As described above, the inference device 100 includes a feature extraction unit 140 and a prototype update unit 150. With this configuration, the feature extraction unit 140 can extract feature vectors using an architecture with a mechanism for distributing feature vectors in a feature space. As a result, the prototype update unit 150 can update prototypes using the feature vectors extracted by the feature extraction unit 140 in a distributed manner. In other words, with the above configuration, the feature vectors of prototypes to be updated according to the extracted feature vectors can also be distributed. By using the prototypes updated by the prototype update unit 150, it is possible to select an appropriate prototype during inference without selecting only a specific prototype. As a result, more appropriate support can be achieved, for example, when inferring state-dependent behavior through machine learning to support various decision-making tasks.

[0064] For example, Fig. 16 shows an example of feature vectors extracted using the technology described in Patent Document 1, and an example of feature vectors extracted by the feature extraction unit 140 using architecture having a mechanism for distributing feature vectors in a feature space. In the example shown in Fig. 16, the left side is an example of feature space when the technology described in Patent Document 1 is used, and the right side is an example of feature space when feature vectors are extracted using the feature extraction unit 140. Referring to Fig. 16, it can be seen that by extracting feature vectors using the feature extraction unit 140, it is possible to better grasp the characteristics of the state-action sequence, making it less likely that the feature vectors will be biased toward one.

[0065] 17 shows an example of a feature space when VAE is used. Referring to FIG. 17, by applying normalization and restricting the representation using VAE, it is possible to further suppress the bias in distribution in the feature space. This shows that the bias in the feature vectors can be further suppressed.

[0066] Second Embodiment Next, a learning device 200 and an inference device 300 will be described with reference to Fig. 18 to Fig. 21. Fig. 18 is a diagram showing an example of the hardware configuration of the learning device 200. Fig. 19 is a block diagram showing an example of the configuration of the learning device 200. Fig. 20 is a flowchart showing an example of the operation of the learning device 200. Fig. 21 is a block diagram showing an example of the configuration of the inference device 300.

[0067] In a second embodiment of the present disclosure, a learning device 200 is described that learns a prototype that serves as a typical example based on a state-action sequence that indicates transitions between states and actions. The present disclosure also describes an inference device 300 that performs inference using the learning results of the learning device 200. FIG. 18 shows an example hardware configuration of the learning device 200. Referring to FIG. 18, the learning device 200 has, as an example, the following hardware configuration. CPU (Central Processing Unit) 201 (arithmetic device) ROM (Read Only Memory) 202 (storage device) RAM (Random Access Memory) 203 (storage device) Programs 204 loaded into RAM 203 Storage device 205 for storing programs 204 Drive device 206 for reading and writing data from and to a storage medium 210 external to the information processing device Communication interface 207 for connecting to a communication network 211 external to the information processing device Input / output interface 208 for inputting and outputting data Bus 209 for connecting the various components

[0068] 19 by the CPU 201 acquiring and executing the program group 204. The program group 204 is stored in advance in the storage device 205 or the ROM 202, for example, and is loaded into the RAM 203 or the like by the CPU 201 for execution as needed. The program group 204 may be supplied to the CPU 201 via the communication network 211, or may be stored in advance in the recording medium 210, with the drive device 206 reading out the programs and supplying them to the CPU 201.

[0069] 18 shows an example of the hardware configuration of the learning device 200. The hardware configuration of the learning device 200 is not limited to the above. For example, the learning device 200 may be configured with only a part of the above configuration, such as not including the drive device 206. Furthermore, the CPU 201 may be a GPU, as exemplified in the first embodiment.

[0070] The extraction unit 221 extracts a feature vector using a mechanism that distributes feature vectors in a feature space in response to an input of a state-action sequence that includes at least one combination of a state and an action. For example, the extraction unit 221 extracts a feature vector using an autoencoder mechanism that includes an encoder that receives an input of a state-action sequence to obtain a feature vector, and a decoder that receives an input of the feature vector to reconstruct the state-action sequence.

[0071] The update unit 222 updates a prototype, which is a typical example of a state-action sequence, based on the feature vector extracted by the extraction unit 221. The update unit 222 may store the updated result in a storage device or the like.

[0072] The above is an example of the configuration of the learning device 200. Next, an example of the operation of the learning device 200 will be described with reference to FIG.

[0073] 20 shows an example of the operation of the learning device 200. Referring to Fig. 20, the extraction unit 221 extracts a feature vector using a mechanism that distributes feature vectors in a feature space in response to an input of a state-action sequence that includes at least one combination of a state and an action (step S301). For example, the extraction unit 221 extracts the feature vector using an autoencoder mechanism that includes an encoder that receives an input of a state-action sequence to obtain a feature vector, and a decoder that receives an input of the feature vector to reconstruct the state-action sequence.

[0074] The update unit 222 updates the prototype, which is a typical example of the state-action sequence, based on the feature vector extracted by the extraction unit 221 (step S302). The update unit 222 may store the updated result in a storage device or the like.

[0075] As described above, the learning device 200 includes an extraction unit 221 and an update unit 222. With this configuration, the extraction unit 221 can extract feature vectors using a mechanism that distributes feature vectors in a feature space. Furthermore, the update unit 222 can update prototypes, which are typical examples of state-action sequences, based on the feature vectors extracted by the extraction unit 221. This allows for the distribution of prototypes, reducing the risk of selecting only a specific prototype during inference. As a result, more appropriate support can be achieved, for example, when inferring state-dependent behavior through machine learning to support various decision-making processes.

[0076] The above-described learning device 200 can be realized by incorporating a predetermined program into an information processing device such as the learning device 200. Specifically, a program according to another aspect of the present disclosure is a program for implementing a process in an information processing device, in which, in response to input of a state-action sequence including at least one combination of a state and an action, the program extracts a feature vector using a mechanism that distributes feature vectors in a feature space, and updates a prototype, which is a typical example of the state-action sequence, based on the extracted feature vector.

[0077] Furthermore, a learning method executed by an information processing device such as the above-mentioned learning device 200 is a method in which the information processing device extracts feature vectors using a mechanism that distributes feature vectors in a feature space in response to an input of a state-action sequence that includes at least one combination of a state and an action, and updates a prototype, which is a typical example of the state-action sequence, based on the extracted feature vectors.

[0078] Any program having the above-described configuration, or a computer-readable recording medium having the program recorded thereon, or a learning method, etc., can achieve the same functions and effects as the above-described learning device 200, and therefore can achieve the above-described objective of the present disclosure.

[0079] Furthermore, the inference device 300 is an information processing device that performs inference using the prototype learned by the learning device 200. The hardware configuration of the inference device 300 may be similar to that of the learning device 200 exemplified in Fig. 18. Furthermore, the inference device 300 can realize the functions of the acquisition unit 321, the inference unit 322, and the output unit 323 shown in Fig. 21 by having a CPU acquire and execute a group of programs.

[0080] The acquisition unit 321 acquires a prototype trained using a mechanism that distributes feature vectors in a feature space according to the state to be inferred. For example, the prototype may be trained by the above-described learning device 200 or the like.

[0081] The inference unit 322 infers behavior based on the state to be inferred and the prototype acquired by the acquisition unit 321. For example, the inference unit 322 may infer behavior by inputting the state to be inferred and the prototype acquired by the acquisition unit 321 into a pre-trained model.

[0082] The output unit 323 outputs the behavior inferred by the inference unit 322 and the prototype used at the time of inference.

[0083] This inference device 300 can also achieve the above-mentioned object of the present disclosure, similar to the learning device 200 and the like.

[0084] <Supplementary Notes> Part or all of the above-described embodiments can be described as follows: Below, an overview of the learning device and the like in the present disclosure will be described. However, the present disclosure is not limited to the following configuration.

[0085] (Supplementary Note 1) A learning device comprising: an extraction unit that extracts a feature vector using a mechanism that distributes a feature vector in a feature space in response to an input of a state-action sequence including at least one combination of a state and an action; and an update unit that updates a prototype that is a typical example of the state-action sequence based on the feature vector extracted by the extraction unit. (Supplementary Note 2) The learning device according to Supplementary Note 1, wherein the extraction unit has an autoencoder mechanism including an encoder that inputs the state-action sequence to obtain the feature vector, and a decoder that inputs the feature vector to reconstruct the state-action sequence. (Supplementary Note 3) The learning device according to Supplementary Note 2, wherein the extraction unit extracts the feature vector from the state-action sequence by calculating a latent variable in response to the input of the state-action sequence. (Supplementary Note 4) The learning device according to Supplementary Note 2, wherein the extraction unit uses a variational autoencoder (VAE) as the autoencoder. (Supplementary Note 5) In the learning device according to Supplementary Note 4, the update unit updates the prototype by solving equations shown in Equations 15 and 16. A learning device. (Supplementary Note 6) The learning device according to Supplementary Note 1, wherein the prototype is data used when inferring an action using a state to be inferred. (Supplementary Note 7) The learning device according to Supplementary Note 1, comprising: a selection unit that selects an option to be a subgoal for each state included in the state-action sequence; and a division unit that divides the state-action sequence into one or more segments based on the option selected by the selection unit, wherein the extraction unit extracts the feature vector in response to input of the segment. (Supplementary Note 8) The learning device according to Supplementary Note 7, wherein the division unit divides the state-action sequence for each sequence in which the option selected by the selection unit is consecutive. (Supplementary Note 9) The learning device according to Supplementary Note 1, comprising: a selection unit that selects an option that will be a subgoal for a state to be inferred, an acquisition unit that acquires the prototype based on the option selected by the selection unit, an inference unit that infers an action based on the state to be inferred and the prototype acquired by the acquisition unit, and an output unit that outputs the action inferred by the inference unit and the prototype used during the inference. (Supplementary Note 10) The learning device according to Supplementary Note 1, comprising: a selection unit that selects an option that will be a subgoal for the state to be inferred, an acquisition unit that acquires the prototype based on the option selected by the selection unit, an inference unit that infers an action for user decision-making based on the state to be inferred and the prototype acquired by the acquisition unit, and an output unit that outputs the action inferred by the inference unit and the prototype used during the inference.(Supplementary Note 11) The learning device according to Supplementary Note 2, wherein the extraction unit uses a machine-learned variational autoencoder (VAE) as an autoencoder, and comprises: a selection unit that selects an option that will be a subgoal for a state to be inferred, an acquisition unit that acquires the prototype based on the option selected by the selection unit, an inference unit that infers an action for user decision-making based on the state to be inferred and the prototype acquired by the acquisition unit, and an output unit that outputs the action inferred by the inference unit and the prototype used for inference. (Supplementary Note 12) A learning method, wherein an information processing device extracts feature vectors using a mechanism that distributes feature vectors in a feature space in response to input of a state-action sequence including at least one combination of a state and an action, and updates a prototype that is a typical example of the state-action sequence based on the extracted feature vector. (Supplementary Note 12-1) The learning method according to Supplementary Note 12, wherein, when extracting the feature vector, the feature vector is extracted using an autoencoder mechanism including an encoder that inputs the state-action sequence to obtain the feature vector, and a decoder that inputs the feature vector to reconstruct the state-action sequence. (Supplementary Note 12-2) The learning method according to Supplementary Note 12-1, wherein, when extracting the feature vector, the feature vector is extracted using a variational autoencoder (VAE) as the autoencoder. (Supplementary Note 12-3) The learning method according to Supplementary Note 12-2, wherein, when extracting the feature vector, the feature vector is extracted from the state-action sequence by calculating a latent variable according to the input of the state-action sequence. (Supplementary Note 12-4) The learning method according to Supplementary Note 12-2, wherein, when extracting the feature vector, the prototype is updated by solving the equations shown in Equations 17 and 18. A learning method. (Supplementary Note 12-5) The learning method according to Supplementary Note 12, wherein the prototype is data used when inferring behavior using a state to be inferred. (Supplementary Note 12-6) The learning method according to Supplementary Note 12, comprising: selecting an option to be a subgoal for each state included in the state-action sequence; dividing the state-action sequence into one or more segments based on the selected option; and extracting the feature vector according to input of the segment. (Supplementary Note 12-7) The learning method according to Supplementary Note 12-6, wherein when dividing the state-action sequence, the state-action sequence is divided into each sequence in which the selected option is consecutive. (Supplementary Note 12-8) The learning method according to Supplementary Note 12, comprising: selecting an option to be a subgoal for a state to be inferred; acquiring the prototype based on the selected option; inferring behavior based on the state to be inferred and the acquired prototype; and outputting the inferred behavior and the prototype used for inference. (Supplementary Note 13) A computer-readable recording medium having recorded thereon a program for implementing the following processing in an information processing device: extracting a feature vector using a mechanism that distributes a feature vector in a feature space in response to input of a state-action sequence including at least one combination of a state and an action, and updating a prototype that is a typical example of the state-action sequence based on the extracted feature vector. (Supplementary Note 13-1) The recording medium according to Supplementary Note 13, wherein, when extracting the feature vector, the feature vector is extracted using an autoencoder mechanism including an encoder that inputs the state-action sequence to obtain the feature vector, and a decoder that inputs the feature vector to reconstruct the state-action sequence. (Supplementary Note 13-2) The recording medium according to Supplementary Note 13-1, wherein, when extracting the feature vector, the feature vector is extracted using a variational autoencoder (VAE) as the autoencoder.(Supplementary Note 13-3) The recording medium according to Supplementary Note 13-2, wherein when extracting the feature vector, the feature vector is extracted from the state-action sequence by calculating a latent variable according to the input of the state-action sequence. (Supplementary Note 13-4) The recording medium according to Supplementary Note 13-2, wherein the prototype is updated by solving the equations shown in Equations 19 and 20. a computer-readable recording medium having recorded thereon a program for implementing the processes of: selecting an option to be a subgoal for each state included in the state-action sequence; dividing the state-action sequence into one or more segments based on the selected option; and extracting the feature vector according to the input of the segment. (Supplementary Note 13-7) a computer-readable recording medium having recorded thereon a program for implementing the processes of: selecting an option to be a subgoal for a state to be inferred; acquiring the prototype based on the selected option; inferring an action based on the state to be inferred and the acquired prototype; and outputting the inferred action and the prototype used during the inference. (Supplementary Note 14) An inference device comprising: an acquisition unit that acquires a learned prototype using a mechanism that distributes feature vectors in a feature space according to a state to be inferred; an inference unit that infers an action based on the state to be inferred and the prototype acquired by the acquisition unit; and an output unit that outputs the action inferred by the inference unit and the prototype used during the inference.

[0086] The programs described in the above embodiments and appendices may be stored in a storage device or a computer-readable recording medium, such as a portable medium such as a flexible disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0087] Although the present disclosure has been described above with reference to the above-described embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the present disclosure. Furthermore, each embodiment can be combined with other embodiments as appropriate.

[0088] REFERENCE SIGNS LIST 100 Inference device 110 Acquisition unit 111 State history database 120 Option selection unit 130 State-action sequence division unit 140 Feature extraction unit 150 Prototype update unit 160 Prototype database 170 Prototype acquisition unit 180 Action inference unit 190 Output unit 200 Learning device 201 CPU 202 ROM 203 RAM 204 Program group 205 Storage device 206 Drive device 207 Communication interface 208 Input / output interface 209 Bus 210 Recording medium 211 Communication network 221 Extraction unit 222 Update unit 300 Inference device 321 Acquisition unit 322 Inference unit 323 Output unit

Claims

1. an extraction unit that extracts a feature vector using a mechanism that distributes the feature vector in a feature space in response to an input of a state-action sequence that includes at least one combination of a state and an action; an update unit that updates a prototype, which is a typical example of the state-action sequence, based on the feature vector extracted by the extraction unit; have Learning device.

2. The learning device according to claim 1 , The extraction unit has an autoencoder mechanism including an encoder that inputs the state-action sequence to obtain the feature vector, and a decoder that inputs the feature vector to reconstruct the state-action sequence. Learning device.

3. The learning device according to claim 2, The extraction unit uses a variational autoencoder (VAE) as an autoencoder. Learning device.

4. The learning device according to claim 3, The extraction unit extracts the feature vector from the state-action sequence by calculating a mean and a variance in response to the input of the state-action sequence and calculating a latent variable. Learning device.

5. The learning device according to claim 1 , The prototype is data used when inferring behavior using a state that is the subject of inference. Learning device.

6. The learning device according to claim 1 , a selection unit that selects an option that will be a subgoal for each state included in the state-action sequence; a dividing unit that divides the state-action sequence into one or more segments based on the option selected by the selecting unit; and The extraction unit extracts the feature vector in response to the input of the segment. Learning device.

7. The learning device according to claim 1 , a selection unit that selects an option that will be a subgoal for a state that is an inference target; an acquisition unit that acquires the prototype based on the option selected by the selection unit; an inference unit that infers an action based on a state to be inferred and the prototype acquired by the acquisition unit; an output unit that outputs the behavior inferred by the inference unit and the prototype used at the time of inference; have Learning device.

8. The information processing device extracting a feature vector using a mechanism that distributes the feature vector in a feature space in response to an input of a state-action sequence including at least one combination of a state and an action; Based on the extracted feature vector, a prototype that is a typical example of the state-action sequence is updated. How to learn.

9. In the information processing device, extracting a feature vector using a mechanism that distributes the feature vector in a feature space in response to an input of a state-action sequence including at least one combination of a state and an action; Based on the extracted feature vector, a prototype that is a typical example of the state-action sequence is updated. A program to realize the processing.

10. an acquisition unit that acquires trained prototypes using a mechanism that distributes feature vectors in a feature space according to the state to be inferred; an inference unit that infers an action based on a state to be inferred and the prototype acquired by the acquisition unit; an output unit that outputs the behavior inferred by the inference unit and the prototype used at the time of inference; have Reasoning device.