Information collection device, information providing device, information collection method, and information providing method

Abstract

An information collection device according to one aspect of the present invention: acquires time series data that indicates a state transition of at least one agent; uses an encoder configured to perform encoding in such a manner as to incorporate information pertaining to a plurality of time points to convert the acquired time series data into a feature vector; and stores, in a database, the obtained feature vector in association with the time series data. An information providing device according to the present invention: receives a query from a request source; extracts, from the database, a feature vector matching the query as a related feature vector; and returns, as related time series data, time series data associated with the extracted related feature vector to the request source.

Description

Information collection device, information provision device, information collection method, and information provision method

[0001] The present invention relates to an information collection device, an information providing device, an information collection method, and an information providing method.

[0002] In recent years, various methods have been developed for generating control models of robotic devices using machine learning. Such machine learning methods sometimes use data collected in advance. For example, in imitation learning, policies are learned using demonstration data (teaching data) collected in advance. Furthermore, in reinforcement learning, data collected in advance may be used as episodes. In addition, in supervised learning and unsupervised learning, models are learned using data collected in advance. However, collecting such data individually for each task is costly.

[0003] Therefore, methods have been studied that select suitable data from data collected in advance on a large scale and use the selected data for machine learning. For example, Non-Patent Document 1 proposes a method that uses data obtained by image search in offline reinforcement learning. This method increases the reusability of data, and is expected to reduce the cost of collecting new data for a target environment or target task.

[0004] Peter C. Humphreys et al., “Large-Scale Retrieval for Reinforcement Learning”, [online], [Retrieved November 17, 2023], Internet <URL: https: / / arxiv.org / abs / 2206.05314>

[0005] The present inventors have found that the above-described conventional method has the following problem. Specifically, behavioral sequences are taken into consideration when controlling the motion of a robot device. In contrast, conventional methods search image data at a single time. Therefore, only visual similarity is referenced in the search, making it difficult to perform a search that takes behavioral sequences into consideration. It is possible to generate time-series data corresponding to a behavioral sequence by repeatedly performing searches and piecing together the obtained data in a time series. However, the generated time-series data may include data with low correlation, which may result in poor accuracy in extracting time-series data corresponding to a behavioral sequence.

[0006] Therefore, with conventional methods, it has been difficult to accurately extract time-series data corresponding to the behavioral sequence of a robot device. This problem does not occur only when searching for data to be used in machine learning. It can occur in any situation where time-series data is searched. For example, this problem can occur even when simply searching for data, such as searching for actions similar to a desired action of a robot device.

[0007] In one aspect, the present invention has been made in view of the above circumstances, and an object of the present invention is to provide a technique for improving the accuracy of extracting time-series data.

[0008] In order to solve the above-mentioned problems, the present invention employs the following configurations. Note that the following configurations of the invention can be combined as appropriate.

[0009] An information collection device according to one aspect of the present invention includes a control unit configured to acquire time-series data indicating state transitions of one or more agents, convert the acquired time-series data into a feature vector using an encoder configured to encode the data by incorporating information at multiple times, and store the acquired feature vector in a database in association with the time-series data.

[0010] In this configuration, an encoder incorporates information about multiple times contained in the time series data into a feature vector. The generated feature vector is then stored in association with the time series data. By using this feature vector as meta-information (tag) in a search, the state transition sequence can be taken into account in the search, which is expected to enable the extraction of appropriate time series data for the sequence being searched. Therefore, this configuration is expected to improve the accuracy of extracting time series data indicating the state transitions of one or more agents.

[0011] In the information collection device according to the above aspect, the time-series data may be configured to indicate state transitions of the multiple agents, and the encoder may be configured to encode the data by further incorporating spatial information. In the conventional method described above, image data of a single agent is the search target. Therefore, the search only references the visual similarity of the entire environment, making it difficult to perform a search that takes into account the behavioral sequences of multiple agents. It is possible to generate time-series data corresponding to a behavioral sequence by repeatedly searching image data of a single agent and piecing together the obtained data in a time series. However, since the generated time-series data may include data with low correlation, the accuracy of extracting time-series data corresponding to a behavioral sequence is likely to be poor, making it difficult to apply to time-series data of multiple agents. In contrast, the configuration described above enables efficient extraction of time-series data of multiple agents.

[0012] In the information collection device according to the above aspect, the encoder may be configured to be trained to predict a future state of one or more agents based on their state transitions, thereby incorporating information from multiple time points into the encoding. Because a future state is related to states at multiple previous times, an encoder trained in this manner can be expected to have properly acquired the ability to incorporate information from multiple time points into a feature vector. Therefore, according to this configuration, by using such an encoder, time series data can be properly encoded, and by using the resulting feature vector, it is expected that the accuracy of extracting time series data can be improved.

[0013] In the information collection device according to the above aspect, the time-series data may include trajectory data. With this configuration, improvement in extraction accuracy can be expected when searching for trajectory data.

[0014] In the information collection device according to the above aspect, the time-series data may include video data. With this configuration, improvement in extraction accuracy can be expected when searching for video data.

[0015] In the information collection device according to the above aspect, the control unit may be configured to further acquire attribute information of the one or more agents. Converting the acquired time series data into a feature vector may be configured by converting the acquired time series data and the attribute information of the one or more agents into a feature vector. With this configuration, the attribute information of the agents can be further incorporated into the feature vector. This makes it possible to search for data that further takes agent attributes into consideration, which is expected to further improve the accuracy of extracting time series data.

[0016] In the information collection device according to the above aspect, the time-series data may be configured to further indicate state transitions of one or more objects, and the encoder may be configured to encode the data by further incorporating spatial information. This configuration allows the state transitions of the objects (information at multiple times) to be further incorporated into the feature vector. This allows for data searches that further take into account the state transitions of the objects, which is expected to further improve the accuracy of extracting time-series data.

[0017] In the information collection device according to the above aspect, the control unit may be configured to further acquire attribute information of the one or more objects. Converting the acquired time series data into a feature vector may be configured by converting the acquired time series data and the attribute information of the one or more objects into a feature vector. This configuration allows the attribute information of the objects to be further incorporated into the feature vector. This enables data searches that further take into account the attributes of the objects, which is expected to further improve the accuracy of extracting time series data.

[0018] In the information collection device according to the above aspect, the control unit may be configured to further acquire environmental information related to an environment in which the one or more agents exist. Converting the acquired time series data into a feature vector may be configured by converting the acquired time series data and the environmental information into a feature vector. With this configuration, environmental information can be further incorporated into the feature vector. This allows for data search that further takes into account the environment in which the agent exists, which is expected to further improve the accuracy of extracting time series data.

[0019] In the information collection device according to the above aspect, the agent may be a robotic device or a living organism. This configuration can be expected to improve extraction accuracy when searching for time-series data of a robotic device or a living organism. Note that the agent (target agent) in the search query does not necessarily have to match the agent in the extracted time-series data. The target agent in the search query may at least partially match the agent in the extracted time-series data, or may be different.

[0020] Furthermore, the present invention is not limited to an information processing device for collecting information. One aspect of the present invention may be an information processing device that provides collected time-series data. The time-series data may be provided for any purpose, such as use in machine learning or reference to similar behavior.

[0021] For example, an information providing device according to one aspect of the present invention is connected to a database that stores feature vectors associated with time series data and includes a control unit. The time series data is configured to indicate state transitions of one or more agents. The feature vectors are obtained by converting the time series data using an encoder configured to encode information at multiple times. The control unit is configured to receive a query for the database from a requester, extract feature vectors that match the query from the database as related feature vectors, and return time series data associated with the extracted related feature vectors to the requester as related time series data. This configuration can be expected to enable highly accurate extraction of time series data that match a target series and provide suitable time series data.

[0022] In the information providing device according to the above aspect, accepting the query may be configured by accepting a plurality of the queries, and extracting the related feature vectors may be configured by extracting a plurality of related feature vectors that match the plurality of queries, respectively. For the plurality of extracted related feature vectors, the information source of the time-series data associated with some of the related feature vectors may be allowed to differ from the information source of the time-series data associated with the other related feature vectors. This configuration is expected to improve the reusability of time-series data.

[0023] In the information providing device according to the above aspect, for the extracted feature vector, the types of the one or more agents in the related time-series data may be allowed to differ from the types of one or more target agents in the task targeted by the query. With this configuration, it is expected that the reusability of time-series data can be improved.

[0024] In the information providing device according to the above aspect, the query may be generated by encoding time-series data of a target by the encoder. The scale of the time-series data of the target may be changed by the requester. This configuration can be expected to increase the search precision and improve convenience.

[0025] In the information providing device according to the above aspect, receiving the query from the requester may include receiving time-series data of a target from the requester, changing a scale of the time-series data of the target, and encoding the changed time-series data of the target by the encoder to generate the query. This configuration can be expected to improve search precision and convenience.

[0026] In the information providing device according to the above aspect, returning the time-series data to the requester as related time-series data may include changing a scale of the related time-series data and returning the changed related time-series data to the requester. With this configuration, improved convenience can be expected.

[0027] The present invention is not limited to the information processing device (information collection device, information providing device) described above. As another aspect of the information processing device according to each of the above aspects, one aspect of the present invention may be an information processing method that realizes all or part of each of the above configurations, a program, or a storage medium that stores such a program and is readable by a machine such as a computer. A storage medium that is readable by a machine such as a computer is a medium that stores information such as a program by electrical, magnetic, optical, mechanical, or chemical action.

[0028] For example, an information collection method according to one aspect of the present invention may be an information processing method in which a computer acquires time series data indicating state transitions of one or more agents, converts the acquired time series data into a feature vector using an encoder configured to encode the data by incorporating information from multiple times, and stores the acquired feature vector in a database in association with the time series data.

[0029] Furthermore, for example, an information providing method according to one aspect of the present invention may be an information processing method by a computer connected to a database that stores feature vectors associated with time-series data, wherein the computer may execute the following steps: receive a query to the database from a requester; extract feature vectors that match the query from the database as related feature vectors; and return time-series data associated with the extracted related feature vectors to the requester as related time-series data.

[0030] According to the present invention, it is possible to improve the accuracy of extracting time-series data.

[0031] FIG. 1 schematically shows an example of a situation in which the present invention is applied. FIG. 2 schematically shows an example of the configuration of an encoder according to an embodiment. FIG. 3 schematically shows an example of the configuration of time-series data according to an embodiment. FIG. 4 schematically shows an example of input / output of an encoder according to an embodiment. FIG. 5 schematically shows an example of the hardware configuration of an information collection device according to an embodiment. FIG. 6 schematically shows an example of the hardware configuration of an information provision device according to an embodiment. FIG. 7 schematically shows an example of the software configuration of an information collection device according to an embodiment. FIG. 8 schematically shows an example of the software configuration of an information provision device according to an embodiment. FIG. 9 is a flowchart showing an example of the processing procedure of an information collection device according to an embodiment. FIG. 10 is a flowchart showing an example of the processing procedure of an information provision device according to an embodiment. FIG. 11 shows experimental results of examples and comparative examples.

[0032] An embodiment according to one aspect of the present invention (hereinafter also referred to as "the present embodiment") will be described below with reference to the drawings. However, the present embodiment described below is merely an example of the present invention in all respects. Various improvements or modifications may be made without departing from the scope of the present invention. In implementing the present invention, a specific configuration according to the embodiment may be appropriately adopted. Note that while data appearing in the present embodiment is described in natural language, more specifically, it is specified using computer-recognizable pseudo-language, commands, parameters, machine language, etc.

[0033] §1 Application Example Fig. 1 schematically shows an example of a situation in which the present invention is applied. The system according to this embodiment includes an information collection device 1, a database R, and an information provision device 2. The information collection device 1 is one or more computers configured to collect time-series data 30. The information provision device 2 is one or more computers configured to provide related time-series data 60 (time-series data 30) in response to a query S from a request source U.

[0034] Specifically, the information collection device 1 according to this embodiment acquires time-series data 30 indicating state transitions of one or more agents K. The information collection device 1 converts (i.e., encodes) the acquired time-series data 30 into a feature vector 35 using an encoder 5. The encoder 5 is configured to encode the data by incorporating information from multiple times. The information collection device 1 stores the acquired feature vector 35 in a database R in association with the time-series data 30.

[0035] On the other hand, the information providing device 2 according to this embodiment is connected to a database R that stores feature vectors 35 associated with time-series data 30. The information providing device 2 receives a query S for the database R from a requester U. The information providing device 2 extracts feature vectors 35 that match the query S from the database R as associated feature vectors 65. The information providing device 2 returns the time-series data 30 associated with the extracted associated feature vectors 65 to the requester U as associated time-series data 60.

[0036] As described above, in this embodiment, the encoder 5 incorporates information on multiple times included in the time-series data 30 into the feature vector 35. The feature vector 35 obtained by the conversion is stored by the information collection device 1 in association with the time-series data 30. The information providing device 2 uses this feature vector 35 as meta-information (tag) in a search, allowing the search to take into account the state transition sequence, which is expected to enable extraction of time-series data 30 appropriate for the sequence being searched for. Therefore, the information collection device 1 according to this embodiment is expected to improve the accuracy of extracting time-series data 30 indicating state transitions of one or more agents K. The information providing device 2 according to this embodiment is expected to enable highly accurate extraction of such time-series data 30 and provision of appropriate time-series data 30.

[0037] [Agent] Agent K is a target whose state transitions are recorded as time-series data. The type of agent K is not particularly limited and may be selected appropriately depending on the embodiment. Agent K may be any object (object) whose state transitions can be tracked. In one example, agent K may be a robotic device or a living organism. The type of robotic device is not particularly limited and may be selected appropriately depending on the embodiment. The robotic device may be, for example, an industrial robot used in a production line, an autonomous robot configured to operate autonomously, or a mobile object configured to move. The industrial robot may be, for example, a vertical articulated robot, a horizontal articulated robot (SCARA robot), a parallel link robot, an orthogonal robot, or the like. The autonomous robot may be, for example, a humanoid robot, a guide robot, an agricultural robot, a nursing robot, a security robot, a transport robot, or the like. The content of the autonomous processing may be selected appropriately depending on the embodiment. The mobile object may be, for example, a cleaning robot, the above-mentioned autonomous robot configured to move (including a mobile robot), a vehicle configured to be self-driving, or an air vehicle capable of self-flying (such as a drone), or the like. The living organism may be a human or a non-human organism. The type of living thing may be selected appropriately depending on the embodiment. The agent K may be, for example, a flock-forming living thing such as a bird, a fish, or a sheep. According to one example of the present embodiment, improvement in extraction accuracy can be expected when searching for time-series data of a robot device or a living thing (time-series data 30). The agent K may be an entity in real space or virtual space.

[0038] The number of agents K is not particularly limited and may be selected appropriately depending on the embodiment. The time-series data 30 may be configured to show state transitions of multiple agents K. The conventional method described above searches image data of a single agent, making it difficult to apply this to searching for multiple agents. In contrast, according to an example of this embodiment, it is possible to efficiently extract time-series data 30 of multiple agents K. Note that when multiple agents K appear in the time-series data 30, the type of each agent K may be selected appropriately depending on the embodiment. The types of all agents K may be the same, or may be at least partially different. Furthermore, the time-series data 30 may be configured to show state transitions of a single agent K. The number of agents K may be the same among collected time-series data 30, or may be at least partially different.

[0039] [Encoder] The encoder 5 is configured to perform encoding so as to incorporate information on multiple times in the time series data. Incorporating information on multiple times means incorporating (e.g., integrating, aggregating, etc.) information on multiple times into a single vector so as to reflect the correlation between the times.

[0040] When multiple objects appear in the time series data, such as when the time series data 30 is configured to indicate state transitions of multiple agents K, the encoder 5 may be configured to further incorporate spatial information into the encoding. Incorporating spatial information means incorporating information about multiple objects existing in space into a single vector so as to reflect the relationships between the objects. The spatial information may include any information related to the state of an object in real space or virtual space. In one example, the spatial information may include position information of the object (e.g., coordinate values, image information, etc.). The spatial information may further include information other than position, such as the speed and attitude (tilt) of the object.

[0041] As long as the encoder 5 can execute a computation process incorporating the above-described information, the configuration of the encoder 5 is not particularly limited and may be determined appropriately depending on the embodiment. In a typical example, the encoder 5 may be configured using a trained machine learning model. The machine learning model is configured to have one or more computation parameters that can be adjusted by machine learning. The one or more computation parameters are used in computing the desired inference. In this embodiment, the desired inference is the conversion of time-series data into a feature vector. The machine learning model may be configured using, for example, a neural network, a regression model, a decision tree model, a support vector machine, or other functional formulas (computation models).

[0042] In one example, the encoder 5 may include a neural network. The structure of the neural network is not particularly limited and may be determined appropriately depending on the embodiment. The structure of the neural network may be specified, for example, by the number of layers from the input layer to the output layer, the type of each layer, the number of nodes (neurons) included in each layer, the connection relationships between the nodes in each layer, etc. In one example, the neural network may include any mechanism such as a recurrent structure, a self-attention mechanism, or a cross-attention mechanism. Furthermore, the neural network may include any layer such as a fully connected layer, a convolutional layer, a pooling layer, a deconvolutional layer, an unpooling layer, a normalization layer, a dropout layer, a long short-term memory (LSTM), or a gated recurrent unit (GRU). The neural network may include any type of model, such as a diffusion model, a transformer model, or a generative model. The connection weights between each node included in the neural network and the threshold value of each node are examples of calculation parameters.

[0043] The machine learning method may be appropriately selected depending on, for example, the type, structure, and other aspects of the machine learning model employed (e.g., backpropagation). Machine learning involves adjusting (optimizing) values ​​of computational parameters using training samples. Typically, multiple data sets may be collected, each consisting of a combination of input samples (training samples) and true values ​​(teacher signals, labels). In machine learning, the values ​​of computational parameters of the machine learning model may be adjusted so that the output obtained from the machine learning model when an input sample is provided conforms to the corresponding true value. In machine learning, the values ​​of computational parameters of the entire machine learning model constituting the encoder 5 may be adjusted, or the values ​​of some of the computational parameters may be adjusted. Furthermore, the machine learning model constituting the encoder 5 may be trained together with other modules, such as in the machine learning of an autoencoder.

[0044] In one example, the encoder 5 may be configured to be trained to predict a future state from the state transitions (time series data) of one or more agents K, thereby encoding the data by incorporating information from multiple times. In this case, the feature vector 35 obtained by the encoder 5 is a prediction result (predicted value) of the future state of one or more agents K. The number of time steps (times) of the state transitions in the input data is not particularly limited and may be determined appropriately depending on the embodiment. The number of time steps of the state transitions in the input data may be fixed or variable. Furthermore, the future time steps (times) to be predicted are also not particularly limited and may be determined appropriately depending on the embodiment. The number of future time steps to be predicted may be one or more. In other words, the encoder 5 may be trained to predict the state of an object at one or more future times.

[0045] For example, the number of time steps of a state transition in the input data may be given by n, the number of time steps between the last (latest) state in the input data and a future state may be given by k, and the number of future time steps to be predicted may be 1. In this case, the encoder 5 is trained to predict the state at time t+k from the states (time series data) from time t-n+1 to time t. n may be 2 or more, and k may be 1 or more. The values ​​of n and k may be determined appropriately depending on the embodiment. The interval between the time steps corresponds to the sampling period. The interval between the time steps may be determined appropriately depending on the embodiment.

[0046] When performing this training, time series data including state transitions for n+k or more time steps may be collected. The time series data used for machine learning may be collected by any method. For example, the time series data may be collected by manual demonstration, manually designed control, automatic control using a policy obtained by machine learning (reinforcement learning), passive data recording, mixing of collected data, data augmentation, etc. Manually designed control may include the use of any control algorithm. Control using a policy obtained by machine learning may be performed for any task. Passive data recording may include, for example, recording of user interactions, recording of autonomous control of a robotic device (e.g., autonomous vehicle driving), etc. Data recording may be performed by, for example, taking images (video recording). Data mixing may be performed between data collected by the same method or between data collected by different methods. In one example, time series data may be obtained by generating multiple policies through machine learning and mixing data obtained from demonstrations of each policy. In another example, multiple policies obtained by machine learning may be mixed, and time-series data may be obtained from demonstrations performed using the resulting policies. The multiple policies preferably include policies with different characteristics (i.e., policies that select different actions) due to differences in training algorithms, differences in proficiency, etc. Demonstration and control may be performed in a real space or in a virtual space, such as a simulator. In another example, data may be expanded by changing the scale (e.g., time scale, space scale, etc.) of the acquired data, thereby increasing the number of data items. The scale change may include at least one of scaling up (e.g., expansion, enlargement) and scaling down (e.g., contraction, reduction). The scale change may include applying different modification processes to different parts, such as scaling up a portion and scaling down the rest. Other known methods may be used to expand the data.

[0047] Of the obtained time series data, data for n steps may be obtained as training samples, and data k steps from the training samples may be obtained as true values. This allows a dataset consisting of training samples and true values ​​to be obtained. One or more datasets may be obtained from one sample of time series data. By repeatedly obtaining training samples and true values, multiple datasets can be collected. By training using the above method using the multiple collected datasets, an encoder 5 (trained machine learning model) can be generated.

[0048] A future state is related to states at multiple previous times. Therefore, the encoder 5 trained as described above can be expected to have properly acquired the ability to incorporate information from multiple times into a feature vector. Therefore, according to one example of the present embodiment, the encoder 5 trained in this manner can be used to properly encode the time series data 30. Furthermore, by using the obtained feature vector 35, it is expected that the accuracy of extracting the time series data 30 can be improved. Note that, as long as the encoder 5 can acquire the ability to incorporate information from multiple times, the method for obtaining the encoder 5 is not limited to this example and may be modified as appropriate depending on the embodiment. In another example, the encoder 5 may be trained to restore the input of the current state (time t) rather than predicting a future state.

[0049] (Example of Encoder Configuration) FIG. 2 schematically illustrates an example of the configuration of the encoder 5 according to this embodiment. The example in FIG. 2 assumes a scenario in which multiple agents K exist, and an encoder 5 trained to predict the above-described future states is used to incorporate information from multiple times and spaces. Specifically, the encoder 5 is trained to predict a future state (time t+1) from the states from time t-n+1 to time t. k=1. The feature vector 35 obtained by the encoder 5 is a predicted value of the state of each agent K at time t+1, calculated from the time-series data of each agent K from time t-n+1 to time t. Training the encoder 5 may use multiple data sets, each composed of a combination of training samples including state transitions from time t-n+1 to time t and true values ​​45 of the states at time t+1. The value of k may be changed as appropriate to a value other than 1.

[0050] In the example of FIG. 2 , in order to incorporate information from multiple times and spaces, the encoder 5 includes a self-attention mechanism 51 and a cross-attention mechanism 53. Information about each agent K at each time is input to the self-attention mechanism 51. The self-attention mechanism 51 is configured to calculate the degree of relevance between the agents K at each time. The calculation results of the self-attention mechanism 51 are input to the cross-attention mechanism 53. In addition to the calculation results of the self-attention mechanism 51, information about each agent K at time t is input to the cross-attention mechanism 53. The cross-attention mechanism 53 is configured to predict the state of each agent K at time t+1 from the state at time t, reflecting the calculation results of the self-attention mechanism 51. The specific configurations of the self-attention mechanism 51 and the cross-attention mechanism 53 may be, for example, publicly known configurations proposed in references (Ashish Vaswani et al., “Attention Is All You Need”, [online], [searched November 17, 2023], Internet <URL: https: / / arxiv.org / abs / 1706.03762>).

[0051] The self-attention mechanism 51 incorporates spatial information through its calculation, and the cross-attention mechanism 53 incorporates multiple pieces of time information through its calculation. Therefore, according to the structure of FIG. 2 , the encoder 5 can incorporate multiple pieces of time and space information for encoding. Note that the input / output form of at least one of the self-attention mechanism 51 and the cross-attention mechanism 53 is not limited to the example of FIG. 2 and may be changed as appropriate depending on the embodiment. Furthermore, the structure for incorporating multiple pieces of time and space information is not limited to the structure of FIG. 2 and may be changed as appropriate depending on the embodiment. In another example, the encoder 5 may be configured to incorporate multiple pieces of time and space information for encoding by having a recurrent structure, an RNN (Recurrent Neural Network) such as an LSTM or GRU, or a structure similar thereto.

[0052] [Time Series Data] The time series data 30 is configured to indicate the state transitions of agent K by arranging information about the state of agent K in chronological order. That is, the time series data 30 includes information about the state of agent K at multiple times. The state information may be any information defined in space, such as position, speed, or posture (tilt). The state transitions of agent K may correspond to the behavioral sequence of agent K. As with the time series data samples used for machine learning, the time series data 30 may be acquired by any method. The time series data 30 may be acquired by, for example, manual demonstration, manually designed control, control using a strategy obtained by machine learning, passive data recording, mixing of collected data, data expansion, or other methods. Furthermore, things other than agent K, such as objects, may exist in the space where agent K exists. Therefore, the time series data 30 may further include any time series information related to things other than agent K.

[0053] FIG. 3 schematically illustrates an example of the configuration of time-series data 30 according to this embodiment. As illustrated in FIG. 3, the time-series data 30 may be configured to further indicate state transitions of one or more objects L. Accordingly, the encoder 5 may be configured to encode the data by further incorporating spatial information. According to this example of the present embodiment, the state transitions of the object L can be further incorporated into the feature vector 35. This allows for data search that further takes into account the state transitions of the object L, which is expected to further improve the accuracy of extracting the time-series data 30.

[0054] In the example of FIG. 3 , the time-series data 30 includes information 301 about the states of one or more agents K and information 303 about the states of one or more objects L at each of a plurality of times, thereby indicating state transitions of the agents K and the objects L. The objects L may be things other than the agents K that exist in the environment. The type of the objects L is not particularly limited and may be selected appropriately depending on the embodiment. In one example, the objects L may include objects manipulated by the agent K, objects involved in the actions of the agent K, other objects existing in the environment, etc. The objects manipulated by the agent K may include, for example, objects involved in a task, such as an object to be carried by the agent K. The objects involved in the actions of the agent K may include, for example, objects that may affect the actions of the agent K, such as a floor or an obstacle. Note that when a form including state transitions of the objects L is employed together with a form of generating the encoder 5 so as to predict the future state, the obtained feature vector 35 may or may not include a predicted result of the future state of one or more objects L.

[0055] (Data Format) The data format of the information included in the time-series data 30 is not particularly limited and may be selected appropriately depending on the embodiment. The information may be expressed in any format, such as numerical values ​​or images. In one example, the time-series data 30 may include at least one of trajectory data and video data. According to one example of the present embodiment, improved data extraction accuracy can be expected when searching for at least one of trajectory data and video data.

[0056] The trajectory data is configured to indicate state transitions by a trajectory. The data format of the trajectory may be selected appropriately depending on the embodiment. In one example, the trajectory data may include numerical data of at least one of position and velocity at each of a plurality of times. The trajectory may be expressed by numerical data of at least one of position (coordinate values) and velocity. The trajectory data may further include numerical data other than position and attitude, such as attitude (tilt). In another example, the trajectory data may be configured by numerical data other than position and velocity.

[0057] The video data is composed of multiple images. The correspondence between images and time may be determined appropriately depending on the embodiment. In one example, one or more images may correspond to information at one time. That is, one or more images may show the state of agent K and object L at one time. In one example, each image may be encoded as is. In another example, analytical processing (preprocessing) such as identification and segmentation of objects may be applied to each image. The results of the analytical processing may be obtained as numerical data such as the category of the object and the range of segmentation. The analytical processing results obtained from each image may be encoded as time-series data 30.

[0058] In one example, a sensor may be used to acquire the time-series data 30. The sensor may include, for example, a camera, a depth sensor, an infrared sensor, an optical sensor, radar, a LiDAR (Light Detection and Ranging), a position sensor, an encoder, a motion capture, a tactile sensor, a force sensor, an acceleration sensor, a gyro sensor, an inertial measurement unit, or other measurement sensors. The position sensor may include, for example, a GPS (Global Positioning System) sensor, a GNSS (Global Navigation Satellite System) sensor, or the like. The sensing data obtained by the sensor may be used as the time-series data 30 as is, or the result of performing any arithmetic processing on the sensing data may be used as the time-series data 30.

[0059] The data format of the feature vector 35 is not particularly limited and may be selected appropriately depending on the embodiment. The feature vector 35 may be composed of multiple numerical values. Each numerical value may be stored in any format. In one example, the feature vector 35 may be composed of an array of numerical values. In another example, each numerical value may correspond to a pixel value, and the feature vector 35 may be composed in an image format. In another example, each numerical value included in the feature vector 35 may have a predetermined meaning corresponding to an inference task. When a configuration is adopted in which the encoder 5 is generated to predict the future state, the inference task is to predict the future state, and each numerical value included in the feature vector 35 may correspond to the future state. However, the format of the feature vector 35 is not limited to this example. In another example, the encoder 5 may be configured to encode without an inference task such as predicting the future state. Accordingly, each numerical value included in the feature vector 35 may have no meaning other than incorporating information about multiple times.

[0060] [Input / Output of Encoder] As long as the input data includes the time-series data 30 and the output data includes the feature vectors 35, the configuration of the input / output data of the encoder 5 is not particularly limited and may be determined appropriately depending on the embodiment. The encoder 5 may be configured to further accept input of information other than the time-series data 30. The encoder 5 may be configured to further output information other than the feature vectors 35.

[0061] 4 schematically shows an example of input and output of the encoder 5 according to this embodiment. As illustrated in FIG. 4 , the information collection device 1 may further acquire attribute information 31 of one or more agents K, attribute information 32 of one or more objects L, and environmental information 33 related to the environment in which the one or more agents K exist. In addition to the time-series data 30, the attribute information 31 of the agent K, the attribute information 32 of the object L, and the environmental information 33 may be input to the encoder 5. Accordingly, converting the acquired time-series data 30 into a feature vector 35 may be performed by converting the acquired time-series data 30, the attribute information 31 of the one or more agents K, the attribute information 32 of the one or more objects L, and the environmental information 33 into the feature vector 35.

[0062] The attribute information (31, 32) may include any information related to the attributes of the target (agent K, object L). The attribute information (31, 32) may include static information such as the type, material, hardness, shape, size (dimensions), and range of motion of the target. The size (dimensions) may include, for example, the size of each part of the robot device, such as the wheel diameter. The attribute information (31, 32) may also include dynamic information such as the temperature of the target. Dynamic information is information that may change over time, while static information is information that is unlikely to change over time. The environmental information 33 may include any information related to the environment. The environment may include all events related to the status of the object (agent K, object L). The environment may be composed of at least one of a real environment and a virtual environment. For example, the environment may be composed of both a real environment and a virtual environment, such as AR (Augmented Reality) and MR (Mixed Reality). The environment may also include VR (Virtual Reality), etc. The environmental information 33 may include, for example, the type of environment (land / on water / underwater, downhill / flat / uphill, etc.), temperature, humidity, weather, brightness (illuminance, etc.), etc.

[0063] Dynamic information among the attribute information (31, 32) and the environmental information 33 may be observed by a sensor. The information collection device 1 may acquire the dynamic information directly from the sensor or indirectly via another computer. The attribute information (31, 32) and the environmental information 33 may be stored in advance or may be provided by input from an operator. The information collection device 1 may store at least a portion of the attribute information (31, 32) and the environmental information 33 in advance, or may acquire them from an external computer or via input from an operator.

[0064] The input unit of the encoder 5 may be configured as appropriate to receive input of the attribute information (31, 32) and the environmental information 33 together with the time-series data 30. In one example, when the encoder 5 has the structure of FIG. 2 , the encoder 5 may be configured to further receive input of the attribute information (31, 32) and the environmental information 33 by increasing the number of input nodes of the self-attention mechanism 51 by the amount of the attribute information (31, 32) and the environmental information 33. However, the configuration of the input unit of the encoder 5 is not limited to this example and may be changed as appropriate depending on the embodiment.

[0065] According to one example of this embodiment, the attribute information 31 of agent K, the attribute information 32 of object L, and the environmental information 33 can be further incorporated into the feature vector 35. This makes it possible to search for data that further takes into consideration the attributes of agent K, the attributes of object L, and the environment. Therefore, it is possible to expect a further improvement in the accuracy of extracting the time-series data 30. Note that the configuration of the input data to the encoder 5 is not limited to this example. At least one of the attribute information 31 of agent K, the attribute information 32 of object L, and the environmental information 33 may be omitted. Accordingly, the relevant information may be omitted in the information acquisition process and conversion (encoding) process.

[0066] (Input / Output Format) The format in which the input data (time series data 30, attribute information 31, attribute information 32, and environmental information 33) is provided to the encoder 5 may be determined appropriately depending on the embodiment. In one example, the input data may be provided to the encoder 5 as is. In another example, preprocessing may be applied to at least a portion of the input data, and the preprocessed input data may be provided to the encoder 5. The preprocessing may include any computational process such as a process of analyzing information, a process of adding information, or a process of reducing information. A computational model that performs the preprocessing may be included in the encoder 5 or may be prepared separately from the encoder 5. The computational model may be configured by at least one of a trained machine learning model and a rule-based model. The rule-based model may be configured to derive an inference result (a result of preprocessing) from a provided input according to rules. The rules may be set appropriately.

[0067] The output format of the encoder 5 may also be determined appropriately depending on the embodiment. In one example, the output of the encoder 5 may be configured to directly indicate the feature vector 35. In another example, the output of the encoder 5 may be configured to indirectly indicate the feature vector 35. In this case, the feature vector 35 may be obtained by performing any information processing (e.g., interpretation processing) on ​​the output of the encoder 5.

[0068] [Requester] The requester U issues the query S using a computer. The requester U is not particularly limited and may be selected appropriately depending on the embodiment. In a typical example, the requester U may be a user, and the requester U's computer may be a user terminal. In another example, the requester U may be an operator operating the information providing device 2. In the case of direct operation, the requester U's computer may be the same as the information providing device 2 (i.e., the information providing device 2 may be the requester U's computer). In the case of indirect control, the requester U's computer may be a terminal connected to the information providing device 2. The requester U does not need to be limited to a person such as a user or operator. In another example, the requester U may be the computer itself. In this case, the query S may be issued by automatic processing of the computer. The requester U's computer may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose personal computer (PC), a tablet PC, a mobile device, a terminal device, etc. The mobile device may include, for example, a smartphone.

[0069] [Query] A query S is generated corresponding to the feature vector 35 to obtain desired time-series data. The query S may be generated as appropriate depending on the embodiment. In one example, the query S may be generated by encoding the target time-series data using the encoder 5.

[0070] The target time series data is configured to show state transitions of one or more target agents. The target agents are agents that appear in the time series data that are the target of query S. If one or more objects exist in the target environment, the target time series data may be configured to further show state transitions of one or more target objects. The target objects are objects that appear in the time series data that are the target of query S. Agent K in the data stored in database R may be referred to as the first agent, and the target agent in query S may be referred to as the second agent. Similarly, object L may be referred to as the first object, and the target object in query S may be referred to as the second object.

[0071] The time-series data that forms the basis of the query S may be obtained by any method, such as manual generation, automatic computer generation, sensor observation, data augmentation, etc. Conversion from the time-series data to the query S (i.e., generation of the query S) may be performed by the computer of the requester U, the information providing device 2, and an external computer. If the computer of the requester U generates the query S, issuing the query S by the requester U may be configured by generating the query S. If the information providing device 2 or an external computer generates the query S, issuing the query S by the requester U may be configured by providing the time-series data that forms the basis of the query S to the information providing device 2 or the external computer. If at least one of the attribute information 31 of the agent K, the attribute information 32 of the object L, and the environmental information 33 is reflected in the feature vector 35, at least one of the attribute information of the target agent, the attribute information of the target object, and the environmental information related to the target's environment may also be reflected in the query S.

[0072] Matching the query S may include a match between the query S and the feature vector 35 and the presence of the feature vector 35 in the vicinity of the query S. The range of the vicinity may be determined as appropriate depending on the embodiment. The metric for comparing the query S and the feature vector 35 may be selected as appropriate depending on the embodiment. In one example, at least one of Euclidean distance and cosine similarity may be used as the metric for comparison. The associated feature vector 65 is a feature vector 35 that matches the query S among the feature vectors 35 stored in the database R. The associated time-series data 60 is time-series data 30 associated with the feature vector 35 (associated feature vector 65) that matches the query S among the stored time-series data 30. In one example, the type and number of agents in the associated time-series data 60 may match the target agents in the query S. In another example, at least one of the type and number of agents in the associated time-series data 60 may differ from the target agents in the query S. The same applies to objects. In one example, the type and number of objects in the associated time-series data 60 may match the target objects in the query S. In another example, at least one of the type and the number of objects in the related time series data 60 may be different from the target objects in the query S. Note that the related time series data 60 may be read as the matched time series data, and the related feature vector 65 may be read as the matched feature vector.

[0073] The time series data 30 may be obtained entirely from the same information source, or at least partially from different information sources. The types and number of information sources from which the time series data 30 are obtained are not particularly limited and may be determined appropriately depending on the embodiment. The time series data 30 may be collected from various information sources. In the information providing device 2, accepting a query S may be performed by accepting multiple queries S. The multiple queries S may be issued for a series of state transitions (e.g., a behavioral sequence). Extracting the associated feature vectors 65 may be performed by extracting multiple associated feature vectors 65 that match each of the multiple queries S. For the multiple extracted associated feature vectors 65, the information source of the time series data 30 (associated time series data 60) associated with some of the associated feature vectors 65 may be allowed to differ from the information source of the time series data 30 (associated time series data 60) associated with the other associated feature vectors 65. In other words, when multiple searches are performed for a series of state transitions, search results may be obtained that include a mixture of time series data 30 obtained from different information sources. In one example, different information sources may be constituted by at least partially different conditions under which the time-series data 30 were obtained. For example, different information sources may be constituted by different aspects of at least one of the agent K, the object L, and the environment. According to one example of the present embodiment, by allowing time-series data 30 from different information sources to be mixed as search results, the time-series data 30 becomes easier to search. As a result, it can be expected that the reusability of the time-series data 30 will be improved.

[0074] Furthermore, the time-series data 30 may be collected in scenes in which one or more agents K perform tasks. The tasks may be, for example, work, travel, etc. The tasks may be, for example, assembly, cooking, cleaning, chemical experiments, etc. The goal of the task may be given as appropriate. As an example, the task may be transporting an object, guiding the movement of an object, etc. Transporting an object may include, for example, transporting supplies to a disaster site, transporting equipment for planetary exploration, etc. Guiding the movement of an object by an agent may include, for example, a robotic device controlling the flow of people, a robotic device controlling the movement of livestock, etc. Depending on the accumulation of time-series data 30 in scenes in which tasks are performed, a task (target task) that will be the target of the search may be set in the query S.

[0075] The type of agent K in the time-series data 30 extracted as a search result may match or may be different from the target agent in the target task of query S. In one example, for the extracted feature vector 35, the type of one or more agents (agent K) in the related time-series data 60 may be allowed to be different from the type of one or more target agents in the target task of query S. In other words, time-series data 30 of an agent of a type different from the target agent set in query S may be extracted as a search result for query S.

[0076] Furthermore, the number of agents K in the time-series data 30 extracted as a search result may match or may differ from the target agents in the target task of query S. In one example, for an extracted feature vector 35, the number of agents (agent K) in the associated time-series data 60 may be allowed to differ from the number of target agents in the target task of query S. In other words, time-series data 30 of a number of agents different from the number of target agents in query S may be extracted as a search result for query S.

[0077] Furthermore, the task performed by agent K in the time-series data 30 extracted as a search result may match or may differ from the target task in query S. In one example, for the extracted feature vector 35, the task performed by one or more agents (agent K) in the related time-series data 60 may be allowed to differ from the target task in query S. In other words, time-series data 30 of a scene in which a task different from the target task set in query S is performed may be extracted as a search result for query S.

[0078] For example, a query intended to obtain time series data showing the behavior of a group of robotic devices (such as a group of robots or drones) may extract time series data showing the behavior of a flock of living organisms, such as fish or birds, as a search result. As a result, the behavior of the flock of living organisms may be diverted to the behavior of a group of robotic devices. Furthermore, a query intended to obtain time series data showing the behavior of a robotic device that guides a flock of livestock may extract time series data showing the behavior of a sheepdog that guides a flock of sheep as a search result. As a result, the behavior of the sheepdog may be diverted to the behavior of a robotic device. Behavior in a virtual space may be diverted to a real space, or behavior in a real space may be diverted to a virtual space.

[0079] According to one example of this embodiment, by allowing differences in at least one of the type, number, and task of agents between the query S and the search results, the time-series data 30 can be more easily searched, as in the above example. As a result, it is expected that the reusability of the time-series data 30 can be improved.

[0080] [Usage Scenarios] The system according to this embodiment may search the time-series data 30 for any purpose. In one example, the time-series data 30 may be searched to obtain data to be used in machine learning such as imitation learning or offline reinforcement learning. In another example, the time-series data 30 may be searched to refer to actions similar to a desired action, such as to obtain reference actions of cleaning patterns of multiple cleaning robots.

[0081] The provided time series data 30 (related time series data 60) may be used as is, or may be used after processing such as data augmentation. In one example, data augmentation may be performed by changing the scale (e.g., time scale, spatial scale, etc.) of the provided time series data 30, thereby generating new time series data. As in the data collection scenario described above, the scale change may include at least one of scaling up (e.g., stretching, enlarging) and scaling down (e.g., contracting, reducing). The scale change may involve applying different modification processes to different parts, such as scaling up a portion and scaling down the rest. The generated new time series data may be used together with the provided time series data 30, or only the generated new time series data may be used.

[0082] The change in scale of the time series data 30 (related time series data 60) may be performed by at least one of the computer of the requester U, the external computer, and the information providing device 2. In one example, the computer of the requester U may change the scale of the time series data 30 received from the information providing device 2 and use the changed time series data 30 together with the time series data 30 with the original scale for the target information processing. The computer of the requester U may change the scale of the time series data 30 to suit the target information processing and use only the changed time series data 30 for the target information processing. Furthermore, the computer of the requester U may generate multiple pieces of time series data 30 by applying different scale changes to each piece, and use the generated multiple pieces of time series data 30 for the target information processing.

[0083] In another example, the information providing device 2 may change the scale of the time series data 30 (the associated time series data 60). That is, returning the time series data 30 to the requester U as the associated time series data 60 may include changing the scale of the associated time series data 60 and returning the changed associated time series data 60 to the requester U. This is expected to improve convenience. For example, the information providing device 2 may change the scale of the searched time series data 30 (the associated time series data 60) and return the changed time series data 30 to the requester U together with the time series data 30 with the original scale. The information providing device 2 may return only the changed time series data 30 to the requester U. Furthermore, the information providing device 2 may generate multiple pieces of time series data 30 by applying different scale changes to each piece and return the generated multiple pieces of time series data 30 to the requester U. This is expected to improve convenience by increasing the amount of data provided.

[0084] Furthermore, the scale may be changed at a timing other than the timing of collecting the data and the timing of providing the data. In one example, the scale may be changed at the timing of generating the query S. That is, the query S may be generated by encoding the time-series data of the target using the encoder 5, and the scale of the time-series data of the target may be changed as appropriate. This scale change may be performed in at least one of the computer of the requester U, the external computer, and the information providing device 2.

[0085] In one example, the scale change in the query S may be performed by the computer of the requester U. That is, for the query S acquired by the information providing device 2, the scale of the target time-series data may be changed by the requester U. This can be expected to increase the search precision and improve convenience. For example, the computer of the requester U may send both the query S in the original scale and the query S with the changed scale to the information providing device 2 and request a search using each query S. The computer of the requester U may also request a search only using the query S with the changed scale. Furthermore, the computer of the requester U may generate multiple queries S by applying different scale changes to each query S and request a search using each query S. This can be expected to increase the search precision and improve convenience by increasing the amount of data provided.

[0086] In another example, the scale change of the query S may be performed in the information providing device 2. That is, in the information providing device 2, receiving the query S from the requester U may include receiving target time-series data from the requester U, changing the scale of the target time-series data, and encoding the changed target time-series data using the encoder 5 to generate the query S. This can be expected to improve the search precision and convenience. For example, the information providing device 2 may perform a search using both the query S with the original scale and the query S with the changed scale, and return the matched associated time-series data 60 to the requester U. The information providing device 2 may perform a search using only the changed query S, and return the matched associated time-series data 60 to the requester U. Furthermore, the information providing device 2 may generate multiple queries S by applying different scale changes to each query S, and perform a search using each query S. This can be expected to improve the search precision and convenience by providing more data.

[0087] Either one of the scale change in the query S and the scale change in the time-series data 30 to be provided may be performed, or both may be performed. When both are performed, the computer of the requester U, the external computer, and the information providing device 2 may freely change the scale of the query S and the associated time-series data 60. For example, the computer of the requester U or the information providing device 2 may change the scale of the query S and perform a search using the changed query S. If a feature vector 35 matching the query S is not extracted in the search, the scale of the query S may be further changed, and a search using the changed query S may be performed again. This scale change and search may be repeated until a feature vector 35 matching the query S is extracted. Then, when providing the matched associated time-series data 60 to the requester U, the computer of the requester U or the information providing device 2 may perform a scale change that is opposite to the scale change applied in the query S (i.e., a change to restore the original scale). This increases the probability of providing time series data 30 (related time series data 60) that has the same scale as the target time series data, which is expected to improve convenience.

[0088] [System Configuration] In one example, the information collection device 1, the database R, and the information providing device 2 may be connected to each other via a network. The information providing device 2 and the computer of the requester U may also be connected to each other via a network. The type of network may be appropriately selected from, for example, the Internet, a wireless communication network, a mobile communication network, a telephone network, a dedicated network, etc. However, the method of data exchange between each device is not limited to this example, and may be appropriately selected depending on the embodiment.

[0089] The database R may be stored in any storage area accessible from the information collecting device 1 and the information providing device 2. The database R may be located in an internal storage area of ​​at least one of the information collecting device 1 and the information providing device 2, or in an external storage area of ​​the information collecting device 1 and the information providing device 2. The internal storage area may be a memory resource of at least one of the information collecting device 1 and the information providing device 2. The external storage area may be a memory resource of an external computer, such as a network-attached storage (NAS). Storing the feature vector 35 by the information collecting device 1 in the database R may include the information collecting device 1 directly storing the feature vector 35 in the database R and storing the feature vector 35 indirectly via an external computer. Indirect storage may include giving an instruction to cause the external computer to store the feature vector 35.

[0090] The time series data 30 may be stored in any storage area as long as it can be read from the corresponding feature vector 35. The time series data 30 may be stored in the database R together with the feature vector 35, or may be stored in a database separate from the database R. The time series data 30 may be appropriately stored in any storage area accessible from the information collecting device 1 and the information providing device 2. The database that stores the time series data 30 may be arranged in an internal storage area of ​​at least one of the information collecting device 1 and the information providing device 2, or may be arranged in an external storage area of ​​the information collecting device 1 and the information providing device 2. The data format of the database R is not particularly limited and may be appropriately selected depending on the embodiment. The data format may include, for example, a table format, a distributed ledger format, etc. The same applies to the database that stores the time series data 30.

[0091] 1, the information collection device 1 and the information providing device 2 may be separate computers. In another example, the information collection device 1 and the information providing device 2 may be configured as an integrated computer. In this case, one or more computers may operate as the information collection device 1 and the information providing device 2. At least one of the information collection device 1 and the information providing device 2 may be configured as multiple computers.

[0092] §2 Configuration Example [Hardware Configuration] (Information Collection Device) Fig. 5 shows a schematic example of a hardware configuration of the information collection device 1 according to this embodiment. In the example shown in Fig. 5, the information collection device 1 according to this embodiment is a computer to which a control unit 11, a storage unit 12, an external interface 13, an input device 14, an output device 15, and a drive 16 are electrically connected.

[0093] The control unit 11 includes a hardware processor such as a CPU (Central Processing Unit), RAM (Random Access Memory), and ROM (Read Only Memory), and is configured to execute information processing based on programs and various data. The control unit 11 (CPU) is an example of a processor resource. The storage unit 12 may be configured, for example, with a hard disk drive, a solid state drive, or the like. The storage unit 12, RAM, and ROM are examples of memory resources. In this embodiment, the storage unit 12 stores various information such as the program 81 and the encoder data 500.

[0094] The program 81 is a program for causing the information collection device 1 to execute information processing (see FIG. 9 , described below) related to the collection of time-series data 30. The program 81 includes a series of instructions for the information processing. The encoder data 500 indicates information related to the encoder 5. As long as the encoder 5 can be reproduced during encoding, the configuration of the encoder data 500 is not particularly limited and may be determined appropriately depending on the embodiment. In one example, when the encoder 5 is configured using a machine learning model, the encoder data 500 may include information indicating values ​​of calculation parameters adjusted by machine learning. In some cases, the encoder data 500 may further include information indicating the configuration of the machine learning model (e.g., the structure of a neural network, etc.). The encoder data 500 may be incorporated into the program 81.

[0095] The external interface 13 is configured to connect to an external device via a wired or wireless connection. The external interface 13 may be, for example, a USB (Universal Serial Bus) port, a communication port, a dedicated port, etc. The type and number of external interfaces 13 may be determined appropriately depending on the embodiment. If the external interface 13 includes a communication port, the communication standard of the communication port may be selected arbitrarily. In this embodiment, if the database R is located externally, the information collection device 1 may be connected to the database R via the external interface 13.

[0096] The input device 14 is a device for inputting, for example, a mouse, a keyboard, etc. The output device 15 is a device for outputting, for example, a display, a speaker, etc. An operator can operate the information collection device 1 by using the input device 14 and the output device 15. The input device 14 and the output device 15 may be connected via an external interface 13. The input device 14 and the output device 15 may be integrally configured, for example, by a touch panel display, etc.

[0097] The drive 16 is a device for reading various information, such as a program, stored in a storage medium 91. At least one of the program 81 and the encoder data 500 may be stored in the storage medium 91 instead of or together with the storage unit 12. The storage medium 91 is configured to accumulate various information (such as the stored program) by electrical, magnetic, optical, mechanical, or chemical action so that a machine such as a computer can read the information. The information collection device 1 may acquire at least one of the program 81 and the encoder data 500 from the storage medium 91. The storage medium 91 may be a disk-type storage medium such as a CD or DVD, or a non-disk-type storage medium such as a semiconductor memory (e.g., a flash memory). The type of the drive 16 may be selected appropriately depending on the type of the storage medium 91. The drive 16 may be connected via an external interface 13.

[0098] Note that, with regard to the specific hardware configuration of the information collection device 1, components may be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 11 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, a field-programmable gate array (FPGA), a digital signal processor (DSP), a graphics processing unit (GPU), an application-specific integrated circuit (ASIC), or the like. The storage unit 12 may be configured with RAM and ROM included in the control unit 11. At least one of the external interface 13, the input device 14, the output device 15, and the drive 16 may be omitted. The information collection device 1 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be identical. Furthermore, the information collection device 1 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a mobile device, a terminal device, or the like.

[0099] (Information Providing Device) Fig. 6 schematically shows an example of the hardware configuration of the information providing device 2 according to this embodiment. In the example shown in Fig. 6, the information providing device 2 according to this embodiment is a computer to which a control unit 21, a storage unit 22, an external interface 23, an input device 24, an output device 25, and a drive 26 are electrically connected.

[0100] The control unit 21 to the drive 26 and the storage medium 92 of the information providing device 2 may be configured similarly to the control unit 11 to the drive 16 and the storage medium 91 of the information collecting device 1. The control unit 21 (CPU) is an example of a processor resource of the information providing device 2, and the storage unit 22 (and RAM, ROM) is an example of a memory resource of the information providing device 2. In this embodiment, the storage unit 22 stores various information such as a program 82.

[0101] The program 82 is a program for causing the information providing device 2 to execute information processing (see FIG. 10 described below) related to the provision of time-series data 30. The program 82 includes a series of instructions for the information processing. The program 82 may be stored in a storage medium 92 instead of or together with the storage unit 22. The information providing device 2 may acquire the program 82 from the storage medium 92. Furthermore, when the information providing device 2 uses the encoder 5 to generate a query S from time-series data received from a request source U, encoder data 500 may be stored in at least one of the storage unit 22 and the storage medium 92.

[0102] When the database R is located externally, the information providing device 2 may be connected to the database R via an external interface 23. When the computer of the requester U is an external computer, the information providing device 2 may perform data communication with the computer of the requester U via the external interface 23. The information providing device 2 may be equipped with multiple external interfaces 23, and different external interfaces 23 may be used to connect the database R and the computer of the requester U. An operator can operate the information providing device 2 by using an input device 24 and an output device 25.

[0103] Note that, with regard to the specific hardware configuration of the information providing device 2, components can be omitted, replaced, or added as appropriate depending on the embodiment. For example, the control unit 21 may include multiple hardware processors. The hardware processor may be configured with a microprocessor, FPGA, DSP, GPU, ASIC, etc. At least one of the external interface 23, the input device 24, the output device 25, and the drive 26 may be omitted. The information providing device 2 may be configured with multiple computers. In this case, the hardware configurations of the computers may or may not be the same. The information providing device 2 may be an information processing device designed specifically for the service provided, as well as a general-purpose server device, a general-purpose PC, a tablet PC, a mobile device, a terminal device, etc.

[0104] [Software Configuration] (Information Collection Device) Fig. 7 schematically illustrates an example of the software configuration of the information collection device 1 according to this embodiment. The control unit 11 of the information collection device 1 loads a program 81 stored in the storage unit 12 into RAM and executes instructions included in the program 81 using the CPU. As a result, the information collection device 1 operates as a computer including a data acquisition unit 111, a conversion unit 112, and a storage processing unit 113 as software modules. That is, in this embodiment, each software module of the information collection device 1 is realized by the control unit 11 (CPU).

[0105] The data acquisition unit 111 is configured to acquire time-series data 30 indicating state transitions of one or more agents K. The conversion unit 112 includes an encoder 5 configured to store encoder data 500 and thereby encode the data incorporating information of multiple times. The conversion unit 112 is configured to convert the acquired time-series data 30 into a feature vector 35 using the encoder 5. The storage processing unit 113 is configured to store the acquired feature vector 35 in the database R in association with the time-series data 30.

[0106] 8 illustrates a schematic example of a software configuration of the information providing device 2 according to this embodiment. The control unit 21 of the information providing device 2 loads a program 82 stored in the storage unit 22 into RAM and executes instructions included in the program 82 using the CPU. As a result, the information providing device 2 operates as a computer including a reception unit 211, an extraction unit 212, and a response unit 213 as software modules. That is, in this embodiment, each software module of the information providing device 2 is realized by the control unit 21 (CPU).

[0107] The receiving unit 211 is configured to receive a query S for the database R from a requester U. When the query S is generated using an encoder 5 in the information providing device 2, the receiving unit 211 may include the encoder 5 by holding encoder data 500. The extracting unit 212 is configured to extract a feature vector 35 that matches the query S from the database R as an associated feature vector 65. The responding unit 213 is configured to return time-series data 30 associated with the extracted associated feature vector 65 to the requester U as associated time-series data 60.

[0108] (Other) In the present embodiment, an example is described in which each software module of the information collection device 1 and the information providing device 2 is implemented by a general-purpose CPU. However, some or all of the software modules may be implemented by one or more dedicated processors or chipsets. Each module may be implemented as a hardware module. With regard to the software configuration of the information collection device 1 and the information providing device 2, modules may be omitted, replaced, or added as appropriate depending on the embodiment.

[0109] §3 Operational Example [Information Collection Device] Figure 9 is a flowchart showing an example of the processing procedure of the information collection device 1 according to this embodiment. The following processing procedure is an example of an information collection method executed by a computer. However, the following processing procedure of the information collection device 1 is merely an example, and each step may be modified as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0110] (Step S101) In step S101, the control unit 11 operates as the data acquisition unit 111 and acquires the time-series data 30 indicating the state transitions of one or more agents K.

[0111] In one example, the control unit 11 may acquire the time series data 30 by methods such as manual generation, automatic generation by a computer, observation by a sensor, data augmentation, etc. The control unit 11 may acquire the time series data 30 by operator input or by executing information processing. The control unit 11 may acquire the time series data 30 from a sensor or an external computer. In one example, the control unit 11 may acquire time series data 30 indicating state transitions of multiple agents K. In one example, the control unit 11 may acquire time series data 30 indicating state transitions of one or more agents K and one or more objects L. In one example, the control unit 11 may acquire time series data 30 including at least one of trajectory data and video data.

[0112] In one example, the control unit 11 may further acquire at least one of attribute information 31 of one or more agents K, attribute information 32 of one or more objects L, and environmental information 33 related to the environment in which the one or more agents K exist. The order in which the information is acquired is not particularly limited and may be determined appropriately depending on the embodiment. After acquiring the time-series data 30, the control unit 11 proceeds to the next step S102.

[0113] (Step S102) In step S102, the control unit 11 operates as the conversion unit 112 and converts the acquired time-series data 30 into a feature vector 35 using the encoder 5.

[0114] In one example, the control unit 11 inputs the time series data 30 to the encoder 5 and executes the arithmetic processing of the encoder 5. Through this conversion (encoding) processing, the control unit 11 acquires a feature vector 35. The encoder 5 is configured to perform encoding so as to incorporate information from multiple times. In one example, the encoder 5 may be configured to perform encoding so as to incorporate information from multiple times by being trained to predict a future state. In one example, the encoder 5 may be configured to perform encoding so as to incorporate information from multiple times and spaces.

[0115] When at least one of the attribute information (31, 32) and the environmental information 33 has been acquired, the control unit 11 may input at least one of the acquired attribute information (31, 32) and the environmental information 33 together with the time-series data 30 to the encoder 5, and execute calculation processing by the encoder 5. As a result, the control unit 11 may use the encoder 5 to convert the time-series data 30 and at least one of the attribute information (31, 32) and the environmental information 33 into a feature vector 35. By this encoding, at least one of the attribute information (31, 32) and the environmental information 33 may be reflected in the resulting feature vector 35. After acquiring the feature vector 35, the control unit 11 proceeds to the next step S103.

[0116] (Step S103) In step S103, the control unit 11 operates as the storage processing unit 113 and stores the obtained feature vector 35 in the database R in association with the time-series data 30. In one example, the control unit 11 may store the feature vector 35 and the time-series data 30 in association with each other in the database R. In another example, the control unit 11 may store the feature vector 35 in the database R, and store the time-series data 30 associated with the feature vector 35 in another database.

[0117] When the storage of the time-series data 30 and the feature vectors 35 is completed, the control unit 11 ends the processing procedure of the information collection device 1 according to this operation example. The control unit 11 may accumulate the time-series data 30 and the feature vectors 35 by executing the processes of steps S101 to S103 every time time-series data 30 is obtained.

[0118] [Information Providing Device] Fig. 10 is a flowchart showing an example of a processing procedure of the information providing device 2 according to this embodiment. The following processing procedure is an example of an information providing method executed by a computer. However, the following processing procedure of the information providing device 2 is merely an example, and each step may be changed as much as possible. Furthermore, steps in the following processing procedure may be omitted, replaced, or added as appropriate depending on the embodiment.

[0119] (Step S201) In step S201, the control unit 21 operates as the reception unit 211 and receives a query S for a database R from a request source U.

[0120] In one example, the query S may be generated from time series data of the target. The time series data of the target may be obtained by methods such as manual generation, automatic computer generation, sensor observation, data augmentation, etc. Furthermore, conversion from the time series data to the query S (i.e., generation of the query S) may be performed in at least one of the computer of the requester U, the external computer, and the information providing device 2. In one example, the computer of the requester U may generate the query S from the time series data of the target, and the control unit 21 may receive the generated query S from the computer of the requester U. In another example, the control unit 21 may receive the query S from an external computer. The time series data of the target may be provided to the external computer by the computer of the requester U or the information providing device 2. In yet another example, the control unit 21 may receive the time series data of the target from the computer of the requester U and encode the time series data using the encoder 5 to obtain the query S. In one example, the scale of the time series data of the target may be changed in at least one of the computer of the requester U, the external computer, and the information providing device 2. When the control unit 21 receives the query S, the control unit 21 proceeds to the next step S202.

[0121] (Step S202) In step S202, the control unit 21 operates as the extraction unit 212 and extracts the feature vector 35 that matches the query S from the database R as the related feature vector 65.

[0122] In one example, the control unit 21 may extract feature vectors 35 that match the query S or exist in the vicinity of the query S from the database R. The method for searching the vicinity range is not particularly limited and may be selected as appropriate depending on the embodiment. A known method may be used to search the vicinity range. The number of items to be extracted is not particularly limited and may be determined as appropriate depending on the embodiment.

[0123] If a matching feature vector 35 does not exist, the control unit 21 ends the processing procedure of the information providing device 2 according to this operation example. The control unit 21 may return a notification to the request source U indicating that a matching time-series data 30 does not exist. On the other hand, if a matching feature vector 35 exists, an associated feature vector 65 is extracted. After extracting the associated feature vector 65, the control unit 21 proceeds to the next step S203.

[0124] (Step S203) In step S203, the control unit 21 operates as the response unit 213 and acquires the time series data 30 associated with the extracted associated feature vector 65 as the associated time series data 60. When the time series data 30 is stored in the database R together with the feature vector 35, the control unit 21 may acquire the associated time series data 60 from the database R. When the time series data 30 is stored in a database other than the database R, the control unit 21 may acquire the associated time series data 60 from the other database. The control unit 21 may return the acquired associated time series data 60 to the request source U.

[0125] In one example, the control unit 21 may accept multiple queries S in step S201, and may extract associated feature vectors 65 matching each query S in step S202. In this case, a search result including a mixture of time-series data 30 obtained from different information sources may be obtained. In step S203, the control unit 21 may return to the requester U a search result including a mixture of associated time-series data 60 obtained from different information sources. In another example, differences in at least one of the type, number, and task of agents between the query S and the search result may be permitted. Accordingly, in step S203, the control unit 21 may return to the requester U a search result that differs from the query S in at least one of the type, number, and task of agents. In another example, the scale of the associated time-series data 60 to be provided may be changed in at least one of the computer of the requester U, the external computer, and the information providing device 2.

[0126] When the provision of the related time-series data 60 (time-series data 30) is completed, the control unit 21 ends the processing procedure of the information providing device 2 according to this operation example. The control unit 21 may provide the requester U with the time-series data 30 that matches the query S as the related time-series data 60 by executing the processes of steps S201 to S203 every time the control unit 21 receives a query S from the requester U.

[0127] [Features] In the information collection device 1 according to this embodiment, information on multiple times included in the time series data 30 is incorporated into a feature vector 35 through encoding by the encoder 5 in step S102. In step S103, the feature vector 35 obtained by the conversion is stored in association with the time series data 30. In the information providing device 2 according to this embodiment, the feature vector 35 is used as meta-information in a search in the process of step S202, thereby allowing the search to take into account the state transition sequence. As a result, in step S203, extraction of appropriate time series data 30 for the search target sequence can be expected. Therefore, with the information collection device 1 according to this embodiment, improvement in the accuracy of extraction of time series data 30 can be expected. Furthermore, with the information providing device 2 according to this embodiment, highly accurate extraction of such time series data 30 and provision of appropriate time series data 30 can be expected.

[0128] §4 Modifications Although the embodiments of the present invention have been described above in detail, the above description is merely an example of the present invention in every respect. Various improvements or modifications may be made to the above embodiments as appropriate. For example, the following modifications are possible. Note that, in the following, the same reference numerals are used for components similar to those in the above embodiments, and descriptions of the same points as in the above embodiments are omitted as appropriate. The following modifications can be combined as appropriate.

[0129] <4.1> In the above embodiment, at least one of the computer of the requester U, the external computer, and the information providing device 2 may use an inference model to generate time-series data of the query S or the target from an instruction in natural language. The instruction in natural language may be given by the requester U using any method such as text input, voice input, or image input.

[0130] A trained machine learning model may be used as the inference model. The configuration of the machine learning model is not particularly limited as long as it can perform inference to generate a query S or time-series data from a natural language instruction, and may be selected appropriately depending on the embodiment. The machine learning model (inference model) may be appropriately trained to acquire the ability to make such inferences.

[0131] In one example, the inference model may be a trained machine learning model capable of in-context learning. In-context learning refers to acquiring the ability to make specific inferences through the context of input (prompts), such as input / output samples. The inference model may be, for example, a large language model (LLM), a large vision-language model (LVLM), or the like. The large vision-language model may include a visual question answering model, an open vocabulary object detection model, an open vocabulary object segmentation model, or the like. Alternatively, the inference model may be a large language model combined with one or more other modalities (e.g., sound), such as audio question answering. Furthermore, the inference model may be a large model of one or more other modalities other than language, such as a large audio model. The data format of the input to the inference model is not particularly limited and may be selected appropriately depending on the embodiment. When using an inference model capable of in-context learning, input / output samples consisting of a combination of a natural language instruction and a query S or time-series data may be provided to the inference model before or when a natural language instruction is provided, thereby promoting in-context learning in the inference model and improving the accuracy of inference.

[0132] When generating time-series data, the generated time-series data may be appropriately converted into a query S by the encoder 5. According to this modification, the query S can be generated based on an instruction in natural language.

[0133] Furthermore, after the query S or target time-series data is generated, the processes of steps S201 to S203 may be automatically executed. This allows for the acquisition of search results for the related time-series data 60 based on natural language instructions. When using an inference model capable of in-context learning, re-prompting may be performed in any manner if the search results are inappropriate. This makes it easier to obtain appropriate search results by correcting the search results. <4.2> In the above embodiment, when saving the time-series data 30, the information collection device 1 (control unit 11) may further associate management information, such as owner information and expiration information, with the time-series data 30. The management information may be obtained as appropriate depending on the embodiment. The management information may be used to manage the time-series data 30, for example, by confirming the owner and specifying a storage expiration date.

[0134] §5 Experimental Examples The following experiments were carried out to verify the effectiveness of the above-described embodiment, but the present invention is not limited to the following examples.

[0135] (Preparation) In the simulation environment for imitation learning, three robotic devices were defined as agents, and a stick was defined as the object. The target task was defined as the task of transporting the stick to a predetermined location using the three robotic devices. The shape of each robotic device was set to be circular, and the diameter of each robotic device was set to 3 cm. The shape of the stick was set to be rectangular, and the dimensions of the stick were set to 24 cm x 2 cm x 3 cm. The distance from the starting point to the target point was set to be between 0 cm and 20 cm.

[0136] We trained policies to accomplish the above task using the Soft Actor-Critic (SAC) method of reinforcement learning. We prepared three policies by categorizing them into three categories—low, medium, and high—based on the cumulative reward value at the end of training. We operated each robot device using the three resulting policies and recorded the position and velocity of each robot device and rod to collect time series data for each policy. We then integrated the time series data collected by each policy to generate time series data for training. The number of training time series data generated was 30,000. Furthermore, we generated time series data for queries by operating each robot device using a high-proficiency policy. The number of query time series data generated was 500.

[0137] (Example) In this example, an encoder network composed of an encoder-decoder type Transformer was prepared. The encoder and decoder were set to two layers. The embedding dimension of the encoder (i.e., the number of dimensions of the feature vector) was set to 512. A trained encoder network was generated by performing supervised learning of the encoder network using a dataset composed of a combination of past trajectories and future trajectories. Feature vectors were generated by encoding time-series data for training using the encoder in the generated trained encoder network. A database was constructed by associating the generated feature vectors with time-series data.

[0138] Furthermore, a query was generated by encoding the query time-series data using an encoder. Using the generated query, a search similar to steps S202 and S203 of the information providing device 2 was performed. As a result, 5,000 pieces of time-series data were extracted from the 30,000 pieces of time-series data. The query time-series data was added to the extracted 5,000 pieces of time-series data, resulting in a total of 5,500 pieces of time-series data being set as the target for imitation learning.

[0139] (Comparative Examples) In the first comparative example, query time-series data (500 items) was set as the target of imitation learning. In the second comparative example, learning time-series data (30,000 items) was set as the target of imitation learning. In the third comparative example, by comparing the time-series data, the query time-series data was used as the query as is, and 5,000 items of time-series data were extracted from the 30,000 items of time-series data. Euclidean distance was used as the comparison index. The query time-series data was added to the extracted 5,000 items of time-series data, and 5,500 items of time-series data were set as the target of imitation learning.

[0140] (Learning Conditions) Imitation learning of a policy for performing a target task was performed using the time-series data set as the target in the example and each comparative example. The same model configuration as the encoder was used for the policy. Adam was used as the learning algorithm. The number of learning epochs was set to 50. Mean Squared Error (MSE) was used as the loss function. The prediction error of future speed was used as the loss. The target task was performed using the learned policy obtained in this way. 100 trials were performed and the success rate was calculated.

[0141] (Experimental Results) FIG. 11 shows the experimental results of the example and each comparative example. As shown in FIG. 11, the experimental example was able to obtain a strategy with a higher success rate than each comparative example. From these results, it was found that the above embodiment can extract time-series data appropriate for imitation learning. In other words, it was found that the above embodiment can improve the accuracy of extracting appropriate time-series data. This demonstrates the usefulness of the above embodiment.

[0142] This specification includes the following disclosures. [Supplementary Note 1] An information collection device (1) including a control unit (11), wherein the control unit (11) is configured to: acquire time-series data (30) indicating state transitions of one or more agents (K); convert the acquired time-series data (30) into a feature vector (35) using an encoder (5) configured to encode the data while incorporating information from multiple times; and store the obtained feature vector (35) in a database (R) in association with the time-series data (30). [Supplementary Note 2] The information collection device (1) according to Supplementary Note 1, wherein the time-series data (30) is configured to indicate state transitions of multiple agents (K), and the encoder (5) is configured to further incorporate spatial information into the encoding. [Supplementary Note 3] The information collection device (1) according to Supplementary Note 1 or Supplementary Note 2, wherein the encoder (5) is configured to encode the data while incorporating information from multiple times by being trained to predict a future state of one or more agents (K) from their state transitions. [Supplementary Note 4] The information collection device (1) according to any one of Supplementary Notes 1 to 3, wherein the time series data (30) includes trajectory data. [Supplementary Note 5] The information collection device (1) according to any one of Supplementary Notes 1 to 4, wherein the time series data (30) includes video data. [Supplementary Note 6] The information collection device (1) according to any one of Supplementary Notes 1 to 5, wherein the control unit (11) is configured to further acquire attribute information (31) of the one or more agents (K), and converting the acquired time series data (30) into a feature vector (35) comprises converting the acquired time series data (30) and the attribute information (31) of the one or more agents (K) into the feature vector (35). [Supplementary Note 7] The information collection device (1) according to any one of Supplementary Notes 1 to 6, wherein the time series data (30) is further configured to indicate state transitions of one or more objects, and the encoder (5) is configured to encode the time series data by further incorporating spatial information.[Supplementary Note 8] The information collection device (1) according to Supplementary Note 7, wherein the control unit (11) is configured to further acquire attribute information (32) of the one or more objects, and converting the acquired time series data (30) into a feature vector (35) comprises converting the acquired time series data (30) and the attribute information (32) of the one or more objects into the feature vector (35). [Supplementary Note 9] The information collection device (1) according to any one of Supplements 1 to 8, wherein the control unit (11) is further configured to acquire environmental information (33) related to an environment in which the one or more agents (K) exist, and converting the acquired time series data (30) into a feature vector (35) comprises converting the acquired time series data (30) and the environmental information (33) into the feature vector (35). [Supplementary Note 10] The information collection device (1) according to any one of Supplements 1 to 9, wherein the agent (K) is a robotic device or a living organism. [Supplementary Note 11] An information providing device (2) connected to a database (R) that stores feature vectors (35) associated with time-series data (30) and comprising a control unit (21), wherein the time-series data (30) indicates state transitions of one or more agents (K), and the feature vectors (35) are obtained by converting the time-series data (30) using an encoder (5) configured to encode the time-series data by incorporating information of a plurality of times, and the control unit (21) is configured to: accept a query (S) for the database (R) from a request source (U); extract a feature vector (35) that matches the query (S) from the database (R) as a related feature vector (65); and return the time-series data (30) associated with the extracted related feature vector (65) to the request source (U) as related time-series data (60).[Supplementary Note 12] The information provision device (2) according to Supplementary Note 11, wherein accepting the query (S) comprises accepting a plurality of the queries (S), and extracting the related feature vector (65) comprises extracting a plurality of related feature vectors (65) that respectively match the plurality of queries (S), and wherein, for the extracted plurality of related feature vectors (65), the information source of time-series data (30) associated with some of the extracted related feature vectors (65) is allowed to be different from the information source of time-series data (30) associated with the other related feature vectors (65). [Supplementary Note 13] The information provision device (2) according to Supplementary Note 11 or Supplementary Note 12, wherein, for the extracted feature vector (35), the types of the one or more agents in the related time-series data (60) are allowed to be different from the types of one or more target agents in a task targeted by the query (S). [Supplementary Note 14] The information providing device according to any one of Supplementary Notes 11 to 13, wherein the query (S) is generated by encoding time series data of a target using the encoder (5), and a scale of the time series data of the target is changed by the requester (U). [Supplementary Note 15] The information providing device according to any one of Supplementary Notes 11 to 14, wherein receiving the query (S) from the requester (U) includes receiving target time series data from the requester (U), changing the scale of the target time series data, and encoding the changed time series data of the target using the encoder (5) to generate the query (S). [Supplementary Note 16] The information providing device according to any one of Supplementary Note 11 to Supplementary Note 15, wherein returning the time series data (30) to the request source (U) as related time series data (60) includes: changing a scale of the related time series data (60); and returning the changed related time series data (60) to the request source (U).[Supplementary Note 17] An information collection method in which a computer (1) executes the following steps: acquiring time-series data (30) showing state transitions of one or more agents (K); converting the acquired time-series data (30) into a feature vector (35) using an encoder (5) configured to encode the data by incorporating information from multiple times; and storing the acquired feature vector (35) in a database (R) in association with the time-series data (30). [Supplementary Note 18] An information providing method by a computer (2) connected to a database (R) that stores feature vectors (35) associated with time-series data (30), wherein the time-series data (30) indicates state transitions of one or more agents (K), and the feature vectors (35) are obtained by converting the time-series data (30) using an encoder (5) configured to encode information incorporating a plurality of time points, and the computer (2) executes the following steps: accepting a query (S) for the database (R) from a requester (U); extracting a feature vector (35) that matches the query (S) from the database (R) as a related feature vector (65); and returning the time-series data (30) associated with the extracted related feature vector (65) to the requester (U) as related time-series data (60).

[0143] REFERENCE SIGNS LIST 1...information collection device, 11...control unit, 12...storage unit, 13...external interface, 14...input device, 15...output device, 16...drive, 81...program, 91...storage medium, 111...data acquisition unit, 112...conversion unit, 113...storage processing unit, 2...information provision device, 21...control unit, 22...storage unit, 23...external interface, 24...input device, 25...output device, 26...drive, 82...program, 92...storage medium, 211...reception unit, 212...extraction unit, 213...response unit, 30...time series data, 31, 32...attribute information, 33...environment information, 35...feature vector, 45...true value, 5...encoder, 500...encoder data, 51...self-attention mechanism, 53...cross-attention mechanism, 60...associated time series data, 65...associated feature vector, K...agent, L...object, R...database, S...query, U...requester

Claims

1. An information collection device having a control unit configured to: acquire time series data indicating state transitions of one or more agents; convert the acquired time series data into a feature vector using an encoder configured to encode the data by incorporating information at multiple times; and store the acquired feature vector in a database in association with the time series data.

2. The information collection device according to claim 1, wherein the time series data is configured to indicate state transitions of a plurality of the agents, and the encoder is configured to further incorporate spatial information into the encoding.

3. The information collection device according to claim 1, wherein the encoder is configured to encode information from multiple points in time by being trained to predict future states of one or more agents from their state transitions.

4. The information collection device according to claim 1, wherein the time series data includes trajectory data.

5. The information collection device according to claim 1, wherein the time series data includes video data.

6. The information collection device of claim 1, wherein the control unit is configured to further acquire attribute information of the one or more agents, and converting the acquired time series data into a feature vector is configured by converting the acquired time series data and the attribute information of the one or more agents into a feature vector.

7. The information collection device according to claim 1, wherein the time series data is configured to further indicate state transitions of one or more objects, and the encoder is configured to further incorporate spatial information into the encoding.

8. The information collection device of claim 7, wherein the control unit is configured to further acquire attribute information of the one or more objects, and converting the acquired time series data into a feature vector is configured by converting the acquired time series data and the attribute information of the one or more objects into a feature vector.

9. The information collection device of claim 1, wherein the control unit is further configured to acquire environmental information relating to an environment in which the one or more agents exist, and converting the acquired time series data into a feature vector is configured by converting the acquired time series data and the environmental information into a feature vector.

10. The information collection device according to claim 1, wherein the agent is a robotic device or a living being.

11. An information providing device connected to a database that stores feature vectors associated with time series data and comprising a control unit, wherein the time series data indicates state transitions of one or more agents, and the feature vectors are obtained by converting the time series data using an encoder configured to encode the data by incorporating information at multiple times, and the control unit is configured to perform the following operations: accept a query to the database from a requesting source; extract a feature vector that matches the query from the database as a related feature vector; and return time series data associated with the extracted related feature vector to the requesting source as related time series data.

12. The information providing device of claim 11, wherein accepting the query comprises accepting a plurality of the queries, and extracting the related feature vector comprises extracting a plurality of related feature vectors that match each of the plurality of queries, and wherein for the plurality of extracted related feature vectors, the information source of time series data associated with some of the related feature vectors is permitted to differ from the information source of time series data associated with other related feature vectors.

13. The information providing device of claim 11, wherein for the extracted feature vector, the type of one or more agents in the related time series data is allowed to differ from the type of one or more target agents in the task targeted by the query.

14. The information providing device according to claim 11, wherein the query is generated by encoding time series data of a target by the encoder, and the scale of the time series data of the target is changed by the request source.

15. The information providing device of claim 11, wherein receiving the query from the request source includes: receiving time series data of a target from the request source; changing a scale of the time series data of the target; and encoding the changed time series data of the target with the encoder to generate the query.

16. The information providing device according to claim 11, wherein returning the time series data to the requester as related time series data includes: changing a scale of the related time series data; and returning the changed related time series data to the requester.

17. An information gathering method, comprising: a computer executing the steps of: acquiring time series data showing state transitions of one or more agents; converting the acquired time series data into a feature vector using an encoder configured to encode the data by incorporating information at multiple times; and storing the acquired feature vector in a database in association with the time series data.

18. A method of providing information by a computer connected to a database that stores feature vectors associated with time series data, wherein the time series data indicates state transitions of one or more agents, and the feature vectors are obtained by converting the time series data using an encoder configured to encode information at multiple times, and the computer performs the following operations: accepting a query to the database from a requesting party; extracting a feature vector that matches the query from the database as a related feature vector; and returning time series data associated with the extracted related feature vector to the requesting party as related time series data.

Images