Information processing system, information processing method, and computer-readable storage medium

Abstract

Provided is an information processing system, an information processing method, and a computer-readable storage medium that present simulation results to a user so that the user can easily study defect data. The information processing system includes: an acquisition unit that acquires a plurality of simulation data corresponding to a plurality of simulation conditions or a plurality of experimental data corresponding to a plurality of experimental conditions; an extraction unit that extracts a plurality of defect data from the plurality of simulation data or the plurality of experimental data; a classification unit that classifies the plurality of defect data into a plurality of groups based on feature data calculated from the plurality of defect data; and a presentation unit that presents representative data extracted from each group to a user visually.

Description

Technical Field

[0001] This disclosure relates to an information processing system, information processing method, and program, and more particularly to a technique for displaying simulation results or experimental results to a user. Background Technology

[0002] Japanese Patent No. 6082759 discloses an information processing system for displaying simulation results. Summary of the Invention

[0003] Results data representing simulation or experimental outcomes sometimes include defective data (defective data) indicating instances where system requirements are not met. In such cases, an information processing system is desired that provides prompts regarding the simulation results to facilitate user investigation related to the defective data.

[0004] This disclosure was made to solve such a problem, and its purpose is to provide an information processing system, information processing method and program that provides prompts on simulation results or experimental results so that users can easily study defective data.

[0005] The information processing system in this embodiment includes: an acquisition unit that acquires multiple simulation data corresponding to multiple simulation conditions or multiple experimental data corresponding to multiple experimental conditions; an extraction unit that extracts multiple defect data from the multiple simulation data or the multiple experimental data; a classification unit that classifies the multiple defect data into multiple groups based on feature data calculated according to each of the multiple defect data; and a prompting unit that visually prompts the user with representative data extracted from each group.

[0006] In the information processing method of this embodiment, the computer obtains multiple simulation data corresponding to multiple simulation conditions or multiple experimental data corresponding to multiple experimental conditions, extracts multiple defect data from the multiple simulation data or multiple experimental data, classifies the multiple defect data into multiple groups based on the feature data calculated according to each of the multiple defect data, and visually prompts the user with representative data extracted from each group.

[0007] The program in this embodiment causes the computer to perform the following information processing method, in which multiple simulation data corresponding to multiple simulation conditions or multiple experimental data corresponding to multiple experimental conditions are obtained, multiple defect data are extracted from the multiple simulation data or the multiple experimental data, the multiple defect data are classified into multiple groups based on feature data calculated according to each of the multiple defect data, and representative data extracted from each group are visually presented to the user.

[0008] According to this disclosure, an information processing system, information processing method, and program are available to provide prompts on simulation or experimental results so that users can easily study defective data.

[0009] The above and other objects, features and advantages of this disclosure will be more fully understood from the detailed description given below and the accompanying drawings, which are given by way of example only, and should therefore not be considered as limitations on this disclosure. Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating the structure of the information processing apparatus according to Embodiment 1.

[0011] Figure 2 This is a flowchart illustrating the flow of the information processing method involved in Implementation Method 1.

[0012] Figure 3 A schematic diagram to represent the outline of a simulation of robots passing each other.

[0013] Figure 4 A schematic diagram showing the robot's starting point and destination.

[0014] Figure 5 A summary diagram to illustrate the overview of search-based testing.

[0015] Figure 6 A schematic diagram illustrating the outline of the feature data extraction method.

[0016] Figure 7 This is a summary diagram illustrating the display of representative data.

[0017] Figure 8 This is a summary diagram illustrating the display of clustering results. Detailed Implementation

[0018] Hereinafter, although the present invention will be described through embodiments, it is not intended to limit the invention to these embodiments. Furthermore, not all structures described in the embodiments are necessarily necessary means to solve the problem.

[0019] Implementation Method 1

[0020] Hereinafter, the information processing apparatus according to Embodiment 1 will be described with reference to the accompanying drawings. Figure 1This is a block diagram illustrating the structure of the information processing apparatus 100 according to Embodiment 1. The information processing apparatus 100 is an example of an information processing system. The information processing apparatus 100 may also be an edge terminal. A system that performs processing within an edge terminal may also be included in the information processing system. As explained last, the information processing system may also include a server.

[0021] The information processing device 100 includes an acquisition unit 110, a defect data extraction unit 120, a calculation unit 130, a classification unit 140, a representative data extraction unit 150, and a prompting unit 160. Furthermore, the information processing device 100 includes a processor and a memory (not shown). By executing a program through the processor, the information processing device 100 functions as the acquisition unit 110, the defect data extraction unit 120, the calculation unit 130, the classification unit 140, the representative data extraction unit 150, and the prompting unit 160.

[0022] The acquisition unit 110 acquires multiple simulation data corresponding to multiple simulation conditions, or multiple experimental data corresponding to multiple experimental conditions. Although the following description focuses on the case where the acquisition unit 110 acquires multiple simulation data, the acquisition unit 110 may also acquire real-world experimental data instead of simulation data. If defect data is included in the multiple simulation data, the defect data is extracted by the defect data extraction unit 120, which will be described later.

[0023] The acquisition unit 110 can also acquire multiple simulated data sets by performing search-based tests aimed at finding data (defect data) that violate system requirements. This allows the user to efficiently analyze the defect data. As a result of the search-based tests, the acquisition unit 110 can also receive information related to whether each simulated data set meets system requirements. Search-based testing can efficiently search for violations of system requirements through optimization.

[0024] In addition, the acquisition unit 110 can also perform tests other than search-based tests. For example, the acquisition unit 110 can also perform tests with simulation conditions set at equal intervals (e.g., grid tests) or tests with simulation conditions arbitrarily set by the user.

[0025] Specifically, the acquisition unit 110 acquires simulation data related to the movement of multiple mobile bodies (e.g., robot 1, robot 2, robot 3). In this case, the starting point and starting time of each mobile body can also be set as simulation conditions. The defect data extraction unit 120, described later, can, for example, set simulation data in which a predetermined mobile body arrives at its destination before a target time (e.g., 120 seconds after the start of the simulation) as successful data, and set other data as defect data.

[0026] In the following description, the focus will be on the case where the acquisition unit 110 acquires simulation data related to the movement of multiple moving bodies. However, the data acquired by the acquisition unit 110 is not limited to data related to the movement of the moving bodies. The data acquired by the acquisition unit 110 can be any data used to verify whether the operation of the control system meets the system requirements under various conditions. The control that is being simulated can be, for example, engine control, merging control of an autonomous vehicle, or smart grid control.

[0027] In the case of engine control, the acquisition unit 110 acquires data (e.g., the input trajectory of the accelerator and brake) to verify the deviation between the target value and the measured value of the vehicle speed, based on the inputs of the accelerator and brake. In the case of merging control of an autonomous vehicle, the acquisition unit 110 acquires data to verify collisions at various initial positions of the autonomous vehicle (e.g., the movement trajectory of the autonomous vehicle). In the case of smart grid control, the acquisition unit 110 acquires data (e.g., power consumption, power generation) to verify power surplus or deficiency in various facilities under various supply and demand conditions.

[0028] As described above, the defect data extraction unit 120 extracts multiple defect data from multiple simulation data or multiple experimental data. Hereinafter, the defect data extraction unit 120 will sometimes be referred to simply as the extraction unit. Sometimes, multiple defect data are included in the simulation data related to the movement of multiple moving bodies, derived from the degrees of freedom of the individual movements of the multiple moving bodies. Specifically, when a moving body passes by other moving bodies, it may sometimes collide, stop, or detour in a manner that is not necessary.

[0029] Specifically, the defect data extraction unit 120 sets simulated data that violates system requirements as defect data. System requirements can be arbitrarily determined by the user. For example, system requirements could be the condition that a predetermined moving object arrives at its destination within a predetermined time. Information related to whether system requirements are violated can be included in the results of a search-based test. Defect data can be judged based on thresholds or on whether conditions expressed in STL (Signal Temporal Logic) form are met.

[0030] Furthermore, the defect data extraction unit 120 may extract defect data based on the requirements of a subsystem, rather than the requirements of the entire system. For example, sometimes, in the system requirement of enabling multiple mobile entities to reach their destination within a time period, there is a subsystem requirement that the position inference error of each autonomous mobile entity is below a threshold. In such cases, the defect data extraction unit 120 can designate simulation data where the position inference error of each autonomous mobile entity is below the threshold as successful data, and designate simulation data other than successful data as defect data.

[0031] The calculation unit 130 calculates feature data based on each of the multiple defect data. Feature data is also called feature parameters. The feature data is used to determine the similarity between defect data when the classification unit 140 performs clustering, as described later. The calculation unit 130 may also calculate feature data by, for example, extracting the position coordinates (e.g., x-coordinate of robot 1, y-coordinate of robot 1, x-coordinate of robot 2, y-coordinate of robot 2, x-coordinate of robot 3, y-coordinate of robot 3) of multiple moving bodies at each time point (e.g., 0 seconds, 12 seconds, 24 seconds, 36 seconds, 48 ​​seconds, 60 seconds, 72 seconds, 84 seconds, 96 seconds, 108 seconds, 120 seconds after the start of the simulation) and arranging (connecting) the extracted position coordinates.

[0032] Alternatively, the calculation unit 130 may use univariate data (e.g., the velocity data of robot 1) as feature data instead of multivariate data. Furthermore, the calculation unit 130 may also use data other than time-series data (e.g., the x and y coordinates of the final positions of each robot) as feature data.

[0033] The method for calculating feature data is not limited to the method described above, which involves extracting and concatenating data values ​​from multiple time points. The calculation unit 130 can also use methods such as sparse coding, wavelet transform, shapelet transform, singular spectral decomposition, and nonnegative matrix decomposition to calculate feature data.

[0034] The classification unit 140 classifies the multiple defect data into multiple groups based on feature data calculated for each of the multiple defect data. The classification unit 140 groups similar defect data groups through clustering. Clustering is performed, for example, by implementing the Ward method. The number of clusters can be set by the user or automatically determined based on indicators such as inter-cluster distance and intra-cluster data similarity.

[0035] The clustering method implemented by the classification department 140 is not limited to Ward's method (sum of squared deviations). Hierarchical clustering using distance metrics different from those used in Ward's method can also be performed. Furthermore, non-hierarchical clustering methods such as k-means (k-means clustering algorithm) and DBSCAN (Density-based spatial clustering of applications with noise) can also be performed.

[0036] The representative data extraction unit 150 extracts representative data from each group. Representative data, also known as core data, refers to the data that forms the core of each group. Specifically, the representative data extraction unit 150 processes all data within each group as candidates for representative data, takes the difference (error) between each candidate and every other data point within the group, and calculates the average of these differences (also known as the surrounding error). The surrounding error is also called the error average. Then, the representative data extraction unit 150 determines the candidate with the smallest surrounding error as the representative data.

[0037] Furthermore, the representative data extraction unit 150 can also extract representative data based on the median or center of gravity of the data within each group. Additionally, the representative data extraction unit 150 can also extract representative data based on the degree of defect in the data within each group. For example, the representative data extraction unit 150 can also set the data with the highest degree of defect as the representative data.

[0038] The prompting unit 160 visually presents representative data extracted from each group to the user. This allows the user to easily grasp the entire defect data. The prompting unit 160 can also further prompt the user regarding the classification results (clustering data) from the classification unit 140. This allows the user to determine whether the classification was appropriate and to accurately address the defect data. At this time, the prompting unit 160 can also emphasize data with large differences from the representative data for each group (e.g., data with surrounding errors exceeding a threshold). Data with large differences from the representative data is considered data that requires user confirmation, preventing data from being overlooked. In addition to representative data and classification results, the prompting unit 160 can also provide a summary table of the defect data. The user can confirm the original defect data as needed.

[0039] When providing results, the prompting unit 160 does not need to display the original data, clustering results, or all representative data of the defect data. The prompting unit 160 may also only prompt the user with the data they require. Furthermore, the representative data prompted is not limited to time-varying signal data. The representative data can be any information associated with the ID of the extracted representative data, and can be, for example, a dynamic image or a chart other than signal data.

[0040] The prompt unit 160 can also display the screen on a display device such as a monitor (not shown). The user can check the screen and perform actions such as determining the cause of a defect and correcting the system.

[0041] Next, refer to Figure 2 The information processing method involved in Implementation Method 1 will be described. Specifically, a case will be described in which three robots are simulated passing each other and a search-based test verification is performed. Figure 3 This is a schematic diagram to illustrate the simulation. Robots 1, 2, and 3 cross each other at a T-shaped passage (corner) A. Symbol T1 represents the movement trajectory of robot 1, symbol T2 represents the movement trajectory of robot 2, and symbol T3 represents the movement trajectory of robot 3.

[0042] Reference Figure 4 This section describes the starting and ending points of robots 1, 2, and 3. The positions of robots 1, 2, and 3 indicate their starting points. Symbol T1 schematically represents the movement trajectory of robot 1, with the top of the arrow indicating its destination. Symbol T2 schematically represents the movement trajectory of robot 2, with the top of the arrow indicating its destination. Symbol T3 schematically represents the movement trajectory of robot 3, with the top of the arrow indicating its destination.

[0043] The simulation time is 120 seconds, with each time point represented by t = 0 to 120 [s]. The starting time of robot 1 is fixed at t = 16 [s]. The starting time of robot 2 is selected from a time point between t = 16 [s] and t = 26 [s]. The starting time of robot 3 is selected from a time point between t = 7 [s] and t = 37 [s].

[0044] The system requirements are that either robot 1 or robot 2 must reach the destination within 120 seconds, and robot 3 must also reach the destination within 120 seconds. The inventors conducted search-based tests to search for scenarios that violated the system requirements, and performed 500 simulations.

[0045] The start times of Robot 2 and Robot 3 correspond to the simulation conditions. When the horizontal axis is set to the start time of Robot 2 and the vertical axis to the start time of Robot 3, points corresponding to each of the 500 simulation conditions can be plotted. In search-based testing, the distance between adjacent points is generally not fixed. On the other hand, in mesh testing, the distance between adjacent points is generally fixed. Search-based testing can search for defect data more efficiently than mesh testing.

[0046] Figure 5 A summary diagram to illustrate the overview of search-based testing. Figure 5 The system shown includes an SBT (Search-Based Testing) unit 111 and a target model 112. Alternatively, the acquisition unit 110 of the information processing device 100 may also include an SBT unit 111 and a target model 112.

[0047] The SBT unit 111 uses the target model 112 to perform search-based testing. The target model 112 includes a model / simulator 1121 and control software 1122 for controlling the model / simulator.

[0048] In SBT unit 111, the aforementioned system requirements are set, and a test scenario is input to SBT unit 111. The test scenario includes simulation conditions. Target model 112 receives the test scenario and returns an output signal representing the simulation result. The output signal may also include signals representing the time change of the position coordinates of robot 1, signals representing the time change of the position coordinates of robot 2, and signals representing the time change of the position coordinates of robot 3.

[0049] Return to Figure 2 The information processing method according to Embodiment 1 will be described below. First, the acquisition unit 110 of the information processing device 100 acquires the results of a search-based test verification implemented by simulation (step S101). Next, the defect data extraction unit 120 of the information processing device 100 extracts multiple defect data (e.g., data on a predetermined robot failing to reach its destination) from 500 simulation data (step S102). Assume that among the aforementioned 500 simulation data, there are 134 defect data. There is a problem that a large amount of time is required to grasp all defect events based on these 134 defect data. On the other hand, among the 134 defect data, there is also a large number of similar data. Therefore, it is desirable to remove similar data to grasp defect events.

[0050] Next, the calculation unit 130 of the information processing device 100 calculates feature data based on each of the 134 defect data (step S103). Specifically, the calculation unit 130 calculates feature data by extracting the position information (also called discrete values) at each of the discrete multiple time points from the movement trajectory of each robot and arranging (linking) them.

[0051] Figure 6 This is a diagram used to illustrate an example of feature data. Figure 6 The left side shows an example of simulation data, where symbol T1 represents the trajectory of robot 1, symbol T2 represents the trajectory of robot 2, and symbol T3 represents the trajectory of robot 3. Additionally, Figure 6 The horizontal axis represents the x-direction; the further to the right, the larger the x-coordinate. Figure 6 The vertical axis represents the y-direction; the higher up the axis, the larger the y-coordinate.

[0052] exist Figure 6 On the right side, the movement trajectories T1 of robot 1, T2 of robot 2, and T3 of robot 3 are shown in magnification. The hollow circles represent the positions of the robots at various time points (t = 0, t = 12, t = 24, ..., t = 120). Sometimes, the hollow circles are omitted for ease of observation. The hollow circles may also overlap when the robots have stopped.

[0053] The computing unit 130 calculates feature data by extracting and concatenating the (x, y) coordinates of robot 1, robot 2, and robot 3 at various time points. The feature data preserves the temporal sequence and multivariate correlation of the output signal. Since the (x, y) coordinates of each robot are extracted every 12 seconds, and the (x, y) coordinates of the three robots are extracted at 11 time points, the feature data has a dimension of 3 × 2 × 11 = 66 dimensions.

[0054] Return to Figure 2 Continuing the explanation, next, the classification unit 140 of the information processing device 100 clusters the 134 defect data based on the feature data of each of them (step S104). Specifically, the classification unit 140 specifies the number of clusters as 16 and performs clustering using the Ward method. The number of clusters is set to a number that is easy for the user to confirm.

[0055] Next, the representative data extraction unit 150 of the information processing device 100 extracts representative data from each of the 16 groups (step S105). The representative data can be, for example, the data that minimizes the average difference (surrounding error) between the data and each of the other data in the group.

[0056] Next, the prompting unit 160 of the information processing device 100 visually prompts the user with 134 raw data, clustered data, and representative data extracted from each group (step S106). Figure 7 An example of a screen displaying representative data is shown. Figure 7 Sixteen representative data sets are shown. Each representative data set contains an ID11 for identifying the simulated data and a graph 12 representing the movement trajectory of each robot. In addition to the graph 12, ID11 can be further correlated with dynamic images and other data.

[0057] Figure 12 shows the shape of the T-shaped channel, the movement trajectory of robot 1, robot 2, and robot 3. The horizontal axis of Figure 12 represents the x-direction, and the vertical axis represents the y-direction.

[0058] Figure 8 This represents the classification results (clustering data) for Classification Division 140. 134 defect data points were classified into 16 groups. Each group is enclosed in a box. The first data point of each group represents the representative data. Representative data can also be highlighted using red boxes, etc. Furthermore, defect data within each group can be arranged in ascending order of peripheral error, as described above.

[0059] As already described, data with significant differences from the representative data (e.g., data with surrounding errors exceeding a threshold) can also be highlighted. For example, data with significant differences from the representative data can be highlighted using a blue box or similar method. Although data may sometimes be missed during the extraction of representative data, users can confirm data with significant surrounding errors through the displayed screen.

[0060] Users can confirm the information prompted by the prompting unit 160. Based on the prompted information, users can determine the data related to the defect event that should be addressed. If the cause of the defect cannot be determined solely by the trajectory of each robot, users can also confirm the dynamic image and markers associated with ID11 (e.g., indicating the presence or absence of robot collisions).

[0061] Finally, the effects of the information processing device involved in Implementation Method 1 will be explained. There are cases where tests are conducted through simulation or experimentation to verify whether the control system has been created according to the developer's requirements. If the test results contain defective data, the developer needs to confirm the system's operation and determine the necessity of correction, identify the cause of the defect, and correct the system.

[0062] When multiple defect data exist, if events associated with each defect data are corrected sequentially, the method for one defect event may be opposite to the method for others. Therefore, for effective correction, it is preferable to consider a method where developers simultaneously identify and resolve multiple defect events. However, with a large amount of defect data, parsing requires significant time. On the other hand, if defect data is randomly extracted to reduce the amount of defect data to be identified, there is a risk of overlooking defect events.

[0063] The information processing apparatus described in Implementation 1, by clustering large amounts of defect data and extracting representative data (core data), can significantly reduce the amount of defect data that developers need to verify while minimizing the number of missed defect events. This allows for efficient and effective identification of defect events, thereby shortening the time required for system correction.

[0064] According to the information processing apparatus of Embodiment 1, the user can only confirm defective data. Although there is a possibility of overlooking defective data when the displayed results include normal data, this possibility can be reduced according to Embodiment 1. Furthermore, if the feature data used in clustering is preset, even users without expertise related to the control system being targeted can determine the cause of the defect.

[0065] The information processing system may not necessarily have all functional elements integrated into the information processing device 100. For example, the function of the representative data extraction unit 150 may also be performed by the computing unit of a server connected to the information processing device 100 via a network. In this case, the server sends representative data to the information processing device 100. The prompting unit 160 of the information processing device 100 uses the received representative data to provide the same prompts as in Embodiment 1. Thus, the information processing system can also be configured to include a server and the information processing device 100. The aforementioned processor and memory may be located in the server, or they may be located in both the information processing device 100 and the server.

[0066] In the above examples, the program includes a set of instructions (or software code) that, when read by a computer, causes the computer to perform one or more functions described in the implementation. The program may be stored on a non-transitory computer-readable medium or a tangible storage medium. By way of example, and not limitation, computer-readable media or tangible storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, cassette tape, magnetic tape, disk storage, or other magnetic storage devices. The program may also be transmitted on a transient computer-readable medium or a communication medium. By way of example, and not limitation, transient computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagation signals.

[0067] Furthermore, the present invention is not limited to the above-described embodiments, and appropriate modifications can be made without departing from the spirit of the invention.

[0068] It will be apparent from this disclosure that embodiments of the present disclosure can be modified in a variety of ways. Such modifications should not be considered as departing from the spirit and scope of the present disclosure, and all such modifications should be included within the scope of the invention by those skilled in the art.

Claims

1. An information processing system, comprising: The acquisition unit acquires multiple simulation data corresponding to multiple simulation conditions or multiple experimental data corresponding to multiple experimental conditions. The extraction unit extracts multiple defect data from the multiple simulation data or the multiple experimental data; The classification department classifies the multiple defect data into multiple groups based on feature data calculated according to each of the multiple defect data. The prompting department will visually present representative data extracted from each group to the user. The acquisition unit acquires simulation data or experimental data related to the movement of multiple moving bodies. The extraction unit extracts the multiple defect data originating from the respective degrees of freedom of movement of the multiple moving bodies. The prompting section also displays the classification results from the classification section, and for each group, highlights defect data that differ significantly from the representative data. The acquisition unit acquires the multiple simulated data by conducting search-based tests aimed at finding data that violates system requirements. The extraction unit extracts the multiple defect data that violate the system requirements. The defect data within each group are arranged in ascending order of surrounding error.

2. The information processing system as described in claim 1, wherein, It also includes a calculation unit that performs calculations on the feature data by extracting the position coordinates of the multiple moving bodies at each time point from the various defect data and arranging the extracted position coordinates.

3. An information processing method, wherein, The computer acquires multiple simulation data points corresponding to multiple simulation conditions or multiple experimental data points corresponding to multiple experimental conditions. Extract multiple defect data from the multiple simulation data or the multiple experimental data. Based on the feature data calculated according to each of the plurality of defect data, the plurality of defect data are classified into multiple groups. Representative data extracted from each group will be visually presented to the user. Obtain simulation or experimental data related to the movement of multiple moving objects. The defect data originating from the individual degrees of freedom of the multiple moving bodies are extracted. It also displays the categorized results and highlights defective data that differ significantly from the representative data for each group. The aforementioned simulated data were obtained by conducting search-based tests aimed at finding data that violated system requirements. Extract the multiple defect data that violate the system requirements. The defect data within each group are arranged in ascending order of surrounding error.

4. A computer-readable storage medium storing a program that causes a computer to perform the following information processing method, In the information processing method, Obtain multiple simulation data corresponding to multiple simulation conditions or multiple experimental data corresponding to multiple experimental conditions. Extract multiple defect data from the multiple simulation data or the multiple experimental data. Based on the feature data calculated according to each of the plurality of defect data, the plurality of defect data are classified into multiple groups. Representative data extracted from each group will be visually presented to the user. Obtain simulation or experimental data related to the movement of multiple moving objects. The defect data originating from the individual degrees of freedom of the multiple moving bodies are extracted. It also displays the categorized results and highlights defective data that differ significantly from the representative data for each group. The aforementioned simulated data were obtained by conducting search-based tests aimed at finding data that violated system requirements. Extract the multiple defect data that violate the system requirements. The defect data within each group are arranged in ascending order of surrounding error.

Images