Information processing method, information processing program, and information processing device

Abstract

The information processing method according to the present disclosure comprises: an application step for applying, to a robot system to be evaluated, an evaluation scene including a context derived from a mixed reality in which a virtual environment and a real environment are combined; an evaluation step for evaluating the behavior of the robot system to which the evaluation scene is applied; and a switching step for switching the evaluation scene applied to the robot system to another evaluation scene in the plurality of evaluation scenes according to the evaluation of the behavior of the robot system in the evaluation scene under the application in the evaluation step.

Description

Technical Field

[0001] This disclosure relates to information processing methods, information processing procedures, and information processing apparatus. Background Technology

[0002] Evaluating robotic systems is challenging and requires significant cost and time. Robots are complex systems of software and hardware that interact with their environment in real time, making individual tests inherently unreproducible and incomplete. Furthermore, hardware performance and characteristics are easily altered due to degradation and other factors, and these effects propagate non-linearly, making their impact difficult to predict. Therefore, robotic system testing needs to be repeated over extended periods under various conditions. Moreover, real-world constraints, where a physical entity can only occupy one state at a time, hinder the efficiency of testing.

[0003] In response, numerous attempts have been made to evaluate robotic systems in virtual spaces. However, in most cases, actually accomplishing everything is unacceptable unless the differences between virtual simulation and reality can be resolved. In recent years, mixed reality (MR) systems, which combine real and virtual elements, have also been used for evaluation purposes.

[0004] [List of Citations]

[0005] [Non-patent literature]

[0006] [Non-patent document 1] Ian Yen-Hung Chen, Bruce MacDonald, Burkhard Wunsche, "Mixed Reality Simulation for Mobile Robots", [online], 2009, IEEE, 2009 IEEEInternational Conference on Robotics and Automation, Kobe, Japan, 2009, pp.232-237, [searched June 10, 2024], Internet, https: / / ieeexplore.ieee.org / abstract / document / 5152325 Summary of the Invention

[0007] [Technical Issues]

[0008] However, even when performing robot system evaluations using mixed reality systems, the real-world constraint that a physical entity can only occupy one state at a time cannot be avoided because the evaluations are performed using actual existing robot systems.

[0009] Therefore, this disclosure aims to provide information processing methods, procedures, and apparatus that can more effectively perform robot system evaluations.

[0010] [Solution to the problem]

[0011] The information processing method according to this disclosure includes: an application step, applying an evaluation scenario, including a mixed reality scenario combining a virtual environment and a real environment, to a robot system to be evaluated; an evaluation step, evaluating the behavior of the robot system to which the evaluation scenario has been applied; and a switching step, switching the evaluation scenario applied to the robot system to another evaluation scenario among multiple evaluation scenarios based on the evaluation of the robot system's behavior in the currently applied evaluation scenario by the evaluation step. Attached Figure Description

[0012] Figure 1 This is a functional block diagram used to explain an example of the functionality of a robot evaluation system using MR that can be applied to this disclosure.

[0013] Figure 2 This is a block diagram illustrating an example hardware configuration applicable to this disclosure.

[0014] Figure 3 This is a schematic diagram illustrating the concept of a robot evaluation system that uses mixed reality to evaluate robot systems based on existing technology.

[0015] Figure 4 This is a schematic diagram used to explain an example of the evaluation of a turning system based on existing technology.

[0016] Figure 5 A schematic diagram of a simplified test route example is shown.

[0017] Figure 6 This is a more detailed schematic diagram illustrating an example configuration of a robot evaluation system using mixed reality based on existing technology.

[0018] Figure 7 This is a schematic diagram used to illustrate the use of mixed reality in existing robot evaluation systems.

[0019] Figure 8 This is a schematic diagram showing in more detail an example configuration of a robot evaluation system using mixed reality according to an implementation method.

[0020] Figure 9 This is a schematic diagram showing an example of a test route.

[0021] Figure 10 This is a block diagram illustrating an example configuration of a robot evaluation system according to a first variation of an implementation.

[0022] Figure 11 This is a block diagram illustrating an example configuration of a robot evaluation system according to a second variation of an implementation.

[0023] Figure 12 This is a block diagram illustrating an example configuration of a robot evaluation system according to a third variation of the implementation.

[0024] Figure 13 This is a block diagram illustrating an example of a schematic configuration of a vehicle control system.

[0025] Figure 14 This is an explanatory diagram showing an example of the installation location of the vehicle exterior information detection unit and the imaging unit. Detailed Implementation

[0026] In the following description, embodiments of the present disclosure will be presented in detail based on the accompanying drawings. In the embodiments described below, the same parts are given the same reference numerals to omit redundant descriptions.

[0027] In the following text, embodiments of the present disclosure will be described in the following order.

[0028] 1. Overview of the Invention

[0029] 2. Regarding existing technology

[0030] 2-1. Overview of Robotic System Evaluation Using Existing Technologies

[0031] 2-2. Details of Evaluation of Existing Robotic Systems

[0032] 2-3. The effectiveness of using mixed reality for evaluating robotic systems

[0033] 3. Implementation methods of this disclosure

[0034] 3-1. Processes related to improving time efficiency

[0035] 3-2. Treatments related to improving space efficiency

[0036] 4. A first variation of the embodiments of this disclosure

[0037] 5. A second variation of the embodiments of this disclosure

[0038] 6. A third variation of the embodiments of the present invention

[0039] 7. Application Examples of the Embodiments of this Disclosure

[0040] (1. Overview of the present invention)

[0041] This disclosure relates to techniques for evaluating robotic systems using mixed reality (MR). More specifically, in the techniques according to this disclosure, multiple scenarios are prepared for application to the robotic system to be evaluated. Then, based on the behavior of the robotic system, the scenarios applied to the robotic system to be evaluated are switched using the multiple prepared scenarios.

[0042] By applying the techniques according to this disclosure to robot system evaluation, the efficiency of time and space utilization for evaluation implementation can be improved, enabling robot system evaluation to be performed more efficiently.

[0043] Figure 1 This is a functional block diagram illustrating an example of the functionality of a robot evaluation system applicable to the MR of this disclosure. Figure 1 In this system, the robot evaluation system 1 includes an adaptive MR generation unit 10, a sensor 11, a storage unit 12, and a drive control unit 13. The drive unit 14 drives the robot to be evaluated according to the control of the drive control unit 13.

[0044] There are no particular restrictions on the type of robot to be evaluated. Examples of robot systems to be evaluated include manipulators, mobile robot systems such as autonomous vehicles and drones, humanoid robot systems that combine manipulators and mobile robot systems, and mobile robot systems such as animal robot types. In the following text, "robot system to be evaluated" can be simply described as "robot system".

[0045] exist Figure 1 In this context, storage unit 12 stores evaluation scenarios Snr used in the robot system. Storage unit 12 can store multiple evaluation scenarios Snr.

[0046] Evaluation scenario Snr includes mixed reality (MR) based scenarios where virtual and real environments are combined. In this context, the scenario can include concepts such as situations and scenes within a specific environment. Evaluation scenario Snr may also include sensor converters and motion converters. The sensor converters convert sensor outputs from both the virtual and real environments into sensor outputs to be presented to the robotic system based on the scenario. The motion converters, based on the mixed reality, convert the robotic system's actions into actions in the virtual and real environments, respectively. Evaluation scenario Snr may also include a virtual environment tailored to the scenario.

[0047] The adaptive MR generation unit 10 generates mixed reality based on the evaluation scene Snr stored in the storage unit 12. More specifically, the adaptive MR generation unit 10 generates virtual reality, sensor converters, and motion converters based on the evaluation scene Snr and the context for application to the robot system. In this way, the adaptive MR generation unit 10 can generate the evaluation scene Snr. The adaptive MR generation unit 10 uses the motion converters to convert the motion output at the output of the drive control unit 13 into motion output av in the virtual environment and motion output ar in the real environment through the robot system.

[0048] The adaptive MR generation unit 10 transmits the state Sv of the robot system in the virtual environment to the sensor 11 based on the context of the evaluation scenario Snr. The sensor 11 also receives the state Sr of the robot system in the real environment as input. The states Sv and Sr input to the sensor 11 are converted by the sensor converter into the state St corresponding to the sensor output, which is then presented to the robot system.

[0049] The drive control unit 13 controls the drive unit 14 according to the state St and transmits the action output corresponding to the control of the drive unit 14 to the adaptive MR generation unit 10.

[0050] The adaptive MR generation unit 10 records the system's inputs and outputs, including the robot system's inputs, and evaluates the recorded information. Based on this evaluation result, the adaptive MR generation unit 10 selects an evaluation scene Snr stored in the storage unit 12 and switches the currently applied evaluation scene Snr to the selected evaluation scene Snr. In this disclosure, this configuration allows for adaptive switching of the evaluation scene Snr used to evaluate the robot system.

[0051] Among the components constituting the robot evaluation system 1, at least the adaptive MR generation unit 10 can be implemented by executing an information processing program according to this disclosure on a CPU. Furthermore, some functions of the sensor 11 and the evaluation scene Snr stored in the storage unit 12 can be implemented by executing the information processing program on the CPU.

[0052] Figure 2 This is a block diagram illustrating an example hardware configuration applicable to this disclosure. Figure 2 In the robot evaluation system 1, there are CPU (central processing unit) 2000, ROM (read-only memory) 2001, RAM (random access memory) 2002, storage device 2003, UI (user interface) unit 2004, data I / F 2005, sensor I / F 2010, and control I / F 2020, and these components are connected to each other via bus 2006 so that they can communicate with each other.

[0053] In this way, the robot evaluation system 1 can be implemented by adding sensor I / F2010 and control I / F2020 to a general-purpose computer configuration. In this configuration, sensor I / F2010 and control I / F2020 can share data I / F2005.

[0054] Storage device 2003 is a non-volatile storage medium such as flash memory or hard disk. CPU 2000 controls the overall operation of the robot evaluation system 1 using RAM 2002 as its working memory, based on the program stored in ROM 2001 and storage device 2003. Storage device 2003 corresponds to the aforementioned storage unit 12 and can store multiple evaluation scenes Snr.

[0055] UI unit 2004 includes an input unit for accepting user operations and a display unit for presenting information to the user and providing an interface. UI unit 2004 can be configured using a touch panel in which the input unit and the display unit are integrated.

[0056] The Data I / F2005 is an interface for inputting and outputting data to external devices. As the Data I / F2005, it can be compatible with USB (Universal Serial Bus) connectors, Bluetooth (registered trademark), and other technologies. The Data I / F2005 can also control communication with networks such as the Internet or LAN (Local Area Network).

[0057] Sensor I / F 2010 is an interface for transmitting and receiving data, commands, status, etc., with sensor 11. Sensor 11 can be a camera or a ranging device such as LiDAR (light detection and ranging, laser imaging detection and ranging). Control I / F 2020 is an interface for sending and receiving data, commands, status, etc., with drive unit 14.

[0058] In the robot evaluation system 1, for example, the CPU 2000 configures the aforementioned adaptive MR generation unit 10 as a module in the main storage area of ​​RAM 2002 by executing an information processing program according to the embodiment. The information processing program can be obtained from an external source via a communication network through communication via data I / F 2005, or from a storage medium connected to data I / F 2005, and is installed on the robot evaluation system 1.

[0059] (2. Regarding the prior art)

[0060] This document will describe prior art in relation to the technology disclosed herein.

[0061] (2-1. Overview of Robotic System Evaluation Using Existing Technologies)

[0062] Evaluating robotic systems is challenging and requires significant cost and time in current technologies. Robots are complexes of software and hardware that interact with their environment in real time, making individual tests inherently unreproducible and incomplete. Furthermore, hardware performance and characteristics are easily altered due to degradation and other factors, and these effects propagate non-linearly, making their impact difficult to predict. Therefore, robotic system testing needs to be performed repeatedly over extended periods under various conditions.

[0063] Furthermore, due to real-world constraints (where physical entities can only occupy one state at a time and cannot rewind time or remotely communicate to different locations), the effectiveness of testing inevitably decreases. Additionally, when conducting rigorous evaluations, it is necessary to consider costs such as failures and damages during incidents, and the costs and constraints for ensuring security are also important. Therefore, even software update barriers requiring re-QA (Quality Assurance) testing for already deployed systems can lead to very high costs.

[0064] To address these issues, numerous attempts have been made to evaluate robotic systems in virtual spaces. However, in most cases, actually accomplishing everything is unacceptable unless the differences between virtual simulation and reality are resolved.

[0065] In recent years, mixed reality (MR) systems that combine real and virtual elements have been increasingly used for evaluation purposes (e.g., non-patent literature 1).

[0066] Figure 3 This is a schematic diagram illustrating the concept of a robot evaluation system that uses mixed reality to evaluate robot systems based on existing technology. Figure 3 In this system, the robot system 500 automatically controls the manipulator 550. Here, as an evaluation object, we consider the operation of the manipulator 550 holding a holding target object 551 that does not exist in actual space.

[0067] As the operating environment for the robot system 500, a virtual environment 510 and a real environment 520 are prepared. The virtual environment 510 is provided to simulate the operation of the robot system 500 in a virtual space. In this example, in the virtual environment 510, the virtual manipulator 550' of the simulated manipulator 550 virtually performs the operation of grasping the simulated target object 551'. The real environment 520 is the real space where the main body of the manipulator 550 exists.

[0068] When the robot system 500 performs the operation of manipulator 550 grasping target object 551 in real environment 520, the motion output related to the operation is reflected in real time in both virtual environment 510 and real environment 520. Based on the motion output from robot system 500, the operation of manipulator 550 is simulated in virtual environment 510, and the operation of manipulator 550' grasping and holding target object 551' in virtual environment 510 is virtually executed.

[0069] The movement of the manipulator 550' in the virtual environment 510 is detected by virtual sensors, and the sensor output is transmitted to the robot system 500 in real time as virtual sensor information. On the other hand, the movement of the manipulator 550 in the real environment 520 is detected by real sensors, and the sensor output is input to the robot system 500 in real time as sensor input.

[0070] The robot system 500 combines virtual sensor information transmitted from the virtual environment 510 with sensor input from the real environment 520 to generate sensor information (composite sensor output) for the robot system 500. Based on this sensor information and existing knowledge / context 501, the robot system 500 is controlled to perform the next operation.

[0071] In this way, the evaluation of robot system 500 using mixed reality based on existing technology has the following basic concepts. That is, when the robot system 500 to be evaluated ( Figure 3 When the manipulator 550 in the example operates in the real environment 520, the target object that interacts with the robot system 500 ( Figure 3 In the example, the grasping target object 551 exists only in the virtual environment 510. The behavior simulated in the virtual environment 510 is then given to the robot system 500 to be evaluated as sensor information, as if it were real. This forms a closed loop connecting the real environment 520 and the virtual environment 510. Therefore, for example in Figure 3 In the example, the performance of the robot system 500 can be quantitatively evaluated and is highly reproducible for grasping target objects 50 of various sizes and shapes and their behavior (smoothness, flexibility, etc.).

[0072] The robot systems mentioned here are not limited to, for example, 500 Figure 3 The robotic arm 550 shown is not commonly used in robotic systems such as mobile robot systems like autonomous vehicles and drones, human-type systems that combine both, or mobile robots such as animal robots.

[0073] However, since the evaluation is performed using a real-world robotic system, it is still impossible to avoid real-world constraints such as the inability to rewind time or transmit data over long distances to different locations.

[0074] Here, as a specific example, we consider the evaluation of a turning system in an autonomous vehicle. Figure 4 This is a schematic diagram illustrating an example of evaluating a turning system according to existing technology. Figure 4 In the test, vehicle 70 was traveling counterclockwise on test route 60 when the test was performed at a U-turn. Figure 4 The example emphasizes three U-shaped turns to be evaluated, each with three types of curvature (small, medium, and large): small corner 61a, medium corner 61b, and large corner 61c.

[0075] Here, the connections from small corners 61a and 61b to large corner 61c are roughly straight and do not require special evaluation. However, due to the structure of test track 60, it is necessary to travel through these connections. The same applies to the travel section returning to the evaluation section after passing large corner 61c. From a testing perspective, the time and other costs required to travel through these non-evaluation sections can be considered wasteful.

[0076] Figure 5 This is a schematic diagram illustrating an example of a simplified test route. Figure 5 In the test route 62, two U-turns of different sizes and two straight sections connecting the U-turns are included. For example, if there exists a... Figure 5 The test route 62 shown is considered valid for testing this type of U-turn. However, testing requirements typically vary for every use case, so there is no single optimal process that applies to all situations.

[0077] (2-2. Details of the evaluation of robot systems using existing technologies)

[0078] Figure 6 This is a schematic diagram illustrating in more detail an example configuration of a robot evaluation system using mixed reality based on existing technology.

[0079] exist Figure 6 In the robot evaluation system 1000, there are MR generation unit 110, recording / evaluation unit 120, virtual environment 130 and real environment 131, and evaluation robot system 200.

[0080] The MR generation unit 110 includes a sensor converter 111, an action converter 112, and an existing knowledge / context retention unit 113, and corresponds to the evaluation scenario Snr described above.

[0081] Sensor converter 111 calculates the sensor output to be evaluated, i.e., state St, based on the state Sv(t) detected at time t in virtual environment 130 and the state Sr(t) detected at time t in real environment 131. Motion converter 112 calculates motion outputs av(t) and ar(t) to be applied to the simulator (virtual environment 130) and real environment 131, respectively, based on the motion outputs of robot system 200. Motion output av(t) is reflected in the motion of robot system 200 in virtual environment 130. Furthermore, motion output ar(t) is reflected in the motion of robot system 200 in real environment 131.

[0082] In this way, the MR generation unit 110 is used as an application unit to apply an evaluation scenario, including a mixed reality-based context, to the robot system to be evaluated, wherein, in mixed reality, the virtual environment and the real environment are combined.

[0083] The existing knowledge / context retention unit 113 retains knowledge and parameters related to the content of the task simulated in the virtual environment 130 and the nature of the environment.

[0084] Furthermore, the recording / evaluation unit 120 includes a data logger 121 and an evaluation index 122. The data logger 121 records the system inputs / outputs and internal states of the robot system 200. The recording / evaluation unit 120 evaluates the information recorded in the data logger 121 based on the evaluation index 122, scores it, and records the score. That is, the recording / evaluation unit 120 serves as an evaluation unit for evaluating the behavior of the robot system 200.

[0085] The operation of the robot evaluation system 1000 will be described in more detail. In the real environment 131, when the action output ar(t) is output at time t, the state Sr of the system in the real environment 131 changes as shown in the following equation (1).

[0086] Sr(t+1) = f(Sr(t), ar(t))…(1)

[0087] The state Sv that simulates this situation in the virtual environment 130 can be modeled as shown in the following equation (2).

[0088] Sv(t+1) = f(Sv(t), av(t))…(2)

[0089] Sensor converter 111 calculates the sensor input of robot system 200 at time t from the states Sr(t) and Sv(t) of the real environment 131 and virtual environment 130 at time t, respectively, i.e., state St = F(Sr(t), Sv(t)). Motion converter 112 calculates the motion outputs {ar(t), av(t)} = Fa(at) to be output to the real environment 131 and virtual environment 130 respectively from the motion outputs of robot system 200 at time t.

[0090] Both sensor converter 111 and motion converter 112 are modified by the existing knowledge / context retention unit 113 and operate in a manner consistent with the tasks performed in the real environment 131 and the virtual environment 130, respectively. As a specific example, for an object grasping operation, sensor converter 111 and motion converter 112 perform calculations (such as simulation of tactile sensors) and forces acting on the virtual grasped object during grasping, and for other tasks and environments, they perform similar calculations depending on each corresponding situation.

[0091] (2-3. The effect of using mixed reality for evaluating robotic systems)

[0092] Next, we will describe the effects of using mixed reality for the evaluation of robotic systems in existing technologies.

[0093] So far, we have explained that evaluating robotic systems is difficult and requires significant cost and time, and we have already... Figure 6 The robot evaluation system 1000 described herein is an existing technology for addressing these challenges. Using an autonomous vehicle as an example, we will again illustrate specific aspects of robot system evaluation using the robot evaluation system 1000.

[0094] Consider an automated vehicle and perform tests on test route 60, such as Figure 4 As shown. The test items focus on the existence of dynamic entities other than themselves and whether their behavior can be responded to appropriately. Examples of such situations include emergency braking in response to a pedestrian jumping out, overtaking a slower vehicle in the presence of oncoming traffic, and responding when other vehicles move closer. Note that for tests that do not assume moving objects other than themselves in the environment, it is considered unnecessary to use a mixed reality system.

[0095] Without a mixed reality system, for test vehicles traveling on a route, it would be necessary to have simulated pedestrian mannequins jump out at appropriate times as the test vehicle approaches, or to have one or more vehicles driven by other drivers approach or pass by depending on the scenario, and to observe and analyze the behavior of the test vehicle in the anticipated scenario. Such testing is obviously time-consuming and labor-intensive, and difficult for testing in truly dangerous edge-case situations. Furthermore, even for overtaking or passing scenarios, in order to pass or be overtaken again after a vehicle has already passed or been overtaken once, each vehicle needs to be driven back to its initial position, making the test itself not very efficient.

[0096] Conversely, when using mixed reality for robotic evaluation systems, a virtual world (virtual environment) is first constructed to simulate the real world (real environment). (See reference) Figure 7 This will describe the use of mixed reality in existing robotic evaluation systems.

[0097] The test route 600v in the virtual environment 130 has the same size, shape, road surface characteristics, etc. as the test route 600r in the real environment, and is also co-aligned with the real environment 131. As an example, consider the case where a vehicle is near the entrance of the first corner of the track in the real environment 131. In this case, when the vehicle is placed at the same coordinate position in the virtual environment 130, calibration is performed so that the vehicle position also becomes the entrance of the first corner of the same track in the virtual environment 130.

[0098] Then, the test vehicle 610 (which is an autonomous vehicle) drives in the real environment 131, and the position, speed, and attitude information of the test vehicle 610 are transmitted in real time to the robot evaluation system 1000 and reflected in the virtual environment 130. In the virtual environment 130, in addition to the real vehicle existing in the real environment 131 (test vehicle 610'), there exists a virtual vehicle 611 generated through simulation. The position, speed, attitude, and other information of the virtual vehicle 611 are sent back to the real test vehicle 610 in real time, covering a portion of the sensor information of the test vehicle 610.

[0099] Overwriting sensor information involves, for example, superimposing an image of a virtual vehicle 611 onto the captured image if the sensor information is an image captured by a camera. Furthermore, for example, if the sensor information is point cloud data acquired via LiDAR, the point cloud data of the region corresponding to the virtual vehicle 611 is replaced with point cloud data indicating the virtual vehicle 611.

[0100] In this way, by replacing a portion of the sensor information of the test vehicle 610 with information from the virtual vehicle 611, situation 612 can be generated, which is equivalent to the presence of a (virtual) oncoming vehicle directly in front of the test vehicle for the autonomous driving system that controls the test vehicle 610 as a real vehicle.

[0101] By using this mixed reality robotic evaluation system 1000, the behavior of the test vehicle 610 to be evaluated can be assessed, particularly its response and interaction with other moving objects, even when only one test vehicle 610 exists in the real environment 131. The control of the (virtual) moving object existing in the virtual environment 130 can be performed by a human or automatically by an algorithm or software.

[0102] The advantages of this robot evaluation system 1000 using mixed reality can be listed below.

[0103] (1) Safety: Even in the event of a (virtual) accident, there is no danger to other vehicles or pedestrians, and it is also safe for the test vehicle if there is no collision with roadside obstacles in a real environment, no physical accident occurs.

[0104] (2) Synchronization: While it is not easy to coordinate the timing of multiple moving objects in a real environment, certain evaluation scenarios (Snr) only occur when their timings are well aligned. For example, events such as pedestrians begin to cross when entering an oncoming lane to overtake. In response to these, it becomes easy to achieve the desired evaluation scenario (Snr) in a virtual environment by triggering events based on the position of objects and the timing of specific actions in a simulator that includes the test vehicle.

[0105] (3) Reproducibility: In a real environment, it may be difficult to reproduce a situation under exactly the same conditions, such as an oncoming vehicle passing by in the same way as in the past. Conversely, in a virtual environment, it is relatively easy to reproduce the same situation synchronized with the real environment through methods such as replaying recorded logs. This makes it easy to compare the behavior of different algorithms and software systems and determine their superiority or inferiority. It also makes it easy to determine whether an updated system meets the same safety standards as before.

[0106] (3. Implementation of this disclosure)

[0107] The embodiments of this disclosure will now be described.

[0108] Figure 8This is a schematic diagram illustrating in more detail an example configuration of a robot evaluation system using mixed reality according to an embodiment. The robot evaluation system 100 according to the embodiment is provided with an adaptive MR generation unit 1100 including multiple MR generation units 1101, 1102, 1103, ... instead of being based on... Figure 6 The prior art robot evaluation system 1000 shown in the figure has an MR generation unit 110. The adaptive MR generation unit 1100 corresponds to the use of Figure 1 The adaptive MR generation unit 10 is described. The adaptive MR generation unit 1100 can generate MR generation unit 110n (i.e., the evaluation scene Snr).

[0109] In addition, the robot evaluation system 100 is provided with an MR scenario switcher 140, which switches between multiple MR generation units 1101, 1102, 1103, ... included in the adaptive MR generation unit 1100.

[0110] The configuration of MR generation units 1101, 1102, 1103, ... is equivalent to Figure 6 The MR generation unit 110 shown is omitted here.

[0111] It should be noted that any of the MR generation units 1101, 1102, 1103, ... is appropriately described as MR generation unit 110n.

[0112] The adaptive MR generation unit 1100 is not limited to a configuration that includes multiple MR generation units 1101, 1102, 1103, ... For example, the adaptive MR generation unit 1100 may have a configuration that includes multiple sets of parameters controlling the operation of the sensor converter 111, motion converter 112, and existing knowledge / context retention unit 113 included in the MR generation unit 110n.

[0113] The MR scenario switcher 140 switches between multiple MR generation units 1101, 1102, 1103, ... included in the adaptive MR generation unit 1100 or multiple parameter sets for evaluating the robot system 200, based on the output of the recording / evaluation unit 120.

[0114] The robot evaluation system 100 according to this embodiment can perform robot system evaluation more effectively by adaptively switching between multiple MR generation units 1101, 1102, 1103, ... or between multiple parameter sets using an MR context switcher 140.

[0115] The processing according to the implementation method will be described in more detail.

[0116] (3-1. Processes related to improving time efficiency)

[0117] First, the processing related to improving time efficiency through the robot evaluation system 100 according to this embodiment will be described.

[0118] As an example of evaluating the Snr scenario, consider a test vehicle overtaking a virtual vehicle in an overtaking scenario. In the overtaking scenario, for example, the following items can be considered as test items.

[0119] First, the following five items can be considered as test items related to overtaking sections.

[0120] (a) Overtaking on a straight section of road

[0121] (2) Overtaking on a gentle right curve

[0122] (3) Overtaking on a gentle left curve

[0123] (4) Overtaking on a sharp right turn

[0124] (5) Overtaking on a sharp left turn

[0125] Next, the following four items can be considered as test items related to the overtaking target.

[0126] (i) Overtaking low-speed vehicles

[0127] (ii) Overtaking medium-speed vehicles

[0128] (iii) Following high-speed vehicles (without overtaking)

[0129] (iv) Overtaking two-wheeled vehicles

[0130] In this example, it is necessary to perform 20 test modes based on a combination of 5 items related to the overtaking section and 4 items related to the overtaking target. Furthermore, parameters such as the inter-vehicle distance between the overtaking target and the test vehicle need to be appropriately adjusted according to the test vehicle's position on the route and its current speed, and it is necessary to confirm whether the expected overtaking has occurred.

[0131] Now, suppose the first test item, the combination of item (a) and item (i), namely, the experiment of "overtaking a low-speed vehicle on a straight section" for the autonomous vehicle as the test target vehicle, has just been completed. It is possible to continue driving the autonomous vehicle toward the next straight section to implement the next test item, the combination of item (a) and item (ii), namely, "overtaking a medium-speed vehicle on a straight section". At this time, suppose the recording / evaluation unit 120 identifies, based on the identification results of the speed and position of the test vehicle, that there is no sufficient straight section ahead but there is a gentle right curve. In this case, the MR scenario switcher 140 activates the evaluation scenario Snr, which implements the combination of item (2) and item (i) "overtaking a low-speed vehicle on a gentle right curve", that is, selects the MR generation unit 110n from the adaptive MR generation unit 1100 to implement this operation, thereby reducing the wasted driving time unrelated to the test item.

[0132] In this way, the robot evaluation system 100 according to this embodiment can adaptively optimize the effective order of tests by considering the timing and condition of the test vehicle when it completes the test, as well as the remaining test items. By applying the robot evaluation system 100 according to this embodiment, the execution time of the total test set can be shortened. Of course, test failures can also be identified and additional test scenarios can be activated.

[0133] These effects of the robot evaluation system 100 according to the implementation method can be achieved by using the function of the MR scenario switcher 140 to adaptively switch the evaluation scenario Snr with reference to the data logger 121 and the evaluation index 122.

[0134] (3-2. Treatments related to improving space efficiency)

[0135] Next, the process related to improving space efficiency through the robot evaluation system 100 according to this embodiment will be described.

[0136] Figure 9 This is a schematic diagram illustrating an example of a test route. Assuming in... Figure 9 Driving experiments are performed on test route 80 shown on the right. Since actual test vehicles need to drive on routes in at least the real-world environment 131, this test route 80 is practically required in the MR system according to the prior art. Conversely, by applying the robot evaluation system 100 according to this embodiment, driving tests on test route 80 can be performed using test route 90 having a different shape than test route 80, such as... Figure 9 As shown on the left. This eliminates the need for wide and long test courses, improving space utilization efficiency.

[0137] Specifically, by performing the following actions, it is possible to conduct driving tests on test route 80 using test route 90, which has a different shape from test route 80.

[0138] Analyzing test course 80, it can be seen that it is connected counterclockwise as follows: the "straight road segment" in range 81a, the "gentle left curve", "gentle right curve", "gentle left curve" and "gentle right curve" included in range 82b, the "gentle left curve" in range 81c, the "close right corner" in ranges 81d and 81e, the "straight road segment" in range 81f, and the "close left corner" in range 81g, etc. Each of these ranges 81a to 81g is considered to be an evaluation scenario Snr used to drive the test.

[0139] On the other hand, it can be seen that test route 90 is connected counterclockwise through the "gentle left curve" in range 91a, the "straight section" (and its opposite side) in range 91b, and the "sharp right turn" in ranges 91c and 91d. Moreover, if test route 90 is traveled clockwise, it can be seen that it becomes a route connected by the following: the "straight section" on the opposite side of range 91b, the "sharp right turn" in ranges 91c and 91d, the "straight section" in range 91b, and the "gentle right turn" in range 91a.

[0140] Therefore, by rearranging the evaluation scenario Snr for driving tests on test route 80 according to the route shape of real environment 131, regardless of the order, a driving experiment equivalent to driving counterclockwise on test route 80 can be performed by lapsing test route 90 several times in real environment 131. Figure 9 In the example, it is possible to perform a test equivalent to driving within range 81c of test route 80 by traveling counterclockwise within range 91a of test route 90. By traveling clockwise within ranges 91c and 91d of test route 90, it is possible to perform a test equivalent to driving within ranges 81d and 81e of test route 80. Furthermore, regardless of the direction, the same test as within range 81c of test route 80 can be performed by traveling within range 91b of test route 90.

[0141] The rearrangement of evaluation scenarios Snr allows for repetition, such as using the same evaluation scenario Snr multiple times. Therefore, the robot evaluation system 100 according to this embodiment can generate a sequence of evaluation scenarios Snr with actually executable behaviors.

[0142] In this case, the constraint is that the driving trajectory must not deviate from the test route 80 in the real environment 131. That is, as long as the driving trajectory does not deviate from the test route 80 in the real environment 131, the shape of the test route 90 in the virtual environment 130 does not necessarily need to be a perfect match. In other words, it is sufficient to cover a portion of the desired driving trajectory with a specified margin.

[0143] Note that in existing MR systems, typically only virtual information such as the position and speed of oncoming and ahead vehicles is replaced to make it appear as if the vehicles are actually there. In contrast, in the robot evaluation system 100 according to the embodiment, by further replacing the position information of all its own vehicles (test vehicles), the road surface information ahead, the route shape information, etc., with virtual information, the test can be performed as if it were on a route completely different from the test route in the real environment 131.

[0144] Unlike humans, the robot evaluation system 100 according to this embodiment can quickly switch between situations regarding "where the test vehicle (robot system 200) is and what it is doing." For example, even if the surrounding situation suddenly changes and the test vehicle jumps to a sharp turn at the first corner, and then the test vehicle is about to leave the third corner and enter a straight section, the robot evaluation system 100 according to this embodiment can react and operate immediately in response to the new situation.

[0145] In this manner, the robot evaluation system 100 according to the embodiment uses an MR scenario switcher 140 to switch between multiple evaluation scenarios Snr (MR generation units 1101, 1102, 1103, ... and virtual environments 1301, 1302, 1303, ...) based on the behavior of the robot system 200. Therefore, the constraints of the real environment 131 can be taken into account when switching evaluation scenarios Snr, and the time and space utilization efficiency for evaluation implementation is improved.

[0146] That is, according to this embodiment, the robot evaluation system 100 improves the time efficiency of evaluation by rewriting part of the sensor outputs of the robot system 200 by evaluating the scene Snr. Moreover, according to this embodiment, the robot evaluation system 100, by essentially completely rewriting the sensor outputs of the robot system 200 by evaluating the scene Snr, not only improves time efficiency but also enables effective evaluation without spatial constraints in the real environment 131.

[0147] (4. A first variation of the embodiments of this disclosure)

[0148] Next, a first variation of the embodiments of this disclosure will be described.

[0149] Due to their characteristics, MR systems require communication between the robot system 200 in the real environment 131 and the virtual environment 130. Especially for high-speed moving objects such as autonomous vehicles, the communication latency between the robot system 200 in the real environment 131 and the virtual environment 130 can become hundreds of milliseconds or even greater. Therefore, it would be preferable if this communication latency could be hidden in the robot evaluation system 100 according to the embodiment.

[0150] It should be noted that a method has also been conceived in which a computer performing the simulation in a virtual environment is mounted on the test vehicle to be evaluated and connected to the MR system via wires to make the communication latency practically negligible, but this is outside the scope of this disclosure.

[0151] Figure 10 This is a block diagram illustrating an example configuration of a robot evaluation system 100a according to a first variation of an embodiment. Note that... Figure 10 Concentrated on different from the basis Figure 8 The robot evaluation system 100 of the illustrated embodiment is shown in part, and the details are omitted. Figure 8 The configuration shares common components (adaptive MR generation unit 1100, virtual environment 130, and real environment 131).

[0152] The robot evaluation system 100a according to the first variation of this embodiment has added to the system according to Figure 8 The robot evaluation system 100 of the embodiment shown has a robot system behavior prediction unit 150. The robot system behavior prediction unit 150 learns a behavior model for the action output at =fc(St) based on information representing the behavior of the robot system 200 to be evaluated obtained from the recording / evaluation unit 120.

[0153] In the robot evaluation system 100a, the MR context switcher 140a switches the MR generation unit 110n in the adaptive MR generation unit 1100 according to the operation of the robot system 200. The operation of the robot system 200 is predicted by the robot system behavior prediction unit 150 using a learned behavior model for the action output at =fc(St).

[0154] That is, the robot evaluation system 100a according to the first variant of the implementation includes, as follows: Figure 10 The robot system behavior prediction unit 150 shown is connected to the MR context switcher 140a, which enables the effective hiding of communication delays.

[0155] The robot system behavior prediction unit 150 is a function approximator that predicts operations based on a behavior model of the action output of the robot system 200 to be evaluated at =fc(St). Since the behavior of the robot system 200 is recorded in the data logger 121, a predictor can be constructed that simulates the operation using the data and machine learning methods such as supervised learning or imitation learning.

[0156] Here, we consider the case where the predictor configured in this way can learn with sufficient accuracy above a certain level. Assume the effective communication delay between the robot system 200 to be evaluated and the virtual environment 130 is time Δt. In this case, the predictor predicts the action output of the robot system 200 in the current state St, and the simulator predicts the resulting state St+1 as shown in the following equation (3). Furthermore, the action of the robot system 200 at +1 in this predicted state is predicted by the following equation (4).

[0157] St+1 = fc(St,at)...(3)

[0158] at+1 = fc(St+1)...(4)

[0159] Repeat the predictions in equations (3) and (4) until the communication delay time Δt is reached. Alternatively, the predictor can be configured to accurately predict the communication delay time Δt as shown in equation (5) below.

[0160] St+Δt = fe(St, at)...(5)

[0161] By inputting the state St+Δt predicted by equation (5) instead of the state St in the real environment 131 into the robot system 200 for evaluation, it can be made to appear as if the actual delay is 0. The deviation of this predicted value from the real environment 131 can be accurately calculated after time Δt caused by the communication delay. Therefore, if the error remains within a certain range, it is safe to continue using the predictor as is. On the other hand, if a sudden large error is observed, safety mechanisms such as stopping the use of the predictor should be activated.

[0162] Furthermore, the accuracy can be further improved by learning the predictor using the operational data of the robot system 200 collected by the data logger 121 during this period.

[0163] In the robot evaluation system 100a of the first variant according to the embodiment, even more flexible operation of the adaptive MR generation unit 1100 in the robot evaluation system 100a is possible when the behavior of the MR scenario switcher 140a is adjusted using the robot system behavior prediction unit 150. Specifically, further improvements to the evaluation scenario Snr included in the adaptive MR generation unit 1100 are possible.

[0164] Specifically, by using a pre-generated behavior pre-evaluation evaluation scenario Snr, generated by a predictor, before applying the evaluation scenario Snr to the robot system 200 in the real environment 131, the evaluation scenario Snr can be screened and prioritized. For example, actual testing can be omitted for evaluation scenarios Snr that are clearly successful or unsuccessful. On the other hand, more accurate performance evaluation is possible by focusing on evaluation scenarios Snr that have small margins or low safety margins.

[0165] By prioritizing the evaluation scenarios (Snr), for example, when testing time is limited, tests can be conducted in a priority order, allowing for mid-assessment termination. Furthermore, by testing the evaluation scenarios (Snr) in the order they have been previously screened and deemed safe, the impact of scenarios such as accidents that temporarily halt the evaluation process can be minimized.

[0166] Furthermore, the comprehensiveness of the test can be pre-confirmed by using a pre-generated behavioral pre-evaluation scenario (Snr) generated by a predictor. For example, if validation of the pre-prepared evaluation scenario (Snr) reveals that the test vehicle avoids turning left in all situations, then testing for situations where it avoids turning right becomes insufficient. In this case, it is possible to change the test parameters, automatically search for situations where the subject is likely to avoid them correctly, and implement the actual test by adding the evaluation scenario (Snr) generated in this way.

[0167] According to the first variation of this embodiment, the robot evaluation system 100a can use the robot system behavior prediction unit 150 to learn and predict the behavior of the robot system 200 to be evaluated, thereby generating or selecting an evaluation scenario Snr that takes into account the predicted movement, and significantly improving evaluation performance and efficiency.

[0168] Furthermore, the robot evaluation system 100a according to the first variation of this embodiment can hide the unavoidable communication delay between the evaluation system and the robot system 200 to be evaluated, thereby enabling more accurate evaluation. In addition, the robot evaluation system 100a can use the robot system behavior prediction unit 150 to predict the evaluation scenario Snr that reveals the worst performance of the robot system 200 to be evaluated, and by generating or selecting such evaluation scenario Snr, more accurate evaluation becomes possible. Furthermore, the robot evaluation system 100a can shorten the evaluation time by generating a minimal set of evaluation scenario Snr covering all behaviors of the robot system 200 to be evaluated, with as little repetition and omission as possible.

[0169] (5. A second variation of the embodiments of this disclosure)

[0170] Next, a second variation of the embodiments of this disclosure will be described.

[0171] Figure 11 This is a block diagram illustrating an exemplary configuration of a robot evaluation system 100b according to a second variation of the embodiment. Note that, similar to the above, Figure 11 Focus on and according to Figure 8 The robot evaluation system 100 of the embodiment shown differs from other systems, and parts that are omitted are not included. Figure 8 The configuration shares common components (adaptive MR generation unit 1100, virtual environment 130, and real environment 131).

[0172] The robot evaluation system 100b according to the second variation of this embodiment has the function of adding to the robot evaluation system according to the embodiment. Figure 10 The robot evaluation system 100a of the illustrated embodiment includes a predictive performance evaluation unit 160. The predictive performance evaluation unit 160 monitors the inputs and outputs of the robot system behavior prediction unit 150. According to a second variation of this embodiment, the robot evaluation system 100b can monitor the inputs and outputs of the robot system behavior prediction unit 150 using the predictive performance evaluation unit 160, and estimate the prediction accuracy of the predictor in the robot system behavior prediction unit 150 in real time for each scenario based on data collected offline and online from the recording / evaluation unit 120 and other sources.

[0173] In the robot evaluation system 100b, the MR scenario switcher 140b switches the MR generation unit 110n in the adaptive MR generation unit 1100 according to the output of the predictive performance evaluation unit 160 and the robot system behavior prediction unit 150.

[0174] When the adaptive MR generation unit 1100 pre-evaluates the evaluation scene Snr, if predictor accuracy information is available, it can also focus on that prediction accuracy information and generate an evaluation scene Snr for cases where the prediction accuracy is insufficient, based on additional data collection. The robot evaluation system 100b can pre-evaluate such an evaluation scene Snr based on the additional data collection and use the recording / evaluation unit 120 to record how the robot system 200 behaves in the real environment 131. Since the robot system behavior prediction unit 150 can perform additional learning on the data recorded in the recording / evaluation unit 120 to improve accuracy, when the original evaluation scene Snr is evaluated again, a better scene evaluation and prioritization can be performed using the improved predictor, thereby improving the overall efficiency of the entire evaluation.

[0175] As described above, the second modified robot evaluation system 100b according to this embodiment, in addition to the configuration of the robot evaluation system 100a, also includes a predictive performance evaluation unit 160. The predictive performance evaluation unit 160 generates a response data collection scenario to minimize the learning error of the predictor learning the behavior of the robot system 200 to be evaluated. Therefore, an evaluation scenario Snr can be generated or selected that takes into account the predicted motion of the robot system 200 to be evaluated according to the first variation of the embodiment, and maximizes the performance and efficiency of the evaluation.

[0176] (6. A third variation of the embodiments of this disclosure)

[0177] Next, a third variation of the embodiments of the present invention will be described.

[0178] Figure 12 This is a block diagram illustrating an example configuration of a robot evaluation system 100c according to a third variation of the embodiment. Note that, similar to the above, Figure 12 Attention and basis Figure 8 The robot evaluation system 100 shown in the embodiment differs from the one described above, and parts that are different from those described above are omitted. Figure 8 The configuration includes common components (adaptive MR generation unit 1100, MR scenario switcher 140, virtual environment 130 and real environment 131).

[0179] The robot evaluation system 100c according to the third variation of the embodiment has the function of adding to the robot evaluation system according to the embodiment. Figure 11 The robot evaluation system 100b of the embodiment shown has a database 170. The database 170 stores a trained predictor 171 that can be applied to the robot system behavior prediction unit 150 and performance metrics 172 that evaluate the prediction performance through the prediction performance evaluation unit 160.

[0180] For example, the robot system behavior prediction unit 150 can call the trained predictor 171 from the database 170 as needed to update its own predictor. In addition, the prediction performance evaluation unit 160 can call the performance metric 172 from the database 170 as needed and measure the prediction accuracy of the robot system behavior prediction unit 150 based on the called performance metric 172.

[0181] In the robot evaluation system 100c, the MR scenario switcher 140b switches the MR generation unit 110n in the adaptive MR generation unit 1100 according to the output of the predictive performance evaluation unit 160 and the robot system behavior prediction unit 150.

[0182] According to the robot evaluation system 100c of the third variation of this embodiment, by providing a database 170, simulations of previously evaluated robot systems 200 can be performed at any time. By using this function, it becomes easy to objectively and quantitatively compare and analyze multiple robot systems 200 to be evaluated.

[0183] For example, suppose a real-world system is evaluated using a specific evaluation scenario (Snr) and an evaluation value of "80" is obtained (assuming the maximum evaluation value is "100"). Suppose that in different past tests using different systems or past versions of the same system, an evaluation value of "82" is obtained. In this case, because the evaluation scenarios (Snr) used for evaluation are not exactly the same, it remains questionable whether these evaluation values ​​are truly comparable. To address this issue, a more accurate comparison becomes possible by using the same evaluation scenario (Snr) and also by using the predictor's evaluation. This functionality is essential for objectively and quantitatively evaluating differences in algorithms and software architectures, the effects of version upgrades, and so on.

[0184] Furthermore, according to a third variation of the implementation, for example, the reliability of the evaluation performed by the prediction performance evaluation unit 160 can be calculated using a performance indicator 172 about the predictor's prediction error stored in the database 170.

[0185] (7. Application examples of the embodiments of this disclosure)

[0186] Next, application examples of the embodiments of this disclosure and various variations thereof will be described.

[0187] The technology disclosed herein (the Technology) can be applied to a variety of products. For example, the Technology disclosed herein can be implemented as a device installed on any type of mobile body (such as automobiles, electric vehicles, hybrid electric vehicles, motorcycles, bicycles, personal mobility devices, airplanes, drones, ships, robots, etc.).

[0188] Figure 13This is a block diagram illustrating a schematic configuration example of a vehicle control system, which is an example of a mobile body control system to which the technology according to this disclosure can be applied.

[0189] The vehicle control system 12000 includes multiple electronic control units connected via a communication network 12001. Figure 13 In the example shown, the vehicle control system 12000 includes a drive system control unit 12010, a body system control unit 12020, an external information detection unit 12030, an internal information detection unit 12040, and an integrated control unit 12050. Furthermore, as functional components of the integrated control unit 12050, a microcomputer 12051, an audio / image output unit 12052, and an in-vehicle network I / F (interface) 12053 are shown.

[0190] The drive system control unit 12010 controls the operation of devices related to the vehicle's drive system according to various programs. For example, the drive system control unit 12010 serves as a control device for generating drive force for the vehicle (such as an internal combustion engine or drive motor), a drive force transmission mechanism for transmitting drive force to the wheels, a steering mechanism for adjusting the vehicle's steering angle, and a braking device for generating braking force for the vehicle.

[0191] The body system control unit 12020 controls the operation of various devices equipped on the vehicle body according to various programs. For example, the body system control unit 12020 acts as a control device for keyless entry systems, smart key systems, power windows, or various lights such as headlights, taillights, brake lights, turn signals, or fog lights. In this case, radio waves transmitted from a portable device that replaces the key or signals from various switches can be input to the body system control unit 12020. The body system control unit 12020 accepts these radio wave or signal inputs and controls the vehicle's door locking devices, power windows, lights, etc.

[0192] The exterior information detection unit 12030 detects information about the exterior of the vehicle equipped with the vehicle control system 12000. For example, the imaging unit 12031 is connected to the exterior information detection unit 12030. The exterior information detection unit 12030 causes the imaging unit 12031 to capture images of the exterior of the vehicle and receives the captured images. In addition, the exterior information detection unit 12030 can also perform object detection processing and distance detection processing on objects such as people, vehicles, obstacles, signs, and text on the road surface based on the received images.

[0193] Imaging unit 12031 is an optical sensor that receives light and outputs an electrical signal corresponding to the amount of light received. Imaging unit 12031 can output the electrical signal as an image or as ranging information. In addition, the light received by imaging unit 12031 can be visible light or invisible light such as infrared light.

[0194] The in-vehicle information detection unit 12040 detects information inside the vehicle. For example, a driver state detection unit 12041, which detects the driver's state, is connected to the in-vehicle information detection unit 12040. The driver state detection unit 12041 includes, for example, a camera that captures images of the driver, and the in-vehicle information detection unit 12040 can calculate the driver's fatigue level or concentration level based on the detection information input from the driver state detection unit 12041, or it can determine whether the driver is dozing off.

[0195] The microcomputer 12051 can calculate control target values ​​for the drive force generating device, steering mechanism, or braking device based on information about the vehicle's interior and exterior acquired by the external information detection unit 12030 or the internal information detection unit 12040, and output control commands to the drive system control unit 12010. For example, after driving based on inter-vehicle distance, speed maintenance, vehicle collision warning, or lane departure warning, the microcomputer 12051 can perform cooperative control aimed at realizing ADAS (Advanced Driver Assistance Systems) functions, including vehicle collision avoidance or impact mitigation.

[0196] In addition, the microcomputer 12051 can control the drive force generation device, steering mechanism, braking device, etc., based on information about the vehicle's surrounding environment obtained by the external information detection unit 12030 or the internal information detection unit 12040, to perform cooperative control aimed at autonomous driving, which involves automatic driving that does not depend on driver operation.

[0197] Additionally, the microcomputer 12051 can output control commands to the vehicle system control unit 12020 based on external information obtained from the external information detection unit 12030. For example, the microcomputer 12051 can perform cooperative control for anti-glare measures, such as controlling the headlights and switching from high beam to low beam based on the position of the vehicle ahead or oncoming vehicle detected by the external information detection unit 12030.

[0198] The audio / video output unit 12052 sends at least one of audio and video output signals to an output device capable of visually or audibly notifying vehicle occupants or entities outside the vehicle. Figure 13In the example, audio speaker 12061, display unit 12062, and instrument panel 12063 are shown as output devices. For example, display unit 12062 may include at least one of an onboard display and a head-up display.

[0199] Figure 14 This is a diagram showing an example of the placement of the imaging unit 12031.

[0200] exist Figure 14 In the imaging unit 12031, imaging units 12101, 12102, 12103, 12104 and 12105 are included.

[0201] Imaging units 12101, 12102, 12103, 12104, and 12105 are, for example, installed inside the passenger compartment of vehicle 12100 at locations such as the front nose, side mirrors, rear bumper, rear door, and the upper part of the windshield. Imaging unit 12101 at the front nose and imaging unit 12105 at the upper part of the windshield inside the passenger compartment primarily acquire images of the front of vehicle 12100. Imaging units 12102 and 12103 at the side mirrors primarily acquire images of the sides of vehicle 12100. Imaging unit 12104 at the rear bumper or rear door primarily acquires images of the rear of vehicle 12100. Imaging unit 12105 at the upper part of the windshield inside the passenger compartment is mainly used to detect vehicles or pedestrians, obstacles, traffic lights, traffic signs, lanes, etc., ahead.

[0202] It is important to note that Figure 14 An example of the imaging range of imaging units 12101 to 12104 is shown. Imaging range 12111 represents the imaging range of imaging unit 12101 located at the front nose, imaging ranges 12112 and 12113 represent the imaging ranges of imaging units 12102 and 12103 located at the side mirrors, respectively, and imaging range 12114 represents the imaging range of imaging unit 12104 located at the rear bumper or rear door. For example, by overlaying the image data captured by imaging units 12101 to 12104, a top-down image of the vehicle 12100 as viewed from above can be obtained.

[0203] At least one of the imaging units 12101 to 12104 may have the function of acquiring distance information. For example, at least one of the imaging units 12101 to 12104 may be a stereo camera composed of multiple imaging elements, or may be an imaging element having pixels for phase difference detection.

[0204] For example, the microcomputer 12051 can determine the distance to each three-dimensional object within the shooting range 12111 to 12114 and the time change of that distance (relative speed to the vehicle 12100) based on distance information obtained from the shooting units 12101 to 12104. It can then extract the nearest three-dimensional object on the vehicle 12100's travel path as the vehicle ahead, which travels at a predetermined speed (e.g., 0 km / h or greater) in approximately the same direction as the vehicle 12100. Furthermore, the microcomputer 12051 can pre-set a safe distance in front of the vehicle ahead and execute automatic braking control (including follow-stop control) or automatic acceleration control (including follow-start control), etc. This enables cooperative control for automated driving that does not rely on driver intervention.

[0205] For example, microcomputer 12051 can classify and extract three-dimensional object data related to three-dimensional objects into two-wheeled vehicles, ordinary vehicles, large vehicles, pedestrians, utility poles, and other three-dimensional objects based on distance information obtained from imaging units 12101 to 12104, and use this data for automatic obstacle avoidance. For example, microcomputer 12051 identifies obstacles around vehicle 12100 as obstacles visible to the driver of vehicle 12100 and obstacles that are difficult to see. Then, microcomputer 12051 determines a collision risk indicating the degree of danger of colliding with each obstacle, and when the collision risk is higher than a set value and there is a possibility of collision, it can provide driving assistance for collision avoidance by outputting an alarm to the driver via audio speaker 12061 or display unit 12062, or by performing forced deceleration or evasive steering via drive system control unit 12010.

[0206] At least one of the imaging units 12101 to 12104 may be an infrared camera that detects infrared light. For example, the microcomputer 12051 can identify a pedestrian by determining whether a pedestrian exists in the images captured by the imaging units 12101 to 12104. This pedestrian identification is performed by a process of extracting feature points from the images captured by the imaging units 12101 to 12104, which are infrared cameras, and performing pattern matching processing on a series of feature points representing the contour of an object to determine whether it is a pedestrian. When the microcomputer 12051 determines that a pedestrian exists in the images captured by the imaging units 12101 to 12104 and identifies the pedestrian, the sound / image output unit 12052 controls the display unit 12062 to overlay and display a rectangular outline to emphasize the identified pedestrian. Furthermore, the sound / image output unit 12052 can control the display unit 12062 to display icons or other indicators of pedestrians at a desired location.

[0207] The above describes examples of vehicle control systems to which the technology according to this disclosure can be applied. For example, the technology according to this disclosure can be applied to the vehicle control system 12000 in the above configuration. For example, the drive system control unit 12010 of the vehicle control system 12000 may include an adaptive MR generation unit 1100 in the robot evaluation system 100 according to an embodiment. In the MR generation unit 110n, a sensor converter 111 overlays an image from the virtual environment 130n onto a portion of a captured image of the vehicle exterior detected, for example, by the vehicle exterior information detection unit 12030, and transmits it to the drive system control unit 12010. Furthermore, in the MR generation unit 110n, an action converter 112 may, for example, acquire control signals from the drive system control unit 12010 and the body system control unit 12020, and generate corresponding action outputs for the virtual environment 130 and the real environment 131. By applying the technology according to this disclosure to the vehicle control system 12000, the vehicle can be used more efficiently in time and space as an autonomous driving test vehicle to perform evaluations.

[0208] Note that the effects described in this specification are illustrative and not restrictive, and other effects may exist.

[0209] It should be noted that this technology can also be configured as follows. (1)

[0211] An information processing method includes: an application step, applying an evaluation scenario, including a mixed reality context combining a virtual environment and a real environment, to a robot system to be evaluated; an evaluation step, evaluating the behavior of the robot system to which the evaluation scenario has been applied; and a switching step, switching the evaluation scenario applied to the robot system to another evaluation scenario among a plurality of evaluation scenarios based on the evaluation of the robot system’s behavior in the currently applied evaluation scenario by the evaluation step. (2)

[0213] According to the information processing method of (1), the evaluation scenario includes a virtual environment based on the context. (3)

[0215] According to the information processing method of (1) or (2), where, The evaluation scenarios include: synthesis processing, synthesizing real sensor output based on a real environment and virtual sensor output based on a virtual environment to generate a composite sensor output, and The application steps also involve applying the composite sensor output to the robotic system. (4)

[0217] According to the information processing method of (3), the synthesis processing includes: processing of rewriting at least a portion of the real sensor output based on the virtual sensor output. (5)

[0219] According to any one of (1) to (4) the information processing method, wherein the evaluation scenario includes: converting the behavior of the robot system into behavior in a virtual environment and applying it to the processing of the virtual environment; and converting the behavior of the robot system into behavior in a real environment and applying it to the processing of the real environment. (6)

[0221] According to any one of (1) to (5) of the information processing method, the switching step switches the evaluation scenario based on the past evaluation results of the evaluation step evaluating the behavior of the robot system. (7)

[0223] The information processing method according to any one of (1) to (6) also includes: The prediction step uses a predictor to predict the robot system's behavior based on past evaluation results from the evaluation step's assessment of the robot system's operations. The switching step uses the prediction results from the prediction step to switch the evaluation scenario. (8)

[0225] According to the information processing method of (7), the prediction step uses a predictor learned from the past behavior of the robot system to perform the prediction. (9)

[0227] According to the information processing method of (7) or (8), it also includes: The predictive performance evaluation step evaluates the predictive performance based on the prediction results of the robot system's behavior obtained by the prediction step and the past evaluation results of the robot system's behavior obtained by the evaluation step. The switching step uses the prediction results from the prediction step and the prediction performance evaluation results from the prediction performance evaluation step to switch the evaluation scenario. (10)

[0229] The information processing method according to any one of (7) to (9) also includes: Accumulated steps: Accumulating predictors learned from the past behavior of the robot system and applicable to predicting steps, where... The prediction step updates the predictor based on the prediction results accumulated by the accumulation step. (11)

[0231] According to the information processing method in (10), where, The accumulation step also accumulates the performance metrics evaluated by the prediction step for prediction performance, and The prediction step calculates the reliability of the prediction results for predicting the behavior of the robot system based on the performance metrics accumulated by the accumulation step. (12)

[0233] An information processing program causes a computer to perform the following steps: an application step, applying an evaluation scenario, including a mixed reality scenario combining a virtual environment and a real environment, to a robot system to be evaluated; an evaluation step, evaluating the behavior of the robot system to which the evaluation scenario has been applied; and a switching step, switching the evaluation scenario applied to the robot system to another evaluation scenario among a plurality of evaluation scenarios based on the evaluation of the robot system's behavior in the currently applied evaluation scenario by the evaluation step. (13)

[0235] An information processing device includes: an application unit that applies an evaluation scenario, the evaluation scenario including a mixed reality scenario based on a combination of virtual and real environments applied to a robot system to be evaluated; an evaluation unit that evaluates the behavior of the robot system applied to the evaluation scenario; and a switching unit that switches the evaluation scenario applied to the robot system to another evaluation scenario among multiple evaluation scenarios based on the evaluation performed by the evaluation unit on the behavior of the robot system in the currently applied evaluation scenario.

[0236] [List of Reference Markers]

[0237] 1, 100, 100a, 100b, 100c, 1000 Robot Evaluation Systems

[0238] 10, 1100 Adaptive MR Generation Unit

[0239] 11 Sensors

[0240] 12 storage units

[0241] 13 Drive Control Unit

[0242] 60, 62, 80, 90, 600r, 600v test routes

[0243] 110 MR generation unit

[0244] 111 Sensor Converter

[0245] 112 Action Converter

[0246] 113 Existing Knowledge / Context Preservation Unit

[0247] 120 Recording / Evaluation Unit

[0248] 121 Data Logger

[0249] 122 Evaluation Indicators

[0250] Virtual environments 130, 1301, 1302, 1303, and 510

[0251] 131, 520 Real-world environment

[0252] 140, 140a, 140b MR Scene Switcher

[0253] 150 Robot System Behavior Prediction Unit

[0254] 160 Predictive Performance Evaluation Unit

[0255] 170 Database

[0256] 171 Trained predictor

[0257] 172 Performance Indicators

[0258] 200 and 500 robot systems.

Claims

1. An information processing method, comprising: The application steps involve applying evaluation scenarios, including those based on mixed reality scenarios combining virtual and real environments, to the robotic system to be evaluated. The evaluation step assesses the behavior of the robot system in the evaluation scenario. as well as The switching step involves switching the evaluation scenario applied to the robot system to another evaluation scenario among multiple evaluation scenarios, based on the evaluation of the robot system's behavior in the current application evaluation scenario by the evaluation step.

2. The information processing method according to claim 1, wherein, The evaluation scenario includes the virtual environment based on the context.

3. The information processing method according to claim 1, wherein, The evaluation scenario includes: a synthesis process that synthesizes real sensor outputs based on the real environment and virtual sensor outputs based on the virtual environment to generate a composite sensor output. The application steps also apply the output of the composite sensor to the robot system.

4. The information processing method according to claim 3, wherein, The synthesis process includes: rewriting at least a portion of the output of the real sensor based on the output of the virtual sensor.

5. The information processing method according to claim 1, wherein, The evaluation scenarios include: converting the behavior of the robot system into behavior in the virtual environment, and applying the behavior in the virtual environment to the processing of the virtual environment; and converting the behavior of the robot system into behavior in the real environment, and applying the behavior in the real environment to the processing of the real environment.

6. The information processing method according to claim 1, wherein, The switching step switches the evaluation scenario based on past evaluation results from the evaluation step that assesses the behavior of the robot system.

7. The information processing method according to claim 1, further comprising: The prediction step uses a predictor to predict the behavior of the robot system based on past evaluation results from the evaluation step's assessment of the robot system's operation. The switching step uses the prediction results from the prediction step to switch the evaluation scenario.

8. The information processing method according to claim 7, wherein, The prediction step uses the predictor, which has been learned based on the past behavior of the robot system, to perform the prediction.

9. The information processing method according to claim 7, further comprising: The predictive performance evaluation step evaluates the predictive performance based on the predicted behavior of the robot system obtained by the prediction step and the past evaluation results of the evaluation step that assess the behavior of the robot system. The switching step uses the prediction results from the prediction step and the prediction performance evaluation results from the prediction performance evaluation step to switch the evaluation scenario.

10. The information processing method according to claim 7, further comprising: The accumulation step accumulates the predictor learned based on the past behavior of the robot system and applicable to the prediction step, wherein... The prediction step updates the predictor based on the prediction results accumulated by the accumulation step.

11. The information processing method according to claim 10, wherein, The accumulation step also accumulates the performance metrics evaluated by the prediction step for prediction performance, and The prediction step calculates the reliability of the prediction results for predicting the behavior of the robot system based on the performance metrics accumulated by the accumulation step.

12. An information processing program that causes a computer to perform the following steps: The application steps involve applying evaluation scenarios, including those based on mixed reality scenarios combining virtual and real environments, to the robotic system to be evaluated. The evaluation step assesses the behavior of the robot system in the evaluation scenario. as well as The switching step involves switching the evaluation scenario applied to the robot system to another evaluation scenario among multiple evaluation scenarios, based on the evaluation of the robot system's behavior in the current application evaluation scenario by the evaluation step.

13. An information processing device, comprising: The application unit applies evaluation scenarios, including those based on mixed reality scenarios combining virtual and real environments, to the robot system to be evaluated. An evaluation unit evaluates the behavior of the robot system in the evaluation scenario. as well as The switching unit switches the evaluation scenario applied to the robot system to another evaluation scenario among the multiple evaluation scenarios, based on the evaluation of the robot system's behavior in the current evaluation scenario by the evaluation unit.

Images