Simulator

Abstract

To provide a simulator suitable for performing evaluation in consideration of a psychological state of a subject.SOLUTION: A driving simulator 10 acquires sensor information related to operation of a steering wheel 18 by a subject, so as to control driving simulation for replaying a view seen from a simulated vehicle when the simulated vehicle is made travel in a virtual road environment as a video on the basis of the acquired sensor information. Information on a subject's feeling of astonishment is generated so as to be recorded in association with the sensor information. Additionally, evaluation information related to driving operation by the subject is generated on the basis of the information on the feeling of astonishment.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to a simulator for supporting education, training, research, development, testing, etc., and more particularly to a simulator suitable for performing evaluation taking into account the psychological state of a subject. [Background technology]

[0002] 2. Description of the Related Art Conventionally, as a driving simulator, for example, the technology described in Patent Document 1 is known. The technology described in Patent Document 1 is equipped with operating components such as a steering wheel, accelerator, and brake, and controls a driving simulation that plays back a video of the scenery seen from a simulated vehicle when the simulated vehicle is driven in a virtual road environment based on the subject's operation of the operating components, and simulates how the subject responds to events that occur during the driving simulation. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2015-79224 A Summary of the Invention [Problem to be solved by the invention]

[0004] The technology described in Patent Document 1 simulates how a subject operates an operating unit, for example, by suddenly turning the steering wheel or braking in response to an event in which a person jumps out from the sidewalk onto the roadway. However, even for the same event, whether the subject responded in a calm or agitated psychological state differs depending on the subject. For example, the subject is evaluated differently when the subject suddenly brakes in a calm psychological state and when the subject suddenly brakes in an agitated psychological state. However, the technology described in Patent Document 1 has a problem in that it is not possible to perform an evaluation that takes into account the psychological state of the subject.

[0005] Therefore, the present invention has been made with a focus on the unresolved problems of the conventional technology, and aims to provide a simulator that is suitable for performing an evaluation that takes into account the psychological state of a subject. [Means for solving the problem]

[0006] [Invention 1] In order to achieve the above object, the simulator of Invention 1 comprises a first acquisition means for acquiring operation information related to an operation or motion of a subject, a control means for controlling a movement simulation that plays a video of a view seen from a virtual moving object when the moving object is moved based on the operation information acquired by the first acquisition means, a second acquisition means for acquiring emotional information of the subject, such as surprise, fear, impatience or discomfort, and a recording means for recording log information of the movement simulation including the emotional information acquired by the second acquisition means.

[0007] With this configuration, when the first acquisition means acquires operation information, the control means controls the movement simulation based on the operation information. In the movement simulation, a scene seen from the moving object when the moving object is moved is played as a video. Then, the second acquisition means acquires emotion information, and the recording means records log information including the emotion information.

[0008] Here, the first acquisition means may, for example, input information from an input device or the like, acquire or receive information from an external terminal or the like, read information from a storage device or storage medium or the like, or generate or calculate information by information processing or the like. Therefore, acquisition includes at least input, acquisition, reception, reading (including search), generation, and calculation. The same concept of acquisition applies hereinafter.

[0009] In addition, the present simulator may be realized as a single device, terminal, or other equipment, or may be realized as a system in which multiple devices, terminals, or other equipment are communicatively connected. In the latter case, each component may belong to any one of the multiple devices, etc., as long as they are communicatively connected to each other. The same applies to the simulator of invention 5 below.

[0010] [Invention 2] Furthermore, the simulator of Invention 2 is the simulator of Invention 1, wherein the recording means records the emotion information under predetermined conditions, or records the emotion information in association with the operation information acquired by the first acquisition means or with a time point during the movement simulation.

[0011] With this configuration, the recording means records emotion information under predetermined conditions, or records emotion information in association with operation information or a time point during a movement simulation.

[0012] Here, recording emotion information in association with operation information or a time point during a movement simulation (hereinafter referred to as "operation information, etc." in this paragraph) includes, for example, (1) directly correlating and recording operation information, etc. and emotion information, such as by registering them in the same record, and (2) recording via one or more pieces of intermediate information, such as by providing a table that associates and registers operation information, etc. and intermediate information (e.g., elapsed time or time), and a table that associates and registers emotion information and intermediate information. In other words, any data structure can be adopted as long as emotion information can be traced from operation information, etc. The same concept applies hereinafter to recording information in association.

[0013] [Invention 3] Furthermore, the simulator of Invention 3 is the simulator of Invention 1, wherein the control means generates an event or makes an object appear during the movement simulation, and the recording means records the emotion information in association with the event generated or the object made to appear by the control means.

[0014] With this configuration, the recording device records emotion information in association with an event or an object.

[0015] [Invention 4] The simulator of Invention 4 further comprises, in the simulator of Invention 1, a third acquisition means for acquiring biological information, gaze information, movement information, image information, audio information, smell information or behavioral history of the subject, environmental information around the subject, or information necessary for control of the control means, and the recording means records the emotion information in association with the information acquired by the third acquisition means.

[0016] With this configuration, the third acquisition means acquires biometric information, gaze information, movement information, image information, audio information, smell information, behavioral history, environmental information, or information necessary for control, and the recording means records emotional information in association with the acquired information.

[0017] [Invention 5] Furthermore, the simulator of Invention 5 comprises a first acquisition means for acquiring operation information related to an operation or motion of a subject, a control means for controlling a movement simulation in which a scene seen from a virtual moving object when the moving object is moved is reproduced as a video based on the operation information acquired by the first acquisition means, a second acquisition means for acquiring emotional information of the subject, such as surprise, fear, impatience or discomfort, and a generation means for generating evaluation information related to the movement simulation based on the emotional information acquired by the second acquisition means.

[0018] With this configuration, when the first acquisition means acquires operation information, the control means controls the movement simulation based on the operation information. In the movement simulation, a scene seen from the moving object when the moving object is moved is played as a video. Then, the second acquisition means acquires emotion information, and the generation means generates evaluation information based on the emotion information. Effect of the Invention

[0019] As described above, according to the simulator of invention 1, it is possible to perform an evaluation taking into consideration the subject's feelings of surprise, fear, impatience, or discomfort by referring to the log information.

[0020] Furthermore, according to the simulator of Invention 2, it is possible to perform an evaluation taking into consideration the subject's feelings of surprise, fear, impatience, or discomfort in comparison with predetermined conditions, operation information, or time points during the movement simulation.

[0021] Furthermore, according to the simulator of Invention 3, it is possible to perform an evaluation taking into consideration the subject's feelings of surprise, fear, impatience, or discomfort in comparison with an event or object.

[0022] Furthermore, according to the simulator of Invention 4, it is possible to perform an evaluation taking into account the subject's feelings of surprise, fear, impatience or discomfort by comparing with biological information, gaze information, movement information, image information, audio information, smell information, behavioral history, environmental information or information necessary for control.

[0023] Furthermore, according to the simulator of Invention 5, it is possible to perform an evaluation taking into consideration the subject's feelings of surprise, fear, impatience, or discomfort. [Brief description of the drawings]

[0024] [Figure 1] 1 is a diagram showing a hardware configuration of a driving simulator 10. FIG. [Diagram 2] FIG. 2 is a diagram illustrating a hardware configuration of a control device 100. [Diagram 3] 13 is a flowchart showing a control process. [Figure 4] 13 is a flowchart showing a log information recording process. [Diagram 5] FIG. 11 is a diagram showing evaluation information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0025] BEST MODE FOR CARRYING OUT THE PRESENT DISCLOSURE The present invention will now be described with reference to the accompanying drawings, in which: FIG. 〔composition〕 First, the configuration of this embodiment will be described.

[0026] FIG. 1 is a diagram showing a hardware configuration of a driving simulator 10. As shown in FIG. The driving simulator 10 is installed, for example, in medical institutions such as hospitals, research institutions such as universities, and public institutions such as police stations and driver's license testing centers, and is used to evaluate simulated vehicle driving by patients, visitors, etc. (hereinafter referred to as "subjects").

[0027] As shown in FIG. 1, the driving simulator 10 is configured to include a seat (not shown) on which the test subject sits, a front display 12 installed in front of the seat, a left display 14 installed to the left of the front display 12, and a right display 16 installed to the right of the front display 12.

[0028] The driving simulator 10 further comprises a steering wheel 18 installed between the seat and the front display 12, an accelerator pedal 20 and a brake pedal 22 installed in the front lower part of the seat, a camera 24 for photographing the subject's facial expression, and a control device 100.

[0029] The displays 12-16 are arranged from the front to the left and right so that their display surfaces are located at positions corresponding to the windshield surface of an actual vehicle. The displays 12-16 cover the visual field of the subject looking forward. The seat, steering wheel 18, accelerator pedal 20, and brake pedal 22 are arranged at positions similar to those of an actual vehicle. The displays 12-16 display moving images of the scenery seen from a simulated vehicle (e.g., an automobile) when the simulated vehicle is driven in a virtual road environment based on the subject's operation of the steering wheel 18, etc. This allows the subject seated in the seat to experience simulated driving with the same operating feel as in a real vehicle while viewing the screens of the displays 12-16.

[0030] [Control device] Next, the configuration of the control device 100 will be described. FIG. 2 is a diagram showing a hardware configuration of the control device 100. As shown in FIG.

[0031] The control device 100 controls a driving simulation in which a scene seen from a simulated vehicle when the simulated vehicle is driven in a virtual road environment is reproduced as a video based on the subject's operation of a steering wheel 18, etc. The control device 100 can be configured as, for example, a desktop PC (Personal Computer), a notebook PC, or other computer.

[0032] As shown in FIG. 2, the control device 100 is composed of a CPU (Central Processing Unit) 30 that controls calculations and the entire system based on a control program, a ROM (Read Only Memory) 32 in which the control program etc. of the CPU 30 are stored in advance in a specified area, a RAM (Random Access Memory) 34 for storing data read from the ROM 32 etc. and calculation results required in the calculation process of the CPU 30, and an I / F (InterFace) 38 that mediates the input and output of data to and from external devices. These are connected to each other and capable of sending and receiving data by a bus 39, which is a signal line for transferring data.

[0033] Connected to the I / F 38 are external devices, such as a steering angle sensor 40 that detects the steering angle (steering wheel angle) of the steering wheel 18, an accelerator sensor 42 that detects the amount of depression of the accelerator pedal 20 (accelerator opening), a brake sensor 44 that detects the amount of depression of the brake pedal 22 (brake opening), displays 12-16, the camera 24, and a storage device 46 that stores data, tables, etc. as files.

[0034] The storage device 46 stores course data related to a virtual road environment including a course on which the simulated vehicle runs and various traffic conditions, object data related to objects that appear while the simulated vehicle is running, event data related to events that occur while the simulated vehicle is running, and log information of the driving simulation (hereinafter simply referred to as "log information"). For example, the course data and training scene data in Patent Document 2 (JP Patent Publication No. 2021-128375) can be used as the course data and event data.

[0035] Next, the operation of this embodiment will be described. [Control Processing] The CPU 30 is made up of an MPU (Micro-Processing Unit) or the like, and starts a predetermined program stored in a predetermined area of ​​the ROM 32, and executes the control process shown in the flowchart of FIG. 3 in accordance with the program.

[0036] FIG. 3 is a flowchart showing the control process. The control process is a process for controlling the entire driving simulation, and when it is executed by the CPU 30, first, the process proceeds to step S100 as shown in FIG.

[0037] In step S100, a driving simulation control process is executed to control the driving simulation. In the driving simulation control process, a virtual road environment including a course on which the simulated vehicle runs and various traffic conditions is reproduced based on the course data in the storage device 46. Then, based on the sensor information of the steering angle sensor 40, the accelerator sensor 42, and the brake sensor 44, a scene seen from the simulated vehicle when the simulated vehicle runs in the virtual road environment is reproduced as a video on the displays 12 to 16. The video is generated in real time, for example, using a CG (Computer Graphics) model. Also, based on the object data in the storage device 46, objects such as pedestrians, vehicles, structures, buildings, and tunnels are made to appear while the simulated vehicle is running. Based on the event data in the storage device 46, events such as "pedestrian on a crosswalk," "approach of a pedestrian," and "pedestrian rushing out" are generated while the simulated vehicle is running. For example, the technology of Patent Document 2 can be adopted as the driving simulation control process.

[0038] Next, the process proceeds to step S102, where a log information recording process is executed to record log information, and the process proceeds to step S104, where it is determined whether the driving simulation has ended. If it is determined that the driving simulation has ended (YES), the process proceeds to step S106.

[0039] In step S106, an evaluation information generating process is executed to generate evaluation information regarding the test subject's driving performance based on the log information in the storage device 46, and then the series of processes is terminated.

[0040] On the other hand, if it is determined in step S104 that the driving simulation has not ended (NO), the process proceeds to step S100.

[0041] [Log information recording process] Next, the log information recording process in step S102 will be described. FIG. 4 is a flowchart showing the log information recording process.

[0042] When the log information recording process is executed in step S102, the process first proceeds to step S200 as shown in FIG.

[0043] In step S200, it is determined whether it is time to record log information (for example, every 0.03 [s] (= 30 fps)), and if it is determined that it is time to record (YES), the process proceeds to step S202, but if it is determined that it is not time to record (NO), the process ends and returns to the original process.

[0044] In step S202, elapsed time information indicating the elapsed time from the start of the driving simulation is obtained. The elapsed time can be measured by a timer in the CPU 30, for example.

[0045] Next, the process proceeds to step S204, where distance information indicating the distance traveled by the simulated vehicle from the starting point of the driving simulation is acquired. The traveled distance can be calculated, for example, by measuring the travel path of the simulated vehicle on the course.

[0046] Next, the process proceeds to step S206, where speed information indicating the speed of the simulated vehicle is acquired. The speed of the simulated vehicle can be calculated based on sensor information from the accelerator sensor 42 and the brake sensor 44, for example.

[0047] Next, the process proceeds to step S208, where sensor information is obtained from the accelerator pedal 20, the process proceeds to step S210, where sensor information is obtained from the brake sensor 44, the process proceeds to step S212, where sensor information is obtained from the steering angle sensor 40, and the process proceeds to step S214.

[0048] In step S214, an emotion information generation process is executed to generate emotion information of the subject's surprise. According to Izard, C., basic human emotions are classified into 10 types, namely, "interest / excitement, joy, surprise, distress / anxiety, anger, disgust, contempt, fear, shame, and guilt," and according to Ekman, P., into six types, namely, "happiness, anger, sadness, disgust, surprise, and fear." In the emotion information generation process, the emotion of surprise is detected from the subject's basic emotions based on an image of the subject taken by the camera 24. As a method for detecting the emotion of surprise, for example, the following method can be adopted. The subject's emotion of surprise is estimated from the image of the subject taken by the camera 24 using a trained model by deep learning. The input of the trained model is the image of the subject, and the output is an evaluation value of the emotion. The emotion evaluation value can take a value of, for example, 0 to 100, and when a predetermined value (for example, 15) is output, it is determined that the emotion is a surprise emotion. If a surprise emotion is detected, for example, "1" is generated as surprise emotion information, and if not detected, for example, "0" is generated as surprise emotion information. The trained model can be created, for example, by showing a video that is thought to cause a surprise emotion to a subject, photographing and recording the facial expression at that time, and detecting the movement of the subject's eyes, mouth, nose, cheeks, and eyebrows (parts of the subject's face). As a method for creating this trained model, for example, the technology described in Patent Document 3 (JP Patent Publication No. 2022-140053 A

[0044] to

[0046] ) can be adopted.

[0049] Other methods for detecting the emotion of surprise include, for example, Patent Document 4 (JP Patent Publication No. 2023-56152 A

[0143] to

[0145] ), Non-Patent Document 1 ("What is emotion recognition AI? An analysis mechanism for reading emotions from voice and facial expressions", [online], November 26, 2021, Ice Smiley Co., Ltd., [searched September 29, 2023], Internet<URL:https: / / aismiley.co.jp / ai_news / what-is-emotion-recognition / > ), or non-patent literature 2 ("Phenomenology of Surprise", [online], February 16, 2005, Ichiro Yamane, [searched September 29, 2023], Internet<URL:https: / / web.sugiyama-u.ac.jp / ~yamane / kenkyu / surprise.PDF / > ) can be adopted.

[0050] Next, the process proceeds to step S216, where one record including the elapsed time information, distance information, speed information, sensor information of the accelerator sensor 42, sensor information of the brake sensor 44, and sensor information of the steering angle sensor 40 acquired in steps S202 to S212, as well as the surprise emotion information generated in step S214, is recorded as log information in the storage device 46, the series of processes is terminated, and the original process is resumed.

[0051] [Evaluation Information Generation Process] Next, the evaluation information generating process in step S106 will be described. FIG. 5 is a diagram showing the evaluation information.

[0052] In the evaluation information generation process, for example, as shown in Fig. 5(a), the log information in the storage device 46 can be displayed as it is on the displays 12 to 16 as evaluation information. As shown in Fig. 5(a), one record is registered for each recording timing in the log information. Each record includes a field for registering the elapsed time from the start of the driving simulation, a field for registering the travel distance of the simulated vehicle from the start point of the driving simulation, a field for registering the speed of the simulated vehicle, a field for registering the sensor information of the accelerator sensor 42, a field for registering the sensor information of the brake sensor 44, a field for registering the sensor information of the steering angle sensor 40, and a field for registering the subject's surprise emotion information.

[0053] In the evaluation information generation process, for example, as shown in Fig. 5(b), a graph showing the accelerator opening, brake opening, and surprise emotion information in time series based on the sensor information of the accelerator sensor 42, the sensor information of the brake sensor 44, and the surprise emotion information can be displayed as evaluation information on the displays 12 to 16. In the example of Fig. 5(b), it can be seen that a surprise emotion (line indicated by reference numeral 50) is detected around elapsed time 1.4 [s].

[0054] Next, the effects of this embodiment will be described. In this embodiment, emotion information of a subject's surprise is generated in a driving simulation, and log information including the generated emotion information of surprise is recorded.

[0055] This makes it possible to perform an evaluation that takes into account the subject's feelings of surprise by referring to the log information.

[0056] Furthermore, in this embodiment, elapsed time information indicating the elapsed time from the start of the driving simulation is acquired, and surprise emotion information is recorded in association with the acquired elapsed time information.

[0057] This makes it possible to perform an evaluation that takes into account the subject's feelings of surprise in comparison with the elapsed time from the start of the driving simulation.

[0058] Furthermore, in this embodiment, distance information indicating the distance traveled by the simulated vehicle from the starting point of the driving simulation is acquired, and surprise emotion information is recorded in association with the acquired distance information.

[0059] This makes it possible to perform an evaluation that takes into account the subject's feelings of surprise in comparison with the distance traveled by the simulated vehicle from the starting point of the driving simulation.

[0060] Furthermore, in this embodiment, speed information indicating the speed of the simulated vehicle is obtained, and surprise emotion information is recorded in association with the obtained speed information.

[0061] This makes it possible to perform an evaluation that takes into account the subject's feelings of surprise in comparison with the speed of the simulated vehicle.

[0062] Furthermore, in this embodiment, sensor information regarding the operation of the handle 18 or the like by the subject is acquired, and surprise emotion information is recorded in association with the acquired sensor information.

[0063] This makes it possible to perform an evaluation that takes into account the subject's feeling of surprise in comparison with the operation of the handle 18 or the like.

[0064] Furthermore, in this embodiment, evaluation information regarding the subject's driving performance is generated based on the emotion information of surprise.

[0065] This allows the evaluation to take into account the subject's feelings of surprise, and also allows the subject to reflect on his or her own driving operation together with his or her feelings of surprise, providing effective feedback.

[0066] In this embodiment, step S100 corresponds to the control means of invention 1 or 5, step S106 corresponds to the generation means of invention 5, steps S208 to S212 correspond to the first acquisition means of invention 1, 2 or 5, and step S214 corresponds to the second acquisition means of invention 1 or 5. Also, step S216 corresponds to the recording means of invention 1 or 2, the sensor information corresponds to the operation information of invention 1, 2 or 5, and the simulated vehicle corresponds to the moving body of invention 1 or 5.

[0067] [Modifications] In the above embodiment and its modified examples, emotional information of the subject's surprise is recorded. However, the present invention is not limited to this. (1) A configuration for recording emotional information of the subject's fear, impatience, or discomfort, or (2) A configuration for recording emotional information of the above 10 or 6 types of emotions other than surprise and fear may be adopted.

[0068] Furthermore, in the above-described embodiment and its modified examples, emotion information regarding the presence or absence of surprise is generated, but the present invention is not limited to this, and it is also possible to adopt a configuration in which emotion information (for example, an analog quantity) regarding the degree, change, or aspect of surprise is generated.

[0069] In the above embodiment and its modified example, the surprise emotion information is recorded in association with the elapsed time information, etc., but the present invention is not limited to this, and it is possible to adopt a configuration in which the surprise emotion information is recorded in association with the following information: (1) elapsed time or remaining time from a predetermined point in the driving simulation, the time during the driving simulation, and other information related to a point in time during the driving simulation, (2) the distance traveled by the simulated vehicle from a predetermined point in the driving simulation, the position of the simulated vehicle in the virtual space, and other information related to the location of the simulated vehicle, (3) other information related to the driving history (including the operation history of the steering wheel 18, etc.), (4) information related to the acceleration, roll, pitch, yaw, and orientation of the simulated vehicle relative to the traveling direction of the road and other behaviors of the simulated vehicle, (5) information related to objects present within a predetermined range from the simulated vehicle, (6) information related to an event, and (7) biological information, gaze information, motion information, image information, audio information, odor information, or behavior history of the subject, environmental information around the subject, or information necessary for controlling the driving simulation.

[0070] Specific examples of events in (6) include a vehicle or person rushing out, crossing the road, cutting in line, aggressive driving, crossing immediately behind a car, and traffic violations. Examples of events in the case where the virtual moving body is a railroad vehicle include falling off a platform, rushing to board or disembark, obstructing a railroad crossing, and trespassing on the tracks. Examples of situations in which an event occurs include an intersection without a traffic light, an intersection with a traffic light, an intersection with poor visibility, inside a tunnel, overtaking a stopped vehicle, merging, crossing a crosswalk or crossing a road, and walking on a sidewalk. Examples of situations in which an event occurs in the case where the virtual moving body is a railroad vehicle include passing a railroad crossing, entering a platform, and closing the doors.

[0071] Specific examples of the configuration of (7) include: (1) a wearable device, vital sensor, or other sensor that detects the subject's biometric information; (2) an eye tracking sensor or other sensor that detects the subject's gaze; (3) an acceleration sensor, gyro sensor, or other sensor worn by the subject to detect the subject's movements; (4) a motion sensor, motion capture (including optical, inertial, image, magnetic, mechanical, or any other type; the same applies below) or other sensor that detects the subject's movements; (5) a camera or other sensor that detects an image of the subject; (6) a microphone or other sensor that detects the subject's voice; (7) an odor sensor that detects the subject's odor; (8) a pedometer, wearable device, or other sensor that detects the subject's behavioral history (e.g., number of steps and distance traveled); (9) a sensor that detects environmental information around the subject; or (10) a sensor that detects information necessary for controlling the driving simulation (hereinafter collectively referred to as "other sensors, etc.") can be connected to the control device 100 and a configuration can be adopted in which sensor information is acquired from one or more other sensors, etc. Biometric information includes, for example, information related to the subject's pulse, heart rate, body temperature, blood pressure, blood sugar, pupils, respiration, sweat, oxygen saturation, electromyography, electrocardiography, or brain waves. Environmental information includes, for example, information related to temperature, humidity, air pressure, wind direction, wind speed, illuminance, sunlight (e.g., ultraviolet) intensity, electromagnetic wave intensity, weather, voice, or smell.

[0072] In the above embodiment and its modified example, the surprise emotion information is recorded in association with the elapsed time information, etc., but the present invention is not limited to this, and for example, a configuration for recording the surprise emotion information under a predetermined condition can be adopted. Specific configurations include, for example, (1) a configuration for recording emotion information at a predetermined time interval, (2) a configuration for recording emotion information at a timing based on other predetermined rules, (3) a configuration for recording emotion information when the degree or change of surprise becomes a predetermined level or more or when the state of surprise becomes a predetermined state, (4) a configuration for recording emotion information when the subject's operation satisfies a predetermined condition (for example, abrupt steering or sudden braking), and (5) a configuration for recording emotion information when the behavior of the simulated vehicle satisfies a predetermined condition (for example, abrupt turning or sudden stopping).

[0073] Furthermore, in the above-described embodiment and its modified examples, log information or graphs are generated as evaluation information. However, the present invention is not limited to this. (1) A video (e.g., a replay video) of a driving simulation in which the subject performs operations; (2) information showing emotional information in order of acquisition or in chronological order; (3) information comparing emotional information with sensor information from the steering angle sensor 40 or one or more other sensors; (4) all or a part of the log information; (5) information among the log information that satisfies a predetermined condition (e.g., the degree or change of surprise is above a predetermined level, or the manner of surprise is a predetermined mode); or (6) information among the log information that does not satisfy a predetermined condition (e.g., the degree or change of surprise is below a predetermined level, or the manner of surprise is not a predetermined mode).

[0074] Furthermore, in the above embodiment and its modified examples, surprise emotion information is recorded or evaluation information is generated based on the surprise emotion information. However, the present invention is not limited to this, and it is also possible to adopt a configuration in which an event or object (hereinafter referred to in this paragraph as "event, etc.") is set, added, changed or deleted based on the surprise emotion information. Specific configurations that can be adopted include, for example, (1) a configuration in which an event, etc. that is the same as or similar to the event, etc. in which subject A's feeling of surprise is detected is set, added, or changed as the event, etc. of subject A; (2) a configuration in which an event, etc. that is the same as or similar to the event, etc. in which subject A's feeling of surprise is detected is set, added, or changed as the event, etc. of subject B who has the same or similar attributes as subject A (e.g., age, gender, occupation, level of care required, living environment, disease name, level of cognitive function, details of symptoms related to cognitive function, physical level, excretion situation, dietary situation, economic level, and drug use situation); (3) a configuration in which an event, etc. that is the same as or similar to the event, etc. in which subject A's feeling of surprise is not detected is deleted as the event, etc. of subject A; and (4) a configuration in which an event, etc. that is the same as or similar to the event, etc. in which subject A's feeling of surprise is not detected is deleted as the event, etc. of subject B who has the same or similar attributes as subject A.

[0075] Furthermore, in the above embodiment and its modified examples, surprise emotion information is recorded or evaluation information is generated based on surprise emotion information, but the present invention is not limited to this, and for example, the following configuration can be adopted.

[0076] The first configuration generates driving operation information based on the emotion information of surprise. For example, (1) a configuration for generating driving operation information informing the driver of the direction in which to turn the steering wheel 18, or (2) a configuration for generating driving operation information informing the driver of the timing to depress the accelerator pedal 20 or the brake pedal 22 can be adopted.

[0077] The second configuration generates images or videos showing the situation before, during, or after the detection of a feeling of surprise (hereinafter referred to as "when a feeling of surprise is detected") as images or videos from the subject's viewpoint (e.g., the view displayed on displays 12-16) or from another viewpoint (e.g., a top-down view looking down on the simulated vehicle from directly above). The time when a feeling of surprise is detected can be defined by time, position, or the like. The same definition applies hereinafter to the time when a feeling of surprise is detected.

[0078] The third configuration generates text showing the situation when the emotion of surprise is detected. For example, in the storage device 46, identification information for identifying an intersection without a traffic light, an intersection with a traffic light, an intersection with poor visibility, a right turn road, a straight road, a left turn road, a T-junction, a three-way intersection, a crossroad, a five-way intersection, inside a tunnel, etc. is set to an event or an object, and explanatory text of the situation is registered in an explanatory text table in association with the identification information. Then, identification information is obtained from an object present within a predetermined range from the simulated vehicle when the emotion of surprise is detected or when the emotion of surprise is detected, explanatory text corresponding to the obtained identification information is obtained from the explanatory text table, and text showing the situation when the emotion of surprise is detected is generated based on the obtained explanatory text. If explanatory text tables are prepared for each subject, each group (for example, a subject group or a customer or other organization), each purpose, or each other target, text with contents according to the target can be generated even for the same event or object. Instead of or in addition to text, images, videos, sounds, and other elements can be adopted.

[0079] The fourth configuration generates a behavior history when a surprise emotion is detected. The behavior history includes, for example, the subject's operation history of the steering wheel 18, biological information, gaze information, motion information, image information, audio information, odor information, behavior history, environmental information around the subject, information required for controlling the driving simulation, and sensor information from other sensors.

[0080] The fifth configuration generates statistical information on behavior, etc., when the feeling of surprise is detected. The statistical information on behavior, etc., includes, for example, statistical information on the history of behavior, etc., in the fourth configuration. With the fourth or fifth configuration, it is possible to analyze the state of the subject when the feeling of surprise is detected. For example, when a time limit is set for a driving simulation, it is possible to analyze whether the subject was surprised because he / she was in a hurry to make it in time, or was surprised without any such hurry.

[0081] In the sixth configuration, emotional information of surprise when an accident occurs during driving simulation or driving a real vehicle (hereinafter referred to as "emotional information at the time of the accident") is acquired and stored in the storage device 46, and evaluation information is generated based on the emotional information of surprise generated in step S214 and the emotional information at the time of the accident in the storage device 46. As the emotional information, for example, information on the presence or absence, degree, change, or state of surprise can be adopted. As the evaluation information, for example, information comparing the emotional information of surprise generated in step S214 with the emotional information at the time of the accident can be included. In a real vehicle, an accident cannot occur, but in the driving simulator 10, an accident can be virtually experienced. By utilizing this point, for example, it is possible to analyze (1) that the driver was surprised but not as surprised as when the accident occurred, (2) that the driver was surprised to the same extent as when the accident occurred, (3) that the driver was surprised as in (1) and (2) but did not subsequently cause an accident or make a driving mistake (was able to respond calmly), and (4) that the driver was surprised as in (1) and (2) but was unable to respond calmly.

[0082] The information generated in the first to sixth configurations may be recorded in the storage device 46, or may be displayed on the displays 12-16.

[0083] In addition, in the above embodiment and its modified examples, the evaluation information is displayed on the displays 12 to 16. However, the present invention is not limited to this, and a configuration in which a printer is connected to the control device 100 and the evaluation information is output to the printer can be adopted.

[0084] In the above embodiment and its modified example, the surprise emotion information is generated based on the image captured by the camera 24. However, the present invention is not limited to this. (1) One or more other sensors are connected to the control device 100, sensor information is acquired from the one or more other sensors, and surprise emotion information is generated based on the acquired sensor information. (2) One or more other sensors having a function of generating surprise emotion information are connected to the control device 100, and surprise emotion information is acquired from the one or more other sensors. (3) An external device (including the configurations (1) and (2)) that generates or acquires the surprise emotion information of the subject is connected to the control device 100, and the emotion information generated or acquired by the external device is acquired. The external device can be configured as, for example, a desktop PC, a notebook PC, or other computer. When motion capture is used, surprise emotion information of a subject with unilateral facial paralysis or the like can be effectively detected.

[0085] As a configuration for generating emotion information, for example, the following configuration can be adopted. The first configuration generates surprise emotion information of the subject by analyzing the state of the subject based on sensor information (biometric information) acquired from a sensor that detects biometric information of the subject.

[0086] The second configuration generates surprise emotion information of the subject by analyzing the subject's gaze based on sensor information (gaze information) acquired from a sensor that detects the subject's gaze.

[0087] The third configuration generates surprise emotion information of the subject by analyzing the subject's motion based on sensor information (motion information) acquired from a sensor that detects the subject's motion.

[0088] The fourth configuration generates emotional information of surprise of the subject by analyzing the facial expression of the subject based on sensor information (image information) acquired from a sensor that detects an image of the subject.

[0089] The fifth configuration generates surprise emotion information of the subject by analyzing the subject's conversation or tone of voice based on sensor information (audio information) acquired from a sensor that detects the subject's voice.

[0090] The sixth configuration generates surprise emotion information of the subject by analyzing the stress state of the subject based on sensor information (odor information) acquired from a sensor that detects the odor of the subject.

[0091] The seventh configuration generates surprise emotion information of the subject by analyzing the state of the subject based on sensor information (behavioral history) acquired from a sensor that detects the behavioral history of the subject.

[0092] More specifically, as a method for generating emotion information in the first to seventh configurations, for example, the technology of Patent Document 4 can be adopted.

[0093] In addition, in the above embodiment and its variations, displays 12-16 are used as devices for displaying the screen of the simulated driving, but this is not limited to this, and any number of displays (1 or 2, or 4 or more), head-mounted displays, or other display devices can be used.

[0094] In the above embodiment and its modified example, the driving simulation is controlled based on sensor information related to the operation of the steering wheel 18 by the subject, but the present invention is not limited to this, and a configuration for controlling a driving simulation or other movement simulation based on motion information related to the subject's motion can be adopted. As the motion information, for example, a motion capture device can be provided instead of or in addition to the steering wheel 18, and information related to the subject's motion detected by the motion capture device can be adopted. For example, a configuration for controlling a walking movement simulation with a pedestrian as a moving object can be adopted.

[0095] In addition, in the above embodiment and its variations, no description has been given of defining or playing back a simulated vehicle as an object, etc. during a driving simulation (e.g., during a program), but it is possible to adopt (1) a configuration in which a simulated vehicle is defined or not defined during a driving simulation, and (2) a configuration in which a simulated vehicle is played or not played back as a video.

[0096] Furthermore, in the above embodiment and its modified examples, the virtual space is reproduced as a video, but the present invention is not limited to this, and it is also possible to adopt a configuration in which a real space (for example, a live-action image) is reproduced as a video.

[0097] In addition, in the above embodiment and its variations, a simulated vehicle (e.g., an automobile) is used for the driving simulation, but the present invention is not limited to this, and other objects that can be used include (1) pedestrians, cyclists, and other living beings, (2) bicycles, motorcycles, railroad cars, and other vehicles, (3) aircraft, ships, and other vehicles, and (4) other moving bodies.

[0098] In the above embodiment and its modified example, the driving simulator 10 is realized as a single device, but the present invention is not limited to this and may be realized as a network system. As an example of a network system, a part or all of the functions of the driving simulator 10 may be configured as a virtual server on a server that provides a cloud computing service.

[0099] In addition, in the above embodiment and its variations, when executing the processes shown in the flowcharts of Figures 3 and 4, a program pre-stored in ROM 32 is executed. However, this is not limited to the above, and the program showing these procedures may be read into RAM 34 from a storage medium on which the program is stored and executed.

[0100] Here, storage media refers to semiconductor storage media such as RAM and ROM, magnetic storage type storage media such as FD and HD, optically readable storage media such as CD, CDV, LD and DVD, and magnetic storage type / optically readable storage media such as MO, and includes any storage medium that can be read by a computer, regardless of the reading method (electronic, magnetic, optical, etc.).

[0101] In the above embodiment and its modified example, the use of a large language model has not been described, but a configuration including, for example, (1) sensor information on the operation of the steering wheel 18 by the subject, (2) information on the subject's surprise emotion, (3) sensor information on one or more other sensors, or (4) log information, and inputting a request including a request to generate evaluation information on the subject's driving operation into the large language model, and an acquisition means for acquiring evaluation information output from the large language model in response to the request, can be adopted. As the large language model, for example, known technologies described in the Internet sites "https: / / chatgpt-lab.com / n / n418d3aa56f0b" and "https: / / agirobots.com / chatgpt-mechanism-and-problem / " can be adopted.

[0102] In the above embodiment and its modified example, the present invention is applied to the driving simulator 10 that evaluates the simulated driving, but it is not limited to this and can be applied to other cases without departing from the spirit of the present invention. For example, the present invention can be applied to any simulator that supports education, training, research, development, testing, etc. In the case of education or training, it refers to, for example, education or training of a subject, and in the case of research, development, or testing, it refers to, for example, research, development, or testing using the results of a movement simulation performed by a subject. [Explanation of symbols]

[0103] 10...driving simulator, 12-16...display, 18...steering wheel, 20...accelerator pedal, 22...brake pedal, 24...camera, 30...CPU, 32...ROM, 34...RAM, 38...I / F, 39...bus, 40...steering angle sensor, 42...accelerator sensor, 44...brake sensor, 46...storage device, 100...control device

Claims

1. A first acquisition means for acquiring operation information related to an operation or motion of a subject; a control means for controlling a movement simulation that reproduces, as a video, a view seen from a virtual moving object when the virtual moving object is moved, based on the operation information acquired by the first acquisition means; A second acquisition means for acquiring emotional information of the subject, such as surprise, fear, impatience, or discomfort; a recording means for recording log information of the movement simulation including the emotion information acquired by the second acquisition means.

2. In claim 1, said recording means records said emotion information under predetermined conditions, or records said emotion information in association with operation information acquired by said first acquisition means or with a time point during said movement simulation.

3. In claim 1, The control means causes an event to occur or an object to appear during the movement simulation, The simulator according to claim 1, wherein said recording means records said emotion information in association with an event generated by said control means or an object caused to appear by said control means.

4. In claim 1, a third acquisition means for acquiring biological information, gaze information, motion information, image information, audio information, odor information, or behavior history of the subject, environmental information around the subject, or information necessary for control of the control means; The simulator according to claim 1, wherein the recording means records the emotion information in association with the information acquired by the third acquisition means.

5. A first acquisition means for acquiring operation information related to an operation or motion of a subject; a control means for controlling a movement simulation that reproduces, as a video, a view seen from a virtual moving object when the virtual moving object is moved, based on the operation information acquired by the first acquisition means; A second acquisition means for acquiring emotional information of the subject, such as surprise, fear, impatience, or discomfort; a generating means for generating evaluation information regarding the movement simulation based on the emotion information acquired by the second acquiring means.

Images