Operation support device

Abstract

An operation support device that assists a user in moving a mobile object comprises: an information collection unit that collects data or information related to the user, the mobile object, or its surrounding environment; a situation recognition unit that uses the data or information collected by the information collection unit to recognize the situation of the user, the mobile object, or its surrounding environment and generates information related to the situation as recognition information; a recognition information storage unit that can store the recognition information generated by the situation recognition unit; a comparison unit that compares the recognition information received from the situation recognition unit with the recognition information already stored in the recognition information storage unit; and a storage control unit that, if the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, causes the comparison unit to newly store the received recognition information in the recognition information storage unit. The operation support device assists the user in moving the mobile object based on the recognition information stored in the recognition information storage unit.

Description

Technical Field

[0001] The present invention relates to an operation support device. More specifically, the present invention relates to an operation support device that supports a movement operation of a moving body such as an automobile by a driver.

Background Art

[0002] Originally, a moving body such as an automobile was one that a human boarding it moved by performing a driving operation. However, in recent years, the development of an automatic driving system has been progressing. An automatic driving system that supports the driving of an automobile by a driver has already been put into practical use, and development is still continuing everywhere toward the realization of fully automatic driving.

[0003] However, there are still many problems to be solved in order to realize fully automatic driving. Therefore, it is expected that for some time to come, a human will continue to be responsible for at least a part of the driving operation. The currently widespread automatic driving systems vary depending on the system in terms of the degree to which they support human driving operations, but it can be said that they actually function as driving support systems. As an example of an automatic driving system, the disclosed technology in Japanese Patent No. 5425170 for preventing collisions between automobiles can be cited.

Summary of the Invention

Problems to be Solved by the Invention

[0004] Although it is an automatic driving system that is steadily being developed, there are still cases where it becomes impossible to make an appropriate judgment on how to control an automobile or how to support the driving by a driver. This is because it is almost impossible to incorporate all possible huge situations and countermeasures into the control program in advance.

[0005] In view of such problems, an object of the present invention is to provide an operation support device that learns a control procedure of a moving body and makes the operation support for a driver more appropriate.

[0006] In particular, the operation support device according to the present invention is expected to compensate for inherent human shortcomings, such as difficulty in maintaining concentration and a tendency to make judgments based on preconceived notions. Conventional driving support systems have not yet been able to adequately compensate for these shortcomings.

[0007] The operation support device according to the present invention, which can provide such an objective, is expected to be in high demand, not only until fully autonomous driving is put into practical use, but even if fully autonomous driving systems are put into practical use in the future, particularly among those who enjoy operating moving objects. [Means for solving the problem]

[0008] To solve the above-mentioned problems, the present invention provides an operation support device that assists a user in operating a mobile body, comprising: an information collection unit connected to an information collection device of the mobile body and collecting data or information related to the user, the mobile body, or the surrounding environment of the mobile body acquired by the information collection device; a situation recognition unit that uses the data or information collected by the information collection unit to recognize the situation of the user, the mobile body, or the surrounding environment and generates information related to that situation as recognition information; a recognition information storage unit capable of storing the recognition information generated by the situation recognition unit; a comparison unit capable of receiving the recognition information generated by the situation recognition unit and comparing it with recognition information already stored in the recognition information storage unit each time recognition information is received from the situation recognition unit; and a storage control unit that, if the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, causes the recognition information received by the comparison unit to be newly stored in the recognition information storage unit, thereby assisting a user in operating a mobile body based on the recognition information stored in the recognition information storage unit.

[0009] Alternatively, in order to solve the above-mentioned problems, the present invention provides a storage medium that stores an operation support program for a computer to function as an operation support device to assist a user of a mobile body in moving the mobile body. The storage medium stores the computer as an information collection unit connected to an information collection device of the mobile body that collects data or information related to the user, the mobile body, or the surrounding environment of the mobile body acquired by the information collection device; a situation recognition unit that uses the data or information collected by the information collection unit to recognize the situation of the user, the mobile body, or the surrounding environment and generate information related to the situation as recognition information; a recognition information storage unit capable of storing the recognition information generated by the situation recognition unit; a comparison unit capable of receiving recognition information generated by the situation recognition unit and comparing it with recognition information already stored in the recognition information storage unit each time recognition information is received from the situation recognition unit; and a storage control unit that, if the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, causes the comparison unit to newly store the received recognition information in the recognition information storage unit, and stores an operation support program that assists a user in moving the mobile body based on the recognition information stored in the recognition information storage unit. [Effects of the Invention]

[0010] According to the present invention, for example, as various recognition information is accumulated, a unique effect is achieved in which the range of driving conditions in which appropriate operational assistance can be provided to the driver of a moving vehicle is expanded. [Brief explanation of the drawing]

[0011] [Figure 1] This is a functional block diagram showing one embodiment of the operation support device according to the present invention. [Figure 2] Figure 1 is a schematic diagram showing an example of the hardware configuration of the operation support device. [Figure 3] Figure 1 is a flowchart illustrating an example of the operation of the operation support device shown. [Figure 4] This diagram, following Figure 3, shows an example of the operation of the operation support device shown in Figure 1 in flowchart format. [Figure 5]This diagram, following Figure 3, shows an example of the operation of the operation support device shown in Figure 1 in flowchart format. [Figure 6] This diagram, following Figure 5, shows an example of the operation of the operation support device shown in Figure 1 in flowchart format. [Figure 7] This diagram, following Figure 5, shows an example of the operation of the operation support device shown in Figure 1 in flowchart format. [Modes for carrying out the invention]

[0012] Next, embodiments of the operation support device according to the present invention will be described in detail with reference to the attached drawings. In this application, the term "mobile body" is used to broadly include mobile machines having a power source and a mobile mechanism that is driven by power supplied from the power source, particularly land mobile machines such as four-wheeled automobiles, motorcycles, and autonomous mobile robots, aerial mobile machines such as airplanes and drones, and water or underwater mobile machines such as ships and submarines, regardless of whether they are manned or unmanned. In the following description of embodiments, the mobile body will be described as a four-wheeled automobile that travels on land, but the present invention is also applicable to mobile machines other than four-wheeled automobiles.

[0013] By mounting the operation support device 10, which is one embodiment of the present invention, on an automobile, the operation support device 10 can autonomously move the automobile under certain circumstances, or it can assist the driver operating the vehicle.

[0014] Referring to Figure 1, the vehicle operation support device 10 has a control unit 12 that controls the entire operation support device 10 and the vehicle's hardware components connected to the operation support device 10. The operation support device 10 further has a storage unit 14, which stores an operation support program 16 that defines the vehicle's control procedures related to assisting the driver in operating the vehicle, and data necessary for executing the program 16. The operation support program 16 may include an autonomous driving program that pre-defines conditions for autonomous driving and allows the vehicle to drive autonomously when those conditions are met, or the operation support program 16 may be included as part of such an autonomous driving program.

[0015] The operation support device 10 further includes a calculation unit 18 that performs calculation processing necessary for control related to operation support according to the operation support program 16, and an input unit 20 that acquires information necessary for controlling the automobile and provides the acquired information to the operation support device 10. The operation support device 10 further includes an output unit 22 that provides calculation results and control signals obtained from processing based on the operation support program 16 to the hardware components of the automobile on which the operation support device 10 is installed.

[0016] The input unit 20 is connected to various input hardware and can acquire various information or data in cooperation with the connected input hardware. Specific examples of input hardware connected to the input unit 20 will be described later. The input unit 20 is also connected to the control unit 12 and transmits the acquired information or data to the storage unit 14 under the control of the control unit 12.

[0017] The output unit 22 is connected to the control unit 12 and also to various output hardware. Specific examples of output hardware connected to the output unit 22 will be described later. Under the control of the control unit 12, the output unit 22 works in cooperation with the connected output hardware to send various information-containing signals or command signals to the occupants of the vehicle or to the outside of the vehicle.

[0018] FIG. 2 depicts, as an example of a hardware configuration, an operation support device 10 including these functional components 12, 14, 18, 20, and 22. An embodiment of the operation support device 10 according to the present invention can generally be implemented mainly by a computer 30, which is also commonly referred to as a microcomputer or a microcontroller. That is, it can also be understood that the computer 30 has come to have the functions of an operation support device 10, which is an embodiment of the present invention, by executing an operation support program 16.

[0019] The computer 30 includes a central processing unit (CPU) 32 that controls the operations of the components of the computer 30 and external devices connected to the computer 30, and a storage device that stores the operation support program 16, information acquired from outside the computer 30, calculation results obtained by the CPU 32, and the like.

[0020] The storage device of the computer 30 includes a non-volatile memory 34 that stores various programs, information, and data non-temporarily, and a volatile memory 36 that stores them temporarily. The operation support program 16 is stored non-temporarily in the non-volatile memory 34. Whether the data or information acquired from outside the computer 30 or the storage target such as the calculation result obtained by the CPU 32 is stored in the non-volatile memory 34 or the volatile memory 36 is determined by the operation support program 16 according to the use of the storage target and is allocated by the CPU 32 that executes the operation support program 16. A part of the operation support program 16, such as a frequently used program part, may be stored in the volatile memory 36 during startup of the computer 30.

[0021] The computer 30 further includes an input port 38 for receiving various kinds of information and data from outside the computer 30, and an output port 40 for transmitting various signals such as control signals and information-containing signals obtained from the arithmetic processing by the CPU 32 to the outside of the computer 30. In the computer 30, the non-volatile memory 34, the volatile memory 36, the input port 38, and the output port 40 are all connected to the CPU 32.

[0022] The CPU 32 of the computer 30 functions as the control unit 12 and the arithmetic unit 18 of the operation support device 10 shown in FIG. 1. In FIG. 2, in order to clearly show that the CPU 32 functionally contains the control unit 12 and the arithmetic unit 18 of the operation support device 10, the control unit 12 and the arithmetic unit 18 are shown as dashed blocks within the block of the CPU 32.

[0023] The non-volatile memory 34 and the volatile memory 36 of the computer 30 function as the storage unit 14 of the operation support device 10. In FIG. 2, in order to clearly show that the non-volatile memory 34 and the volatile memory 36 are components of the storage unit 14, the functional block of the storage unit 14 is depicted as a dashed block including the blocks indicating the non-volatile memory 34 and the volatile memory 36.

[0024] Note that the non-volatile memory 34 referred to in the present application means a storage device capable of maintaining the stored content even without power supply from the outside, and is a concept that widely includes not only a so-called ROM (Read Only Memory) but also a device capable of rewriting the once-stored information. Examples of storage devices that can be implemented as the non-volatile memory 34 in the computer 30 include flash memories capable of rewriting the once-stored information using electricity. However, other storage devices capable of rewriting information, not limited to flash memories, may be used as the non-volatile memory 34.

[0025] On the other hand, the volatile memory 36 referred to in this application means a memory device that can maintain its contents only when power is supplied. Typical memory devices that constitute the volatile memory 36 include RAM (Random Access Memory), more specifically DRAM (Dynamic RAM) or SRAM (Static RAM).

[0026] By temporarily storing the operation support program 16 in the non-volatile memory 34, the CPU 32 can perform autonomous driving or driver assistance for the vehicle equipped with the computer 30 according to the processing procedures specified in the operation support program 16. Furthermore, if the non-volatile memory 34 storing the operation support program 16 is a rewritable storage device such as flash memory, the program can be easily updated as needed.

[0027] The input port 38 and output port 40 of the computer 30 function as the input section 20 and output section 22 of the operation support device 10, respectively. Various input hardware is connected to the input port 38. For example, the input port 38 is connected to measuring instruments 42 such as a speedometer, accelerometer, steering sensor, various radars and LiDERs, an imaging device 44 that acquires image signals related to the surrounding environment, a biosensor 45 that detects the driver's state such as gaze, visual focus, face orientation, reaction speed to the external environment requiring operation, and drowsiness, a wireless receiving device 46 consisting of components such as an antenna used for receiving wireless signals from other vehicles or GPS signals, and a driving control system 48 such as a steering wheel and brakes. The input hardware (42, 44, 45, 46, 48) connected to the input port 38 may be broadly considered to constitute a part of the input section 20 of the operation support device 10. Furthermore, input hardware that acquires information inside or around the automobile, such as measuring instruments 42, imaging devices 44, biosensors 45, and wireless receiving devices 46, may also be considered as information gathering devices 49.

[0028] Various output hardware is connected to the output port 40. For example, a drive system 52 that transmits power from a power source and steering wheel operations to the drive wheels, a wireless transmission device 54 consisting of components such as an antenna used for transmitting wireless signals to other vehicles, a display 56 that displays images related to information acquired by the input unit 20 and / or calculation results by the calculation unit 18, and a speaker 58 that emits sounds related to the information acquired from the input unit 20 or calculation results by the calculation unit 18 are all connected to the output port 40. The output hardware (52, 54, 56, 58) connected to the output port 40 may be broadly considered as constituting a part of the output unit 22 of the operation support device 10. In addition, output hardware that notifies the driver of information acquired via the input unit 20 or calculation results of the calculation unit 18, such as the display 56 and speaker 58, may be considered as a notification device 59. The notification device 59 is not limited to the display 56 or speaker 58 shown in the figure. For example, a vibration device may be attached to the seat belt or seat of a car as a notification device 59.

[0029] By storing an operation support program 16 readable by the computer 30 in the non-temporary storage medium 60, the computer 30 can be made to read the operation support program 16 and operate as an embodiment of the operation support device 10 according to the present invention.

[0030] Computer-readable non-temporary storage media (non-temporary computer-readable media) 60 include, but are not limited to, RAM, ROM, UV-EPROM (ROM whose contents can be rewritten by ultraviolet irradiation), EEPROM (ROM whose contents can be rewritten electrically), flash memory, field-programmable gate arrays (FPGA), CD-ROM, digital multipurpose disks (DVDs), Universal Serial Bus (USB) memory, and other optical or magnetic storage.

[0031] The writing or updating of the operation support program 16 to the computer 30 can be performed by connecting the storage medium 60 on which the program 16 is written to the input port 38 by wire or wireless connection, and storing the program 16, which has been taken into the computer 30 via the input port 38, in the non-volatile memory 34 under the control of the control unit 12 (see arrow P1 in Figure 2).

[0032] The connection of the storage medium 60 to the input port 38 can be achieved by connecting a read driver to the input port 38 that is compatible with the type of storage medium 60 and allows the computer 30 to properly read the operation support program 16 stored in the storage medium 60. Alternatively, if the non-temporary computer-readable medium 60 is located within a wireless network, the program 16 may be imported into the computer 30 via a wireless receiving device 46.

[0033] By incorporating the operation support program 16 in this way, even automobiles equipped with conventional operation support systems can be given the functions of the operation support device 10 according to the present invention.

[0034] From here, an embodiment of the operation support device 10 according to the present invention will be described with reference to Figure 1 again. First, the components related to the driving of the vehicle by driver operation or autonomous driving will be listed. Among the components related to the driving of the vehicle, the main ones are the operation signal receiving unit 72 of the input unit 20, the drive output unit 74 of the output unit 22, and the drive control unit 76 of the control unit 12.

[0035] The operation signal receiving unit 72 is connected to the drive system 52 and receives signals related to the operation of the vehicle generated when the driver operates the drive system 52. The drive output unit 74 is connected to the drive system 52 and operates the drive system 52 according to the operation of the drive system 52 by the driver. The drive control unit 76 controls the operation signal receiving unit 72 and the drive output unit 74 and contributes to the smooth operation of the vehicle from the start to the end of the journey.

[0036] Furthermore, if an autonomous driving program is stored in the storage unit 14 as part of the operation support program 16 or separately from the operation support program 16, the drive control unit 76 controls the drive output unit 74 according to the autonomous driving program, and when the predetermined conditions specified in the autonomous driving program are met, the vehicle can be driven autonomously without any driving operation by the driver.

[0037] The components of the operation support device 10, which is responsible for collecting information related to the vehicle equipped with the operation support device 10 or its surrounding environment, include the information collection unit 82 of the input unit 20, the situation recognition unit 84 of the calculation unit 18, and the collection control unit 86 of the control unit 12.

[0038] The information gathering unit 82 is connected to various information gathering devices 49 and accurately and smoothly collects signals containing data or information related to the vehicle or its surrounding environment acquired by these devices 49. The situation recognition unit 84 uses the data or information collected by the information gathering unit 82 to recognize the current situation of the vehicle equipped with the operation support device 10 or its surrounding environment. The collection control unit 86 controls the operation of the information gathering unit 82 and the situation recognition unit 84 to ensure that a series of operations, from collecting information related to the vehicle or its surrounding environment to recognizing the situation based on the collected information, are carried out smoothly.

[0039] The components of the operation support device 10, which is responsible for determining whether the information recognized by the situation recognition unit 84 requires storage in the storage unit 14 and for performing the storage operation if it is determined that storage is necessary, include the recognition information storage unit 92 of the storage unit 14, the comparison unit 94 of the calculation unit 18, and the storage control unit 96 of the control unit 12.

[0040] The situation recognition unit 84 preferably associates the information collected by the information collection unit 82 with each other at regular intervals and comprehensively recognizes the situation of the vehicle and its surrounding environment. While a shorter interval is preferable for the situation recognition unit 84 to recognize the overall situation, it can be arbitrarily determined depending on the processing power of the CPU 32 and other factors.

[0041] The situation recognition unit 84 does not need to use all of the information collected by the information collection unit 82 for an overall situation recognition; it may appropriately select and discard the collected information.

[0042] The recognition information storage unit 92 stores information related to situations previously recognized by the situation recognition unit 84 as part of the storage area of ​​the storage unit 14. The comparison unit 94 receives the newly recognized information from the situation recognition unit 84 and accesses the recognition information storage unit 92. By comparing these two, the comparison unit 94 determines whether the information indicating the situation recognized by the situation recognition unit 84 is the same as the information already stored in the recognition information storage unit 92. The storage control unit 96 controls the operation of the comparison unit 94. Furthermore, if the comparison unit 94 determines that the information indicating the situation recognized by the situation recognition unit 84 has not yet been stored in the recognition information storage unit 92, the storage control unit 96 executes a process to store the unstored recognition information in the recognition information storage unit 92 of the storage unit 14.

[0043] The timing at which the storage control unit 96 controls the start and end of the process of storing the recognition information from the situation recognition unit 84 in the recognition information storage unit 92 is set so that any recognition information that is substantially different from the information stored up to that point is stored without omission. As a preferred example, the start timing of storage is when the comparison unit 94 determines that the recognition information received from the situation recognition unit 84 has not yet been stored in the recognition information storage unit 92, and the end timing of storage is when the comparison unit 94 determines that the recognition information that has continued to be received from the situation recognition unit 84 since the start of the storage process has become the same as the recognition information (past information) already stored in the recognition information storage unit 92.

[0044] The criteria used by the comparison unit 94 to determine whether the recognition information indicating the current situation recognized by the situation recognition unit 84 is the same as the information already stored in the recognition information storage unit 92 are preferably determined according to the processing power of the CPU 32 and the size of the memory areas of the memories 34 and 36 that constitute the recognition information storage unit 92. Even if the recognition information already stored in the recognition information storage unit 92 is not exactly the same as the recognition situation in the situation recognition unit 84 that it is comparing it with, the comparison unit 94 may determine that the information received from the situation recognition unit 84 is within the same range as the stored information if it can determine that the difference is small enough to adequately address the current situation by considering the stored information along with the execution of the autonomous driving program. Using such criteria prevents missing opportunities to store information useful for improving safety by making the same criteria excessively strict. However, it is preferable not to make the criteria too lenient so that substantially similar information is excessively stored, or so as not to impair the processing power of the CPU 32.

[0045] The recognition information stored in the recognition information storage unit 92 can be used for driving control during autonomous driving of a vehicle equipped with the operation support device 10 and for driver operation support during human driving. For example, if a vehicle equipped with the operation support device 10 encounters an imminent danger while driving, the recognition information storage unit 92 stores driving information at the time of the encounter and before and after, the surrounding environment, and the driver's actions taken to avoid the danger. This allows the memory unit 14 to store the warning signs of danger and avoidance actions. Based on this memory, the operation support device 10 can provide driver operation support or vehicle driving control to prevent future dangers from occurring at the time the warning signs are collected. Alternatively, even if no warning signs appear or the vehicle encounters danger because avoidance was not possible in advance, the operation support device 10 can provide driver operation support or vehicle driving control to avoid the crisis as safely as possible.

[0046] When the recognition information storage unit 92 stores recognition information exceeding a predetermined amount, or at predetermined intervals or at any arbitrary timing, the storage control unit 96 may edit the stored recognition information. Editing stored recognition information means, for example, when matching, similar, or common content is found in part or all of the recognition information stored in the recognition information storage unit 92 at multiple past points in time, the storage control unit 96 takes the lead in compressing the recognition information, i.e., improving the efficiency of the recognition information storage unit 92. In addition to this, editing stored recognition information may also include processes to improve the freshness of the information, such as updating time-series information and updating the average value of measurement data.

[0047] The components of the operation support device 10 that are responsible for notifying the driver of the comparison results from the comparison unit 94 include the notification control unit 102 of the control unit 12 and the comparison result notification unit 104 of the output unit 22. The notification control unit 102 receives the determination result from the comparison unit 94 of the calculation unit 18 and controls the operation of the comparison result notification unit 104 so that the comparison result notification unit 104 can notify the driver of the determination result via the notification device 59. The comparison result notification unit 104 performs the processing necessary for the operation of the notification device 59 under the control of the notification control unit 102. Examples of notifications to the driver by the notification device 59 include emitting light or displaying a pictogram from the display 56, or sounding a warning sound from the speaker 58.

[0048] Preferably, the notification device 59 performs a notification operation only when the comparison unit 94 determines that the current situation recognized by the situation recognition unit 84 is not common with past situation information stored in the recognition information storage unit 92. In this case, the period during which the information storage process in the recognition information storage unit 92 is performed coincides with the period during which the notification is sent from the notification device 59. Therefore, the notification control unit 102 may be connected to the storage control unit 96 in a manner that enables signal communication, and the notification control unit 102 may start controlling the comparison result notification unit 104 after receiving a signal from the storage control unit 96 indicating that storage control has started.

[0049] Preferably, the notification control unit 102, the comparison result notification unit 104, and the notification device 59 are configured to change the output content from the notification device 59 in accordance with the judgment basis of the comparison unit 94. For example, the output content from the notification device 59 is changed to different content depending on whether the surrounding environment of a moving vehicle has not been stored in the recognition information storage unit 92 or whether the surrounding environment has been stored but the driving operation content corresponding to that surrounding environment has not been stored in the recognition information storage unit 92. Examples of differences in the output content from the notification device 59 according to the judgment basis of the comparison unit 94 include the content of the voice, the color of the light emission, the presence or absence of light flashing, or the period.

[0050] In this configuration, a driver who receives a notification from the notification device 59 can reflect on their own driving actions. Alternatively, if the autonomous driving system had control of the vehicle when the notification device 59 issued the notification, the vehicle occupant who received the notification can take over the driving operations themselves.

[0051] The calculation unit 18 has an operation subject determination unit 108 that determines whether the vehicle equipped with the operation support device 10 is being primarily operated by a driver (human) or an automated driving system. The operation subject determination unit 108 is connected to the driving control system 48, such as the steering wheel, brakes, and accelerator, via an input port 38, and determines whether, within a predetermined period of time retrospectively from the present, the driving control system 48 has been operated by a human to the extent that it is not necessary to execute the autonomous driving program stored in the memory unit 14. Alternatively, the operation subject determination unit 108 may be configured to be connected to a biosensor 45 that monitors the driver's actions or state via an input port 38 and to receive monitoring results from the biosensor 45.

[0052] The operation support program 16 includes a program that controls the vehicle with safety as the top priority, based on the comparison results from the comparison unit 94 and the determination results from the operator determination unit 108, or, if sufficient countermeasures are not stored in the recognition information storage unit 92, recommends that the vehicle be driven manually.

[0053] Preferably, the operation support device 10 has a communication control unit 112 as part of the control unit 12 that controls communication with external wireless communication devices such as a GPS transmitter, a communication device managed by a traffic control center, or another vehicle equipped with one embodiment of the operation support device according to the present invention. The above-mentioned collection control unit 86 can be considered as part of the communication control unit 112, or as being connected to and cooperating with each other. With this configuration, the operation support device 10 can receive GPS information, traffic control information, and driving information of other vehicles via the wireless receiving device 46. Furthermore, regarding wireless transmission from the operation support device 10, a wireless transmission unit 114 is provided in the output unit 22, which is connected to a wireless transmission device 54 and enables the device 54 to smoothly transmit wireless signals from the operation support device 10, and the communication control unit 112 can be configured to control the operation of the wireless transmission unit 114. With this configuration, the recognition information acquired by the operation support device 10 can be shared with other vehicles that are capable of wireless communication.

[0054] The operation support device 10 may include a recognition information display unit 118 as a component of an output unit 22 that displays an overview or details of the collected information obtained via the information collection unit 82 or the recognition results by the situation recognition unit 84 in a display format such as characters or figures, and a display control unit 120 as a component of a control unit 12 that controls the operation of the recognition information display unit 118. The display devices for the collected information or recognition results by the recognition information display unit 118 and the display control unit 120 may be the same as the display 56 of the notification device 59 described above, which is used to notify and display the comparison results by the comparison unit 94, or a separate display device from the display 56 used for notifying the comparison results may be provided inside the vehicle.

[0055] If the display 56 serves both as a notification device 59 and for displaying collected information and recognition results, a portion of the display area may be allocated as a display area for notifying comparison results, or the display content of the display 56 may be switched when it becomes necessary to notify the driver of the comparison results. Alternatively, the display area of ​​the display 56 may be divided into multiple sections, with different types of information displayed in each section. For example, the first display area may display information as part of the navigation system, the second display area may display visual information acquired from the imaging device 44, and the third display area may display information obtained from other vehicles via wireless communication. Alternatively, multiple displays 56 may be installed in the vehicle, for example, with the first display showing information as part of the navigation system, the second displaying visual information acquired from the imaging device 44, and the third displaying information obtained from other vehicles via wireless communication.

[0056] Preferably, the calculation unit 18 includes a danger determination unit 124 that determines whether or not there is an imminent danger, such as contact or collision with an object outside the vehicle. The determination by the danger determination unit 124 may follow the provisions of the autonomous driving program stored in the memory unit 14. The calculation result by the danger determination unit 124 is used, for example, to determine whether or not to control the movement of the vehicle with safety as the top priority when an imminent danger suddenly occurs while the driver is manually operating the vehicle, i.e., whether or not to immediately stop the vehicle or proceed at a very slow speed. The danger determination by the danger determination unit 124 is determined solely from the perspective of whether or not there is danger, independently of the comparison with the recognition information already stored in the recognition information storage unit 92 by the comparison unit 94.

[0057] The control unit 12, storage unit 14, arithmetic unit 18, input unit 20, and output unit 22 are connected as components of the operation support device 10, enabling the transmission of signals containing various types of data, information, programs, or similar content between them. Furthermore, each sub-control unit (76, 86, 96, 102, 112, 120) contained within the control unit 12 can directly exchange signals with each other. As a result, data and information obtained by each component within the control unit 12 are smoothly shared throughout the entire control unit. The same applies to the connections between each sub-arithmetic unit (84, 94, 108, 124) contained within the arithmetic unit 18, enabling direct signal exchange between them.

[0058] Next, an example of the operation process of the operation support device 10 according to the present invention will be described with reference to Figures 3 to 7. When a mobile body such as a car equipped with the operation support device 10 becomes drivable, the operation support device 10 is also activated (step S10). When a car becomes drivable, it means that preparations for supplying driving force are completed inside the car, which means starting the engine if the car is an engine-powered car, or starting the power supply if it is an electric car.

[0059] The activated operation support device 10 activates various information collection devices 49 under the control of the collection control unit 86, and the information collection unit 82 collects data and information related to the vehicle and its surrounding environment (step S20). Each piece of data and information collected by the information collection unit 82 is supplied to the situation recognition unit 84. Based on the received data and information, the situation recognition unit 84 performs processing to recognize the situation of the vehicle equipped with the operation support device 10 or its surrounding environment (step S30).

[0060] The recognition information from the situation recognition unit 84 is supplied to the comparison unit 94, and under the control of the storage control unit 96, the comparison unit 94 compares the newly supplied recognition information from the situation recognition unit 84 with past information already stored in the recognition information storage unit 92. That is, it determines whether or not the recognition information supplied from the situation recognition unit 84 is new information to the operation support device 10 (step S40).

[0061] The storage control unit 96 receives the determination result from the comparison unit 94. If the comparison unit 94 determines that the recognition information supplied from the situation recognition unit 84 is not yet stored in the recognition information storage unit 92 (Yes in step S40), the storage control unit 96, having received the determination result, starts the processing necessary to store the recognition information from the situation recognition unit 84 in the recognition information storage unit 92 (step S50). The series of operations after the start of storage is shown in Figure 4.

[0062] Upon recognizing a determination result indicating the need for data storage or the start of data storage, the notification control unit 102 controls the comparison result notification unit 104 to activate the notification device 59 and notifies the driver of the vehicle equipped with the operation support device 10 that data storage of unstored information has begun (step S60). In other words, the notification control unit 102 notifies the driver that the current driving situation is one in which the operation support device 10 has no prior learning experience and may not be able to provide sufficient driving assistance.

[0063] Depending on its specific configuration, the operation support device 10 may perform either the start of accumulation of recognition information as described in step S50 or the notification to the driver as described in step S60 first, or both simultaneously.

[0064] The operator determination unit 108, under the control of the control unit 12, determines whether the vehicle equipped with the operation support device 10 is being primarily operated by a driver, i.e., a human, or by an automated driving system (step S70).

[0065] If the operator determination unit 108 determines that a human is primarily operating the vehicle (Yes in step S70), it transmits the determination result to the control unit 12, and in particular to the storage control unit 96. Upon receiving this determination result, the storage control unit 96 continues the storage process for the recognition information continuously supplied from the situation recognition unit 84 (step S80).

[0066] While the storage control unit 96 continues the storage process, the collection of information via the information collection device 49 and the recognition of the situation based on the collected information continue under the control of the collection control unit 86. Each time the situation recognition unit 84 generates comprehensive recognition information, the comparison unit 94 compares the generated recognition information with past information already stored in the recognition information storage unit 92 to determine whether the generated recognition information is new information (step S90).

[0067] If the comparison unit 94 determines that the recognition information sequentially sent from the situation recognition unit 84 is still new information, the storage control unit 96 takes the lead in continuing the storage process (Yes in step S90). In this way, as long as the recognition information sequentially sent from the situation recognition unit 84 while the vehicle equipped with the operation support device 10 is in motion is new information, the storage control unit 96 continues the storage process.

[0068] While driving and continuing the storage process, if the comparison unit 94 determines that the recognition information sent from the situation recognition unit 84 corresponds to information already stored in the recognition information storage unit 92 (No. in step S90), the storage control unit 96 terminates the storage process (step S100).

[0069] The series of accumulated information obtained in the process from steps S50 to S100 described above will serve as decision-making data for the autonomous driving program containing the operation support program 16, which will be used to appropriately support the driver's driving operations when the vehicle equipped with the operation support device 10 is driven under similar conditions in the future.

[0070] For example, based on the series of accumulated information obtained during the process from step S50 to S100, the vehicle's driving control can be made more appropriate in dealing with the arrival of a dangerous situation, thereby more reliably ensuring the safety of people inside the vehicle and those around the vehicle. Furthermore, if a sufficient amount of recognition information is accumulated in the recognition information storage unit 92, the operation support device 10 may be able to make the vehicle equipped with it drive autonomously under the same conditions as the accumulated recognition information.

[0071] Thus, according to the embodiment of the present invention, as the accumulation of recognition information in the recognition information storage unit 92 progresses, the number of matters that can be appropriately addressed by executing the operation support program 16 or the autonomous driving program containing it increases.

[0072] By the way, when the operator determination unit 108 determines that a human is not actively operating the vehicle, that is, that the vehicle equipped with the operation support device 10 is driving autonomously (No. in step S70), an example of the operation of the operation support device 10 is described below. In this case, the operation support device 10 continues the notification in step S60 and performs operation control to ensure the safety of the vehicle under the leadership of the drive control unit 76 (step S110). In most cases, the drive control unit 76 will execute control toward stopping the vehicle, and further, the drive control unit 76 will execute driving direction control as necessary. Control toward stopping the vehicle means that, with ensuring safety as the top priority, the vehicle will be brought to an immediate stop or slowed down to a speed at which it can be stopped at any time.

[0073] If the vehicle comes to a stop, the driver resumes driving the vehicle after confirming safety. Alternatively, a driver who recognizes the need for human intervention through a notification from the notification device 59 can take control of the vehicle themselves while it is moving slowly before coming to a stop, even though it was previously driving autonomously. At this time, the operator determination unit 108 determines that the operator of the vehicle has changed to a human, and the operation support device 10 can then determine that driving by human operation has resumed (step S120).

[0074] When the vehicle is driven again by human operation, and the recognition information obtained by collecting information about the vehicle's surrounding environment returns to what has been stored in the recognition information storage unit 92, the storage control unit 96 finishes the storage process (step S130). The operation support device 10 then returns to its normal operation as shown in Figure 3.

[0075] The series of accumulated information obtained in the processes from steps S50 to S70 and S110 to S130 described above will serve as decision-making data for the autonomous driving program containing the operation support program 16 to appropriately support the driver's driving operations when the vehicle equipped with the operation support device 10 is driven under similar conditions in the future. For example, if a critical situation that the autonomous driving program could not detect or deal with in advance is overcome by a sudden human driving operation, the operation support device 10 will store the precursory events leading to the critical situation and the driver's evasive actions in the recognition information storage unit 92, and will be able to provide more appropriate operational support to the driver when a similar situation is encountered in the future. As such accumulation progresses and a sufficient amount of recognition information is stored in the recognition information storage unit 92, the operation support device 10 will be able to autonomously drive the vehicle equipped with it under conditions similar to those of the already stored recognition information. Thus, according to the embodiment of the present invention, as the accumulation of recognition information in the recognition information storage unit 92 progresses, the number of matters that can be appropriately dealt with by the execution of the operation support program 16 or the autonomous driving program containing it increases.

[0076] Next, with reference to Figure 5, an example of how the recognition information stored in the recognition information storage unit 92 is used will be shown. Figure 5 shows an example of the process that the operation support device 10 executes when the comparison unit 94 determines that the recognition information supplied from the situation recognition unit 84 is information already stored in the recognition information storage unit 92 (No. in step S40).

[0077] The operator determination unit 108 determines whether the operator of the vehicle equipped with the operation support device 10 is a human (driver) (step S140). If it determines that the operator of the vehicle is the autonomous driving system (No. in step S140), the operation support device 10 continues autonomous driving (step S150). While continuing autonomous driving, the operation support device 10 continues the series of processing operations shown in steps S20 to S40 of Figure 3, such as information collection, recognition of collected information, and comparison of recognized information. Furthermore, if there are information items of recognized information that have not yet been accumulated in the surrounding environment of the vehicle during autonomous driving as performed in step S150, the accumulation control unit 96 may appropriately perform the information item accumulation process.

[0078] Even if it is determined that the primary operator of the vehicle is a human (Yes in step S140), the operation support device 10 continues the series of processing operations shown in Figure 3. At this time, the comparison content compared by the comparison unit 94 includes recognition information related to the driver's operations, so the operation support device 10 can also be considered to be monitoring the driver's operations (step S160).

[0079] The calculation unit 18, in particular through comparison processing by the comparison unit 94, determines whether the driver's driving operations of the vehicle are consistent with the recognition information stored in the recognition information storage unit 92 (step S170). In other words, the comparison unit 94 compares the information items related to the driver's driving operations among the recognition information generated by the situation recognition unit 84 with the corresponding information items of past information already stored in the recognition information storage unit 92, and determines whether the newly generated information items are new items.

[0080] If the calculation unit 18 determines that the driving operation is consistent with the accumulated recognition information (Yes in step S170), the operation support device 10 determines that the driving operation by the driver is appropriate and continues the process as before, namely the series of processing operations shown in steps S20 to S40 of Figure 3, which include information collection, recognition of the collected information, and comparison of the recognition information including monitoring of the driving operation.

[0081] On the other hand, if the calculation unit 18 determines that the driving operation is inconsistent with the accumulated recognition information (No. in step S170), the operation support device 10, under the control of the accumulation control unit 96, starts accumulating recognition information related to the driving operation that was determined to be inconsistent with the recognition information accumulated in the recognition information accumulation unit 92 (step S180). Furthermore, simultaneously with or around the time of the start of accumulation as shown in step S180, the operation support device 10, under the control of the notification control unit 102, notifies via the notification device 59 that the current driving operation does not match the accumulated recognition information related to the driving operation (step S190).

[0082] Upon receiving notification via the notification device 59, the driver can recognize that they are driving in a manner unfamiliar to the in-vehicle driving support system 10, in terms of at least one aspect of the vehicle's surrounding environment or their own driving actions. Therefore, the driving support system 10 can make the driver aware that they are driving on a road they have never driven on before, and give the driver an opportunity to pay extra attention to their driving actions. Alternatively, the driving support system 10 can make the driver aware that they are performing driving actions different from past driving actions under similar surrounding environments, and give the driver an opportunity to reconsider their driving actions.

[0083] The hazard determination unit 124 determines whether the driving operation currently being performed by the driver is safe or not (step S200). This determination is made, for example, from the perspective of whether there is an imminent danger of contact or collision with an object outside the vehicle for a vehicle equipped with the operation support device 10.

[0084] An example of the process that the operation support device 10 executes when the hazard determination unit 124 determines that the current driving operation is safe (Yes in step S200) will be explained with reference to Figure 6. The storage control unit 96 continues the storage process of the recognition information supplied from the situation recognition unit 84, in particular regarding the driver's driving operation content (step S210). As a result, the operation support device 10 can learn the newly stored driving operation content as a new example of hazard avoidance.

[0085] This accumulation process continues until the driver's driving operations match the accumulated recognition information (Yes in step S220). Once the comparison unit 94 confirms that the driver's driving operations match the accumulated recognition information, the accumulation control unit 96 terminates the accumulation control it has been performing (step S230). The operation support device 10 then returns to the series of processing operations shown in steps S20 to S40 of Figure 3.

[0086] On the other hand, an example of the process that the operation support device 10 performs when the danger determination unit 124 determines that the current driving operation is unsafe (No. in step S200) will be explained with reference to Figure 7. The operation support device 10 continues to provide notification in step S190 and, under the leadership of the drive control unit 76, performs operation control to ensure the safety of the vehicle (step S240). In most cases, the drive control unit 76 performs control toward stopping the vehicle, and further, the drive control unit 76 performs driving direction control as necessary.

[0087] If the vehicle comes to a stop, the driver will resume driving after confirming safety. Alternatively, even while driving slowly before coming to a stop, the driver may recognize the need to improve their driving skills through notifications from the notification device 59 and adjust their driving skills to be appropriate for the surrounding environment at that time.

[0088] The calculation unit 18, in particular the danger determination unit 124, determines whether the vehicle and its surroundings have escaped a dangerous situation due to the driver's appropriate driving operation (step S250). This determination continues until it is determined that the vehicle and its surroundings have escaped a dangerous situation. When the calculation unit 18 determines that the dangerous situation has passed (Yes in step S250), the storage control unit 96, upon receiving this determination result, terminates the storage control (step S260). The operation support device 10 then returns to its normal operation as shown in Figure 3.

[0089] Furthermore, according to the example of a series of operation processes of the operation support device 10 described with reference to Figures 3 to 7, the start condition for the storage process by the storage control unit 96 is when the comparison unit 94 determines that the newly generated recognition information by the situation recognition unit 84 is different from the information that has been stored in the recognition information storage unit 92 up to that point, and the termination condition is when the comparison unit 94 determines that the latest recognition information supplied from the situation recognition unit 84 is substantially the same as the information that has been stored in the recognition information storage unit 92 up to that point.

[0090] However, the start and end times of the storage process by the storage control unit 96 are not limited to this. For example, the storage control unit 96 may always perform the storage process when the vehicle is ready to drive, that is, when preparations for supplying driving force are complete. Alternatively, the storage control unit 96 may start the storage process when the vehicle starts moving and end the storage process when the vehicle stops. In these cases, there is no time lag between encountering a situation that requires storage and starting the storage process, which has the advantage of ensuring that all necessary recognition information is stored without omission. In the case where the storage process is always performed when the vehicle is ready to drive or while it is moving, a large amount of information will be stored in the recognition information storage unit 92, even if only temporarily. Therefore, it is preferable that the storage control unit 96 performs a process to delete recognition information that overlaps with already stored information from the recognition information storage unit 92 at predetermined intervals or timings.

[0091] <Compensating for human shortcomings> Embodiments of the operation support device 10 according to the present invention excel at compensating for bad human habits that sometimes lead to inappropriate driving operations. Examples of bad human habits that the operation support device 10 can compensate for include distracted driving that occurs when concentration is interrupted, failure to pay attention to the movements of other vehicles or pedestrians, overlooking road signs or road markings, or delayed reaction operations.

[0092] The human bad habits that the operation support device 10 can compensate for are not limited to the ability to maintain concentration. For example, driving operations such as obstructing vehicles or pedestrians that have priority under traffic laws due to misunderstandings or assumptions can also be considered manifestations of human bad habits and are subject to compensation by the operation support device 10. In addition, inappropriate driving operations caused by illusions such as visual attraction, tracking stationary vision, the blending phenomenon at tunnel entrances, the evaporation phenomenon at tunnel exits, or curving orientation while driving on curves can also be considered manifestations of human bad habits and are subject to compensation by the operation support device 10.

[0093] The operation support device 10 preferably has the situation recognition unit 84 generate recognition information that includes information items such as the driver's physical state or behavior resulting from normal driving operations, and stores this information in the recognition information storage unit 92. By continuously storing such recognition information, including information items, for at least a predetermined period, the situation recognition unit 84 can further recognize the driver's driving tendencies, average operation reaction speed, etc., and the storage control unit 96 can store these in the recognition information storage unit 92. The comparison unit 94 compares the information items such as driving tendencies and average operation reaction speed acquired in this way with the recognition information just supplied from the recognition information storage unit 92. If, as a result of this comparison, it is determined that the information items of the newly supplied recognition information do not match those obtained during normal driving operations, the operation support device 10, under the control of the notification control unit 102, can notify the driver via the notification device 59 that the current driving operation is not normal.

[0094] In this way, when the driver's concentration on driving operations is interrupted, the operation support device 10 can detect driver abnormalities that may lead to inappropriate driving operations early through the driver's physical condition or the vehicle's behavior, which are detectable from the outside, and take necessary measures to ensure safety.

[0095] In order to grasp the driver's driving tendencies in each run, the operation support device 10 preferably continues to collect information via an information collection device 49 such as a biosensor 45, perform recognition processing of driving tendencies and average operating reaction speed based on the collected information, and store recognition information including information items such as driving tendencies, from the time the preparation for supplying driving force is completed until a predetermined period of time or a predetermined distance has been traveled, or until a predetermined number of samples have been obtained. This allows the operation support device 10 to grasp the driving tendencies according to the driver for each run, and even if the person driving the vehicle changes each time, it becomes possible to provide operation support processing that is more suited to that driver. Alternatively, the operation support device 10 may be able to identify the driver by other means. For example, the driver can be identified by a biosensor 45 such as an iris sensor or a fingerprint sensor, and the driving tendencies etc. can be derived for each identified driver and stored in the recognition information storage unit 92. The information items such as driving tendencies and average operating reaction speed stored in the recognition information storage unit 92 in this way are used for comparison with the latest recognition information supplied from the situation recognition unit 84, which is performed by the comparison unit 94. The operation support device 10 can use these information items, for example, to determine the driver's level of concentration, as described later.

[0096] <Supplementary examples> From a safety perspective, it is preferable that the operation support device 10 either sets a large number of information items to be stored or sets a short information collection cycle or information recognition processing cycle for recognition information related to the driver's physical condition or the vehicle's behavior, which are likely to occur when the driver's concentration is lost.

[0097] For example, as a human habit, when driving absentmindedly, eye movement decreases or the eyes become unfocused. Therefore, it is preferable that at least a part of the equipment with the biosensor 45 be a device capable of detecting the driver's gaze. If the operation support device 10 is configured to generate recognition information that includes information items related to the state of the driver's gaze, the operation support device 10 can quickly detect any abnormalities appearing in the driver's gaze and send a warning notification to the driver via the notification device 59, or, depending on the situation, perform operation control to ensure safety.

[0098] Furthermore, if the operation support device 10 is connected to a biosensor 45 capable of detecting the driver's gaze, it can detect inappropriate behavior by the driver, such as neglecting safety checks or distracted driving, based on a comparison with recognition information previously stored in the recognition information storage unit 92, particularly recognition information obtained when the vehicle was driven with appropriate driving operations. In this case as well, as soon as it is detected, the operation support device 10 can send a warning notification to the driver via the notification device 59, or, depending on the situation, execute operation control to ensure safety.

[0099] Furthermore, inappropriate habits during driving, such as a lapse in the driver's concentration, often manifest in the behavior of the vehicle being driven. In this case as well, the operation support device 10 can detect the driver's inappropriate behavior by comparing it with recognition information previously stored in the recognition information storage unit 92, particularly recognition information obtained when the vehicle was being driven with appropriate driving operations. In this case as well, as soon as it is detected, the operation support device 10 can issue a warning notification to the driver via the notification device 59, or, depending on the situation, execute operation control to ensure safety.

[0100] Furthermore, the operation support device 10 can also detect and address situations where the driver, while driving, notices the presence of other moving objects such as other vehicles or pedestrians but decides that there is no danger and therefore fails to pay attention to those moving objects, i.e., when the driver fails to pay attention to the moving objects. More specifically, the comparison unit 94 of the operation support device 10 compares newly transmitted identification information from the situation recognition unit 84 with previously stored information and, by comparing it with identification information related to normal driving operations, determines that the driver's gaze upon recognizing a moving object does not correspond to the corresponding driving operation. In such cases, the operation support device 10 detects that the driver is failing to pay attention to the moving objects and can issue a warning to the driver and, if necessary, control the operation of the vehicle.

[0101] According to the operation support device 10, even if recognition information that can derive a solution to a given situation has not yet been stored in the operation support program 16 or other programs such as the autonomous driving program that runs in cooperation with it, the driver's driving operations can be newly stored. As a result, if a similar situation is encountered during subsequent driving, the operation support device 10 will be able to take action based on the recognition information newly stored in the recognition information storage unit 92.

[0102] Furthermore, for example, even if a solution can be derived from accumulated recognition information for a given situation, if the driver overcomes that situation through a different driving operation that has not been accumulated, the operation support program 16 of the operation support device 10 will also newly accumulate the details of that different driving operation that overcame the situation. Therefore, if a similar situation is encountered during subsequent driving, the operation support device 10 can select a more appropriate solution to that situation and control the operation of the vehicle according to the selected method. In other words, as the operation support device 10 accumulates more recognition information, it can increase the number of options for dealing with abnormalities while the vehicle is in motion.

[0103] Furthermore, it is not uncommon for drivers to prevent traffic troubles by using nonverbal communication such as eye contact and gestures between drivers and others (pedestrians or drivers of other vehicles, motorcycles, bicycles, or other mobile machines), or by relying on customary understandings based on the circumstances leading up to the situation. Understanding such nonverbal communication and customary understandings is virtually impossible for conventional operation support programs. Similarly, it would be difficult for the operation support device 10 according to the present invention to anticipate all traffic troubles from the initial setup stage and provide support to prevent all troubles. However, as the operation support device 10 accumulates nonverbal communication such as eye contact and hand gestures that determine priority traffic order, or as it accumulates the circumstances leading to the establishment of customary understandings, the range of situations in which it can make appropriate judgments for operation support or autonomous driving expands.

[0104] Furthermore, the movements and timing of hand signals issued by police officers or other police personnel engaged in traffic control duties, or by civilian traffic controllers deployed at road construction sites or commercial facility parking lots, have individual quirks depending on the person. As a result, drivers of automobiles may need time to understand the intent behind the hand signals issued by traffic controllers or controllers. Even in such cases, according to the embodiment of the operation support device 10 of the present invention, information regarding the movements of traffic controllers or controllers and the movements of other vehicles, etc., that move in accordance with them can be acquired via the imaging device 44, and the meaning of the hand signals and the tendency of the display timing can be recognized based on the acquired information.

[0105] As described above, according to the embodiment of the operation support device 10 of the present invention, as recognition information of various information items is sequentially accumulated, the range of situations in which appropriate operation support can be provided to the driver is expanded, and preferably the range of situations in which the automobile can be driven autonomously appropriately is also expanded. On the other hand, the operation support device 10 can compensate for habits that should not occur while driving an automobile, such as interruptions in concentration and tension, which can be said to be inherent human shortcomings. From this, it can be said that the operation support device 10 is a novel technology that adopts a configuration in which it controls the operation of the automobile in cooperation with the person driving the automobile equipped with it, while utilizing each other's strengths.

[0106] Thus, the technology that ensures the safe operation of the vehicle through a cooperative relationship that leverages the strengths of both the human and the operation support device 10 provides the driver with advantages that surpass even the cooperative system implemented in Level 3 autonomous driving, or conditional driving automation, which is said to be the maximum level of autonomous driving that can be put into practical use with the current level of technology. This is because, in current conditional driving automation technology, although the human entrusts the initiative in operating the vehicle to the system under certain circumstances, the human ultimately has to constantly monitor the autonomous driving, which requires them to maintain a constant state of tension. On the other hand, according to the embodiment of the present invention, assuming that there are moments when the human driver's concentration is interrupted during driving operations, the operation support device 10 can, so to speak, constantly monitor the driver. Furthermore, even when the human is monitoring the autonomous driving system under autonomous driving conditions, the operation support device 10 can recognize a lack of tension in the monitoring human from their gaze and other states as described above, and can prompt attention via the notification device 59.

[0107] While embodiments of the present invention have been described above, the specific methods for implementing the present invention are not limited to the embodiments described above. Modifications to the design, operating procedures, etc., can be made as appropriate, as long as the invention can be implemented. For example, auxiliary elements such as devices, circuits, and software used to assist in the performance of the components used in the present invention can be added or omitted as appropriate. [Explanation of Symbols]

[0108] 10 Operation support equipment 16 Operation support program 60 Storage medium 82 Information Gathering Department 84 Situation Awareness Unit 92 Recognition information storage section 94 Comparison Section 96 Storage Control Unit 104 Comparison result notification section

Claims

1. An operation support device that assists the user of the mobile body in moving the mobile body, the operation support device is An information collection unit connected to an information collection device of the mobile body, which collects data or information related to the user, the mobile body, or the surrounding environment of the mobile body acquired by the information collection device, A situation recognition unit that uses data or information collected by the information collection unit to recognize the situation of the user, the mobile object, or the surrounding environment, and generates information related to the situation as recognition information, A recognition information storage unit capable of storing the recognition information generated by the situation recognition unit, A comparison unit capable of receiving the recognition information generated by the situation recognition unit, and which compares the recognition information received from the situation recognition unit with the recognition information already stored in the recognition information storage unit, If the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, the storage control unit causes the recognition information received by the comparison unit to be newly stored in the recognition information storage unit, If the comparison unit determines that the recognition information received from the situation recognition unit has already been stored in the recognition information storage unit, the system includes an operation subject determination unit that determines whether or not the user is the one primarily operating the mobile body. An operation support device that, when the operation subject determination unit determines that the user is the one primarily operating the mobile body, monitors whether the user is performing the movement operation of the mobile body in a manner consistent with the recognition information stored in the recognition information storage unit.

2. In the operation support device according to claim 1, the operation support device further, An operation support device comprising a comparison result notification unit that, when the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, the comparison unit compares the recognition information with the recognition information already stored in the recognition information storage unit, and the storage control unit has started the process of storing the resulting recognition information in the recognition information storage unit, and notifies the user that the storage control unit has started the process of storing the resulting recognition information in the recognition information storage unit.

3. An operation support device according to claim 1 or 2, wherein the information items of the recognition information generated by the situation recognition unit include the physical state of the user when operating the mobile body or the behavior of the mobile body that manifests in relation to the physical state.

4. An operation support device according to claim 3, wherein the information collection unit is connected to a biosensor that detects data or information related to the user's physical condition.

5. In the operation support device according to claim 1 or 2, the operation subject determination unit determines whether the user or the automated driving system mounted on the mobile body is the one primarily operating the mobile body. An operation support device that, when the operation subject determination unit determines that the automatic driving system is the primary operator of the mobile body, and the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, causes the mobile body to perform movement control to ensure the safety of the mobile body.

6. An operation support program that causes a computer to function as an operation support device that assists the user of the mobile body in moving the mobile body, The operation support program controls the computer, An information collection unit connected to an information collection device of the mobile body, which collects data or information related to the user, the mobile body, or the surrounding environment of the mobile body acquired by the information collection device, A situation recognition unit that uses data or information collected by the information collection unit to recognize the situation of the user, the mobile object, or the surrounding environment, and generates information related to the situation as recognition information. A recognition information storage unit capable of storing the recognition information generated by the situation recognition unit, A comparison unit capable of receiving the recognition information generated by the situation recognition unit, and which compares the recognition information received from the situation recognition unit with the recognition information already stored in the recognition information storage unit. If the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, the storage control unit causes the recognition information received by the comparison unit to be newly stored in the recognition information storage unit, and If the comparison unit determines that the recognition information received from the situation recognition unit has already been stored in the recognition information storage unit, it will function as an operation subject determination unit to determine whether or not the user is the one primarily operating the mobile body. An operation support program that, when the operation subject determination unit determines that the user is the primary operator of the mobile body, causes the computer to monitor whether the user is performing the movement operation of the mobile body in a manner consistent with the recognition information stored in the recognition information storage unit.

7. In the operation support program according to claim 6, The function of the computer as the operator determination unit includes determining whether the user or the automated driving system mounted on the mobile body is the one primarily operating the mobile body. If the operator determination unit determines that the automatic driving system is the primary operator of the mobile body, and the comparison unit determines that the recognition information received from the situation recognition unit has not yet been stored in the recognition information storage unit, then an operation support program causes the computer to execute movement control to ensure the safety of the mobile body.

Images