Information generation device, information generation method, vehicle, display and speaker

Abstract

To provide an information generation device which can output required information without delay, an information generation method, and a vehicle.SOLUTION: An information generation device comprises: an output purpose determination unit 102 which determines an output purpose for outputting information to an operator by using operation information of at least either of the operator or an operation object; an output period determination unit 103 which determines an output period of the information by using the output purpose determined by the output purpose determination unit; a generation technique determination unit 104 which determines a generation technique for generating the information according to the output period; an information generation unit 105 which generates the information corresponding to the output purpose on the basis of the generation technique determined by the generation technique determination unit; and an output unit 106 which outputs the generated information corresponding to the output period, wherein the output period is set to a period in which the operator can recognize the information output from the output unit.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information generating device that generates output information for a worker, an information generating method, a vehicle, a display, and a speaker. [Background technology]

[0002] Patent Document 1 describes a technique for displaying information at an appropriate time.

[0003] Patent Document 1 states that "the driving plan information distribution system includes a driving plan information distribution device having a position information / road shape information acquisition unit that acquires vehicle position information and road shape information ahead, a sensor information acquisition unit that acquires output information from a group of on-board sensors, a driving plan information creation unit that creates driving plan information based on the position information / road shape information acquisition unit and the output information of the sensor information acquisition unit, a driving plan information notification unit that supplies the output information of the driving plan information creation unit to a driving plan information display unit for display, a wireless communication device that performs vehicle-to-vehicle communication with surrounding vehicles, and a driving plan information distribution control unit that controls the operation of each of these units at a predetermined timing, and when there is a change in the driving plan information for a certain time from the current time created by the driving plan information creation unit, the driving plan information distribution control unit outputs the changed driving plan information to the wireless communication device before the start time of the driving plan and a time before the start time of the driving plan that allows surrounding vehicles or drivers to respond, and notifies surrounding vehicles." [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Republished Patent No. 2019 / 234915 Summary of the Invention [Problem to be solved by the invention]

[0005] The process of generating information requires a generation processing time for various calculations, etc. This generation processing time increases or decreases depending on many factors, such as the size of the input information and output information, the communication speed with the generation model, the processing speed on the server side, and the generation means. Patent Document 1 does not take into consideration the generation processing time until the information is generated or the generation means that generates the information.

[0006] An object of the present invention is to provide an information generating device, an information generating method, and a vehicle that output necessary information without delay. [Means for solving the problem]

[0007] The present invention is characterized by comprising an output purpose determination unit that determines an output purpose for outputting information to a worker using motion information of at least one of the worker and the work object, an output period determination unit that determines an output period for the information using the output purpose determined by the output purpose determination unit, a generation method determination unit that determines a generation method for generating information with content corresponding to the output period, an information generation unit that generates information corresponding to the output purpose based on the generation method determined by the generation method determination unit, and an output unit that outputs the generated information corresponding to the output period, wherein the output period is set to a period during which the worker can recognize the information output from the output unit. [Effects of the Invention]

[0008] According to the present invention, it is possible to output necessary information at the necessary timing without delay. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing the overall configuration of an information generating device according to an embodiment of the present invention; [Figure 2] 2 is a configuration diagram showing an example of the configuration of a sensor used in the information generating device shown in FIG. 1. FIG. [Figure 3] 2 is a flowchart illustrating the operation of an output purpose determination unit shown in FIG. 1. FIG. [Figure 4]FIG. 4 is a flowchart for explaining a specific operation for determining an output purpose in the flowchart shown in FIG. 3. [Figure 5] 2 is an explanatory diagram illustrating the operation of an output period determination unit shown in FIG. 1. FIG. [Figure 6] FIG. 6 is an explanatory diagram illustrating the relationship between the output period and the information generation method shown in FIG. 5. [Figure 7] FIG. 10 is an explanatory diagram illustrating the relationship between output periods for events with different priorities. [Figure 8A] FIG. 10 is an explanatory diagram illustrating a first relationship between output periods for a plurality of events. [Figure 8B] FIG. 10 is an explanatory diagram illustrating a second relationship between output periods for a plurality of events. [Figure 9A] FIG. 2 is an explanatory diagram illustrating a first display example of information displayed on a display. [Figure 9B] FIG. 10 is an explanatory diagram illustrating a second display example of information displayed on the display. [Figure 9C] FIG. 10 is an explanatory diagram illustrating a third display example of information displayed on the display. [Figure 10] FIG. 10 is an explanatory diagram illustrating a data format in which an information type, a search label, and an information importance level are assigned to sentences that constitute generated information. [Figure 11] 2 is a configuration diagram showing the overall configuration of a vehicle control system incorporating the information generating device shown in FIG. 1. FIG. [Figure 12] 2 is a flowchart for explaining the overall operation of the information generating device shown in FIG. 1. FIG. [Figure 13] FIG. 2 is a configuration diagram showing a modified example of the information generating device shown in FIG. [Figure 14] FIG. 10 is a configuration diagram showing an example of the configuration of a sensor used in an information generating device (work machine) according to another embodiment of the present invention. [Figure 15] 15 is a flowchart for explaining the overall operation of the information generating device in the work machine shown in FIG. 14. [Figure 16]FIG. 10 is a configuration diagram showing an example of the configuration of a sensor used in an information generation device (education device) according to still another embodiment of the present invention. [Figure 17] 17 is a flowchart for explaining the overall operation of the information generating device in the education device shown in FIG. 16. [Figure 18] FIG. 10 is a flowchart illustrating the overall operation of an information generation device used in a subsequent generation model according to yet another embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments, and various modifications and application examples within the technical concept of the present invention are also included within the scope of the present invention. Note that the embodiments described below are merely examples for explaining the present invention, and some details have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be embodied in various other forms. [Example]

[0011] 1 is a block diagram showing the overall configuration of an information generating device according to a first embodiment. In this embodiment, an example is taken of driving an automobile (an example of a task), and when an operation that should be improved in the driving operation of a driver (an example of a worker) is predicted, the device is targeted for generating information in a timely manner to provide guidance on the operation that should be improved. The configuration and operation of the information generating device 100 according to this embodiment will be described below.

[0012] The situation information acquisition unit 101 acquires operation information of the driver and the automobile (an example of the work object, hereinafter referred to as the vehicle), as well as surrounding information regarding the surrounding situation, from various sensors, maps, and input information from the driver.

[0013] As shown in the example in Fig. 2, the various sensors include an on-board sensor 201 attached to the vehicle, a wearable sensor 202 worn by the driver, sensors 203 attached to infrastructure such as roads and surrounding facilities, and information on a server acquired via the Internet or information 204 disclosed by government agencies, etc. Information on a server includes, for example, the driver's driving history and driving tendencies, and information about roads frequently traveled by the driver. Note that this information may be stored in an edge device (on the vehicle) rather than in a server.

[0014] Specific information obtained from various sensors includes, for example, the vehicle's speed and acceleration, lateral position within the lane, distance to intersections and stop lines, vehicle condition (whether or not there is skidding, whether or not there is a malfunction, etc.), the driver's line of sight, the position and content of signs, road surface conditions, the position and relative speed of other vehicles, the position and relative speed of other traffic participants (pedestrians, motorcycles, etc.), weather, road type, congestion information, the driver's driving experience and usual driving tendencies, physical condition, whether or not there is impatience, etc. (whether or not there is a delay from the driving plan, pulse rate, etc.).

[0015] The output purpose determination unit 102 calculates "ideal driving" from the information acquired by the situation information acquisition unit 101, and determines "output purpose for generating information" based on the deviation between this ideal driving and the driver's actual driving.

[0016] "Ideal driving" refers to the conditions for ideal driving, defined by the vehicle's desired speed, acceleration, driving position, safety confirmation method, etc. Of course, other conditions can also be set.

[0017] In addition, the "output purpose of generating information" is the purpose of generating information required for foreseen events, such as "notification of the urgency of a dangerous situation," "suggestions for correcting usual driving habits," and "suggestions for driving plans."

[0018] The information used to define the ideal driving described above is calculated based on a "driving policy" set according to the purpose of use of the information generating device, "information on the traffic environment around the vehicle," and the like.

[0019] Here, the "driving policy" refers to, for example, the driving method that should be considered safe driving, the driving method that should be considered eco-driving, the driving method that should be considered commercial driving (for example, a driving method that is stricter than general safe driving and specifies upper limits on how to check for safety around the vehicle and on acceleration when starting and stopping), etc. Furthermore, ideal driving may be defined based on multiple driving policies.

[0020] Furthermore, "information about the traffic environment around the vehicle" refers to the positional relationship with other vehicles and pedestrians, road shape and type, speed limit, vehicle traffic division such as one-way streets, the obligation to stop near stop lines and pedestrian crossings, the positional relationship with traffic lights and the color of the traffic lights, etc. Note that, since there is a certain degree of individual difference in vehicle driving, the definition of ideal driving may be given a certain degree of flexibility.

[0021] The classification of such purposes is set in advance according to, for example, the type and degree of deviation between ideal driving and actual driving by the driver. The detailed method for determining the "output purpose for generating information" will be explained later using Figures 3 and 4.

[0022] The output period determination unit 103 determines an output period for outputting information according to the "output purpose for generating information" including information about urgency determined by the output purpose determination unit 102. This output period is determined for each information generation method, and the details of the method will be explained later using FIG.

[0023] The generation method determination unit 104 determines a generation method for generating information based on the output purpose determined by the output purpose determination unit 102, the urgency, the output period determined by the output period determination unit 103, and the expected generation processing time for each generation method.

[0024] Methods for generating information include: (1) A first generation technique that generates a simple notification such as a buzzer; (2) A second generation method that generates short notification messages based on rules from motion information. (3) A third generation method that uses information generated in similar situations in the past. (4) A fourth generation method, which generates information by modifying part of the information generated in a similar situation in the past (for example, "Checking the left rear is insufficient" to "Checking the right rear is insufficient"). (5) A fifth generation method that inputs motion information into a generative model to generate only high-priority elements. (6) A sixth generation method that inputs motion information into a generative model to generate all the information to be generated. Contains one or more of the following:

[0025] Here, the above-mentioned generation methods are prioritized, and the generation method determination unit 104 selects the method with the highest priority as the generation method among the generation methods that can output within the output period. Note that if the expected generation processing time required to generate information is short, the amount of information to be generated may be specified to be small (for example, the length of the sentence (number of words, etc.) may be set short) in a method that outputs past information as is, or outputs information after modifying part of it, or in a method that generates information using a generative model. In other words, if the time from the current time to the end time of the output start period is short, the length of the information to be generated using the fifth generation method or the sixth generation method may be specified to be short.

[0026] The expected generation processing time for each generation method may be set by measuring the average or maximum value in advance for each generation method. Furthermore, since the communication speed and processing speed with the server vary depending on the vehicle's location and time of day, the expected generation processing time can be set in association with location information and time information. Furthermore, dummy information (previous information) may be generated as appropriate when the possibility of an information generation request is low (for example, when the vehicle is stopped) and used for estimation.

[0027] The information generation unit 105 generates information to be output using the generation method determined by the generation method determination unit 104. The information generated by the information generation unit 105 is sent to the output unit 106 via the output information control unit 111, and the output unit 106 outputs the generated information to the driver. The function of the output information control unit 111 will be described later with reference to FIG. 11.

[0028] The method of outputting information may use a display (image), speaker (audio), steering wheel (vibration), pedal (reaction force), etc. installed in the vehicle, or may use a display or speaker on a smartphone or various tablets held by the driver that are not installed in the vehicle.

[0029] The information generating device is, for example, a computer, and includes a processor such as a CPU (Central Processing Unit), a memory, and a storage device such as an HDD (Hard Disk Drive).

[0030] Next, a method for determining an output purpose performed by the output purpose determination unit 102 will be described. Fig. 3 shows an example of a flowchart for determining an output purpose. The method for determining an output purpose is set in advance based on, for example, the amount of deviation between ideal driving and the actual driving by the driver. Here, a plurality of predetermined determination thresholds Th are set as the determination threshold for the amount of deviation (Da) between ideal driving and the actual driving by the driver, and the relationship is, for example, "Th1>Th2>Th3."

[0031] In step S101, the output purpose determination unit 102 determines whether the deviation (Da) between the ideal driving and the actual driving of the driver is equal to or greater than a first determination threshold (Th1). If it is determined in step S101 that the deviation is equal to or greater than the first determination threshold, the process proceeds to step S102. In step S102, the output purpose determination unit 102 sets the output purpose to "emergency notification regarding a dangerous situation (risk level (dr): high)." Examples of indicators of the determination threshold include the speed and acceleration of the driven vehicle. Furthermore, examples of situations where the deviation is equal to or greater than the first determination threshold (Th1) include a situation where the driven vehicle is moving toward an obstacle without decelerating. In such a situation, there is a risk that the driven vehicle will collide with the obstacle.

[0032] In step S103, after determining that the deviation (Da) between the ideal driving and the driver's actual driving is less than the first judgment threshold (Th1), the output purpose determination unit 102 determines whether the deviation (Da) is equal to or greater than the second judgment threshold (Th2). If it is determined in step S103 that the deviation is equal to or greater than the second judgment threshold (Th2), the process proceeds to step S104. In step S104, the output purpose determination unit 102 sets the output purpose to "Suggestion regarding correction of usual driving habits (risk level (dr): medium)." An example of a situation where the deviation is equal to or greater than the second judgment threshold is when the vehicle is decelerating toward an obstacle, but the deceleration is insufficient in light of the driving policy that defines ideal driving. In such a situation, there is little risk of a collision, but greater consideration must be given to safety.

[0033] In step S105, after determining that the deviation is less than the first and second judgment thresholds, the output purpose determination unit 102 determines whether the deviation between the ideal driving and the driver's driving plan is equal to or greater than a third judgment threshold (Th3). If it is determined in step S105 that the deviation is equal to or greater than the third judgment threshold (Th3), the process proceeds to step S106. In step S106, the output purpose determination unit 102 sets the output purpose to "suggestion regarding driving plan (risk level (dr): low)."

[0034] Examples of situations that exceed the third decision threshold include when congestion is detected on the route set by the driver and it is predicted that taking a different route will result in a faster arrival at the destination, or when a dangerous road surface condition (flooding, freezing, etc.) is predicted or detected on the driver's usual route, and it is considered desirable to change the route, cancel the trip, or issue a danger warning.When the deviation amount is equal to or exceeds the third decision threshold, it is desirable to suggest a change to the driving plan, although this is less urgent than when there is an immediate risk of a collision or other danger.

[0035] Next, we will explain how to set the output objective and the corresponding urgency (Ur) based on the predicted time until a certain assumed situation (sometimes referred to as an assumed event) occurs, which is caused by a discrepancy between ideal driving and the driver's actual driving.

[0036] Figure 4 shows how the urgency (Ur) is set according to the risk of an assumed event occurring due to deviation from ideal operation and the predicted time until the assumed event occurs. For example, the higher the risk (dr) and the shorter the event occurrence time (te), the higher the urgency (Ur) is set.

[0037] In Fig. 4, in step S201, the output purpose determination unit 102 calculates the risk (dr) of an anticipated event resulting from the deviation. This is based on the risk calculation flow in Fig. 3. Next, in step S202, the output purpose determination unit 102 predicts the event occurrence time (te) from the current time to the occurrence of a dangerous anticipated event resulting from the deviation from ideal driving. This predicts the event occurrence time (te) until a collision, for example, from the position, traveling direction, and speed of the subject vehicle and the positions, traveling direction, and speed of other vehicles and pedestrians.

[0038] Once the risk level (dr) and the event occurrence time (te) are determined, the urgency level (Ur) is determined in step S203. The urgency level (Ur) is determined by a map determined from the risk level (dr) and the event occurrence time (te).

[0039] In Figure 4, for example, the horizontal axis indicates the risk level (dr), (1) High risk situations, such as "a situation where a collision is predicted"..."High" (2) Situations with a moderately high risk, such as "a situation where approaching other people (vehicles, pedestrians, etc.) (the distance to other people will be less than a specified value)"... "Medium" (3) Low risk situations, such as "a situation where there are no other people nearby but it is expected that the speed limit will be exceeded" - "Low" (4) Situations where there is no risk, such as "a situation where a traffic jam is detected on the route"... "None" It is set to four levels:

[0040] The vertical axis represents the time at which the event occurred (te), (1) te≦t1, (2)t1 <te≦t2、 (3)t2 <te Here, t1 and t2 are predetermined judgment times, and the longer the time, the more time there is until the anticipated event occurs.

[0041] In this way, the urgency (Ur) is calculated from the combination of the risk (dr) and the event occurrence time (te). In this embodiment, the urgency levels are set from 1 to 6, and the higher the risk (dr) and the shorter the event occurrence time (te), the higher the urgency (Ur) is set.

[0042] The value related to the urgency (Ur) may be calculated using the following formula. In other words, the severity of the deviation results is weighted, and then multiplied by a value inversely proportional to the time until the occurrence. The value related to the severity of the deviation results can be calculated by preparing values ​​corresponding to various dangerous events in a database in advance.

[0043] The urgency (Ur) can be calculated using the following formula (1). Ur=k·(r / te)……(1) where "r" is the severity of the deviation, "k" is a coefficient, and "te" is the time to event before the hazardous event occurs.

[0044] Next, the concept of the output period (corresponding to time) for outputting information, which is determined by the output period determination unit 103, will be described with reference to FIG.

[0045] 5, the event occurrence time 501 (shown by a black triangle) at which the anticipated event is predicted to occur is the reference time for the various times and periods described below. As described above, this event occurrence time 501 is the reference time calculated based on the event occurrence time (te) from the current time to the occurrence of the dangerous event.

[0046] It is desirable that times and periods other than the event occurrence time 501 be set for each generation method determined by the generation method determination unit 104. The reason for this is that the amount of information output differs depending on the generation method, as will be described later with reference to FIG.

[0047] In FIG. 5, the output period determination unit 103 sets, based on the event occurrence time 501, an operation start time 502 at which the driver should first understand the content of the output information and then start the required driving operation.

[0048] This operation start time 502 can be obtained by calculating backwards from the event occurrence time 501 an operation execution period (to) during which the required driving operation can be performed. The operation execution period (to) may be a predetermined length of time, or may be determined based on the content of the driving operation.

[0049] Next, an information comprehension period (tu) corresponding to the time required for the driver to understand the content of the output information is calculated, and the information output time 503 is set by counting backward from the operation start time 502. The information comprehension period (tu) may be set by preparing a database of estimated required times for anticipated events and driving deviations in advance. Also, the information comprehension period (tu) may be set longer as the generation method and the amount of information (number of characters, length of audio, number of elements in images and videos, etc.) are predicted to be large.

[0050] Furthermore, if possible, information on the information processing ability (such as text reading speed) of each driver may be acquired, and the time may be shortened for drivers with better information processing ability. Information processing ability can be estimated, for example, from the time it takes for the driver to perform some operation (driving operation or pressing a button such as "OK" to end the output) in response to the presented output information.

[0051] Here, the information output period (tp) is defined as at least the period (tu+to) obtained by adding the information comprehension period (tu) and the operation execution period (to), or the period (tu+to+Δt) obtained by adding a short period (Δt) after the event occurrence time 501 to this added period (tu+to).

[0052] Next, an information output start period (ts) for starting the output of information is set, and information output start time 504 is set by counting backward from information output time 503. Here, the information output start period (ts) is set for the purpose of ensuring an information comprehension period (tu) for the driver to understand the information with ample time to spare.

[0053] In other words, it is desirable to start outputting information immediately after the information output start time 504, which allows the information comprehension period (tu) to be set long. Therefore, it indicates that output should start no later than the end time of the information output start period (ts), in other words, by the information output time 503. In this way, by having time for the information output start period (ts), it is possible to provide the driver with a sufficient information comprehension period (tu).

[0054] The end time of the information output start period (ts) (which corresponds to the information output time 503) is calculated backward from the information comprehension period (tu) required for the driver to understand the information, and the information output period (tp) for outputting information generated for anticipated events is set from this information output time 503.

[0055] If the information output can be started during the output start period (ts), the information output continues throughout the information output period (tp). In this case, time is ensured to generate the necessary information. On the other hand, if the information output cannot be started during the information output start period (ts) and is delayed into the information output period (tp), the information output using the generation method corresponding to the anticipated event at that time is set not to start.

[0056] The information output period (tp) may be set to end simultaneously with the event occurrence time 501 of the anticipated event, or may be set to end a little later (after Δt has elapsed) after the event occurrence time 501. The information non-output period (tn), which is set after Δt has elapsed from the event occurrence time 501, is set immediately after the end of the information output period (tp).

[0057] This is because the period immediately after the occurrence of an anticipated event is a period in which the driver should pay attention to the anticipated event and the surrounding situation. Also, this is a period in which the output of information for review, which will be described below, should not be performed, and during this period, no output is performed regarding the anticipated event, including the output of information for the purpose of review, which will be described below. Here, review refers to the action of reconfirming the information output in the anticipated event.

[0058] When generating an output of information for the purpose of looking back on an anticipated event, the output period determination unit 103 sets a lookback output start time 505 and a lookback output start period (tr), and a lookback output start time 506 and a lookback information output period (tb). The lookback information output period (tb) may be set immediately after the end of the information non-output period (tn), or may be set to wait until the driving operation has settled down (for example, until the next traffic light), or may be set after the driving operation has ended, such as when the driver has arrived at the destination.

[0059] The length of the review information output period (tb) is preferably set in the same manner as the information output period (tp), that is, according to the amount of information and the driver's information processing ability. Furthermore, the method of determining whether or not to output the review information may be to always output the information for an anticipated event, or to output the information when the generation method for generating all the information that needs to be generated is not yet ready and another method is used.

[0060] Furthermore, it may be configured to output when neither output is in time. In other words, if there is no generation method that can output information within the output period, the output period determination unit can switch the output purpose to review and determine the output period and generation method.

[0061] Alternatively, a function for determining whether the output purpose has been achieved may be provided. As will be described later, if a high-priority output request for another anticipated event occurs and the information output cannot be continued within the scheduled length of time, it may be determined that the output purpose has not been achieved, and information for review may be generated and output. Alternatively, if the driver is repeatedly driving in the same manner, information for review may be output.

[0062] Next, examples of the information output period (tp) determined by the output period determination unit 103 for each information generation method will be described with reference to Fig. 6. In this example, the information output period (tp) based on three generation methods is compared: a generation method (method A) that generates information including all information that should be generated for a certain anticipated event, a generation method (method B) that generates only information with high priority, and a generation method (method C) that generates only information with a simple notification such as a buzzer.

[0063] Here, the amount of information output is predicted to be in the following relationship: generation method A > generation method B > generation method C, so the information comprehension period (tu) required for the driver to understand the information in generation method A is longer than the information comprehension period (tu) required for the driver to understand the information in generation method B. As a result, the information output start period (ts) at which the output of information in generation method A should start starts earlier than the information output start period (ts) at which the output of information in generation method B should start. In other words, the amount of information can be said to be the amount of information that fits within the information comprehension period (tu).

[0064] As such, because the start times of the information output start period (ts) are different, the information output start period (ts) of generation method B is later than that of generation method A. The end time of the information non-output period (tn), during which output should not begin, is set to the same time regardless of the generation method. This is because it is believed that the amount of time during which the driver should pay attention to anticipated events and the surrounding situation remains the same regardless of the generation method.

[0065] On the other hand, in generation method C, there is no information comprehension period (tu) required for the driver to understand the output, but rather an information output start period (ts) is set up until just before the anticipated event occurs, and depending on the design policy, an information output period (tp) is set up until a little after the anticipated event occurs, and a non-output period (tn) is set up during which no information should be output for a certain period of time.

[0066] The reason for this is that simple notifications such as buzzers contain less information and place a low cognitive load on the driver's information processing ability, making them less likely to hinder the driver's response to anticipated events, and are thought to have the effect of prioritizing notifications until just before an anticipated event occurs.

[0067] Next, a case where a request for outputting information on a certain anticipated event occurs during a period in which information on another anticipated event is being output will be described with reference to FIG.

[0068] For example, if the vehicle's trajectory sags toward the center of the lane while turning left at an intersection, the system outputs information to advise drivers to be careful with their driving. However, if the system detects a motorcycle approaching from the front while this information is being output, it determines that there is a possibility of a collision with the motorcycle based on the relative speed between the vehicle and the motorcycle and the driver's line of sight, and generates a request to output information urging drivers to be careful of the motorcycle.

[0069] In this situation, the hypothetical event of the trajectory widening while turning left (called hypothetical event A) alone does not pose an immediate risk of collision or the like, so the urgency is not high. In contrast, the event of a collision with a motorcycle (called hypothetical event B) is more urgent. For this reason, the priority of information output is higher for warning motorcycles. We will explain how to output information in this situation.

[0070] In Figure 7, for the event occurrence time 701 when the expected event A is predicted to occur, the operation start time 702 at which the driver should understand the content of the information and begin acting, the information understanding period (tu) required for the driver to understand the output, the information output start period (ts) at which the output of information begins, the information output period (tp) at which the generated information is output, and the information non-output period (tn) at which no information is output are set using the method described in Figures 5 and 6.

[0071] Next, when the event occurrence time 707 at which the expected event B is predicted to occur is calculated, the operation start time 708 at which the driver should understand the content of the output and begin to act, the information understanding period (tu) required for the driver to understand the output, the information output start period (ts) at which the output of information begins, the information output period (tp) at which the generated information is output, and the information non-output period (tn) at which no information is output are set.

[0072] Then, actual information output for the anticipated event A is executed during the period (tA) from the information output start time 709 for the anticipated event A to the information output start time 710 for the anticipated event A.

[0073] On the other hand, after the information output start time 710 for the anticipated event B, the output of information for the anticipated event B is executed over a period (tB). Here, the information is output over a period including the information output time 710 and the information non-output period (tn). However, as mentioned above, it may be output up to the event occurrence time 707.

[0074] Furthermore, if it is determined that there is a high possibility that information regarding anticipated event B will be requested before information regarding anticipated event A is output or before the information is generated, the output period for information regarding anticipated event A may be changed, or information regarding anticipated event A and anticipated event B may be output together.

[0075] Next, a method for changing the information output period (tp) for anticipated event A will be described with reference to FIG. 8A.

[0076] For the event occurrence time 801 when the expected event B is predicted to occur, the operation start time 802 at which the driver should understand the content of the output and begin to act, the information understanding period (tu) required for the driver to understand the output, the information output start period (ts) during which the output of information begins, the information output period (tp) during which the generated information is output, and the information non-output period (tn) during which no information is output are set using the method described in Figures 5 and 6.

[0077] Next, a new operation start time 803 at which the driver should understand the content of the information output for anticipated event A and start the required driving operation is set to the earlier of the original operation start time 804 for anticipated event A shown by the dashed line and the information output start time 805 of the information output start period (ts) at which the output of information for anticipated event B starts. In this case, the operation start time 803 is set to the information output start time 805.

[0078] 8A, information output start time 805 of the information output start period (ts), at which the output of information for anticipated event B starts, is earlier than the original operation start time 804 for anticipated event A. Therefore, the original operation start time 804, at which the content of the information for anticipated event A should be understood and driving operation should be started, is set to information output start time 803 of the information output start period (ts), at which the output of information for anticipated event B starts, and operation start time 804 is brought forward.

[0079] Next, the information comprehension period (tu) required for the driver to understand information about anticipated event A is used to set the information output start period (ts) for starting to output information about anticipated event A. Then, the information output period (tp) for anticipated event A is determined to be the period from the end time of the information output start period (ts) for anticipated event A to the end time of the information output start period (ts) for anticipated event B.

[0080] Furthermore, the information non-output period (tn) during which information for anticipated event A should not be output is set to a range from the end time of the information output period (tp) for anticipated event A to a predetermined time that is slightly beyond the event occurrence time 806 of anticipated event A, similar to the method described in Figures 5 and 6.

[0081] As a result, the information that is actually output is the period from the information output start time 807 for the anticipated event A to the information output start time 808 for the anticipated event B, and during this period (tA), the output of information for the anticipated event A is executed. On the other hand, during the period (tB) from the information output start time 808 for the anticipated event B to the end time of the information output period (tp) for the anticipated event B, the output of information for the anticipated event B is executed.

[0082] By using the above method, once the driver has understood the content of the information regarding anticipated event A and is in a state where they can perform driving operations, they can then recognize the content of the information regarding anticipated event B, thereby enabling the driver to respond to both anticipated events A and B.

[0083] 7 and 8A illustrate a case where anticipated event B, which occurs later in time, is more urgent than anticipated event A, which occurs earlier. On the other hand, if anticipated event A, which occurs earlier, is more urgent, or if anticipated events A and B are equally urgent, it is possible to start outputting information about anticipated event B after outputting information about anticipated event A is completed.

[0084] The determination of completion of output of information on anticipated event A may be made on the condition that the output period set by the output period determination unit 103 has ended, or the time of occurrence of anticipated event A may be set as the completion time of information on anticipated event A even before the end of the output period.

[0085] In addition, provided that driving operations have begun in accordance with the information on anticipated event A, it may be determined that the gist of the information on anticipated event A has been understood and the purpose of outputting the information on anticipated event A has been achieved, and the output period for the information on anticipated event A may be terminated early.

[0086] Next, a method for outputting information combining expected events A and B will be described with reference to FIG. 8B.

[0087] First, the operation start time 813, when the driver should understand the content of the information and start driving operation, and the information comprehension period (tu) required for the driver to understand the information are set relative to the earlier of the event occurrence time 811 of anticipated event A and the occurrence time 812 of anticipated event B. Furthermore, the information output start period (ts), when the output of information starts, the information output period (tp), when the generated information is output, and the information non-output period (tn), when no information is output, are set.

[0088] The end times of the information output period (tp) during which the generated information is output and the information non-output period (tn) are set to coincide with the occurrence time of the event that occurs later (in FIG. 8B, the occurrence time of expected event B (812)).

[0089] This method makes it possible to output information about anticipated events A and B with ample time to spare, regardless of the urgency of anticipated events A and B, or the time interval between the occurrence of anticipated events A and B. This allows the driver to respond to both events with sufficient information.

[0090] The generation method determination unit 104 and the information generation unit 105 create information as shown in Figures 9A to 9C. In the examples shown in Figures 9A to 9C, output examples of three methods are shown: a generation method (A) that generates all information to be generated, a generation method (B) that generates only high-priority elements, and a generation method (C) that uses motion information, in the case where a method of displaying text and video using a display mounted on a vehicle is used.

[0091] 9A shows a display 901 that displays information generated by a generation method that generates all the information that should be generated. Text 902 and video 903 are displayed on the display 901.

[0092] Text 902 consists of two "sentences" that point out problems with the current driving and provide detailed instructions on how to improve it. Video 903 shows the positions of the vehicle 904 and the parked vehicle 905, as well as the current vehicle trajectory 906 (dotted arrow) and the ideal trajectory 907 (solid arrow), explaining the degree of deviation.

[0093] The example in Figure 9B shows a display 908 displaying information generated using a generation method that generates only high-priority elements. Text 909 consists of a single "sentence" and succinctly instructs how to improve. Image 910 shows only the position of a parked vehicle 911 and an ideal trajectory 912, with fewer elements drawn.

[0094] The example in Figure 9C shows a display 913 displaying information generated using a generation method that uses motion information. Text 914 consists of a single short "sentence" and succinctly instructs on a generalized improvement method. Image 915 is a simple pictogram 916 (a steering wheel pictogram) indicating that the operation to be improved is steering.

[0095] Here, in the generation method using motion information in Figure 9C, the information to be output is generated using a rule base or the like from information obtained from a sensor, so there is no need to communicate with a generative model or server, and information can be generated in a short time.

[0096] In this way, by using these generation methods depending on the grace period between when the demand for information generation arises and when output begins, educational guidance as shown in Figure 9A can be provided when there is grace period, and even when there is little grace period, a generation method that provides as much information as possible as shown in Figures 9B and 9C can be selected to provide information to the driver.

[0097] Although these examples show text and images displayed on a display, the method of outputting information is not limited to a display, and any method that can be perceived by the driver may be used. For example, information may be output as audio, such as text being read aloud, or as music, or may be output as a light that lights up or flashes, or as a color that has meaning. Alternatively, information may be output using vibrations, reaction forces, shape changes, temperature changes, etc., applied by the steering wheel, pedals, seat, etc. that the driver is touching.

[0098] Furthermore, the input information for generating the information may be current information from various sensors, etc., but when a change in the situation is predicted, predicted values ​​may be used as input information. For example, if the current speed and acceleration are not a problem in the moment, but the acceleration is being maintained and it is predicted that the speed will deviate from the appropriate speed in the near future, the timing at which the vehicle speed will deviate may be predicted, and the generation of information to warn about the vehicle speed may be started accordingly.

[0099] This increases the likelihood that information generation can be completed in advance, making it easier to output messages to the driver at the appropriate time. Furthermore, using the results of situation predictions, multiple pieces of information may be generated in advance in preparation for cases where it is predicted that there will be a high need to output information, and the information may be stored appropriately in the memory on the edge device side. In other words, information that may be generated now or in the near future can be generated in advance and stored on the edge device side. This makes it easier to output information appropriate to the situation, even if communication with the server becomes poor.

[0100] Next, using Figure 10, we will explain how to store the generated information in a database as a data format by dividing it into elements and adding information about "search labels" and "importance of information" to it, using information consisting of multiple "sentences" (in this case, a sentence) as an example.

[0101] The target information is an example of generated text such as text 1001 and text 1002 .

[0102] These sentences 1001 and 1002 are stored in a database as a data format 1003. The data format 1003 is assigned the following items: "element," "information type," "search label," and "information importance" of the sentence. As shown in the data format 1003, the generated sentences 1001 and 1002 are divided into, for example, "sentences" (or phrases), which are defined as elements, and the information type, search label, and information importance are associated with these and stored.

[0103] Here, the information type, search label, and information importance may be assigned by analyzing the generated text using a text analysis method, or, for example, when generating texts 1001 and 1002, the structure of the text may be specified (problem operation information in the first sentence, cause information in the second sentence, improvement method in the third sentence, etc.), and that information and information on urgency at the time of generation may be assigned on a rule-based basis.

[0104] Furthermore, by starting the search from elements with high information importance when generating data using past output, it is possible to extract important elements first, making it possible to respond even when the output start period is short.

[0105] In this way, the generation method determined by the generation method determination unit 104 is one of the following: (1) a first generation method that generates a simple notification; (2) a second generation method that uses action information; (3) a third generation method that uses information generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the information generated in a similar situation in the past; (5) a fifth generation method that inputs action information into a generation model and generates it using high-priority elements; or (6) a sixth generation method that inputs action information into a generation model and generates all the information.

[0106] Furthermore, the third generation method or the fourth generation method determined by the generation method determination unit 104 divides the information to be generated into elements and generates it using information with high priority in order to shorten the time required to generate information. Furthermore, the third generation method or the fourth generation method determined by the generation method determination unit 104 generates information by searching elements stored in a database with reference to search labels. Furthermore, the generation method determination unit 104 has a function of dividing the sentence generated by the fifth generation method into specific elements, assigning search labels to each element, and storing the elements in a database.

[0107] In addition, when the time from the current time to the end time of the output start period is short, the generation method determination unit 104 determines that the input information for generating information using the fifth generation method or the sixth generation method is input information of high importance.

[0108] Furthermore, if the time from the current time to the end time of the output start period is short, the length of the information to be generated by the fifth generation method or the sixth generation method can be specified to be short. Furthermore, when estimating the time required for the fifth generation method or the sixth generation method, the generation method determination unit 104 can generate information by the fifth generation method or the sixth generation method at a time when there is no request to generate information.

[0109] Next, a specific system configuration and a flowchart of this embodiment will be described with reference to FIGS.

[0110] 11 shows the configuration of an information generating device 100 related to this embodiment in a generating device control system 10 mounted on a vehicle. This generating device control system 10 includes an information generating device 100. The information generating device 100 has a so-called generating function, and is provided with a function for outputting necessary information at appropriate timing.

[0111] As described above, the information generating device 100 includes a situation information acquiring unit 101, an output purpose determining unit 102, an output period determining unit 103, a generation method determining unit 104, an information generating unit 105, and an output unit 106. The functions and actions of these units have already been described, so a description thereof will be omitted here.

[0112] Here, information from various vehicle sensors 107A mounted on the vehicle and road sensors 107A arranged on the road is input to the situation information acquisition unit 101. Furthermore, various information is also input from the cloud server 107B, such as information on the driver's driving history and driving tendencies, information on roads frequently traveled by the driver, and information disclosed by the government, etc.

[0113] In addition, the information generation unit 105 is connected to the generative model server 109 via a communication network (wired or wireless), and can transmit the generated information to the generative model server 109, and conversely, receive necessary data from the generative model server 109.

[0114] Furthermore, the output information from the output unit 106 is sent to an output device 108 such as a display or speaker of the vehicle after the output timing is controlled by an output information control unit 111, and is converted into information such as an image or sound and notified. The output information from the output unit 106 is also sent to a post-stage generation device 110, where it can be used for another model calculation. The output information is information generated in accordance with an output purpose.

[0115] 9A to 9C, for example, may be included. The output information control unit 111 controls the output timing by selecting information to be presented or terminated on the output device 108 in accordance with the information output period (tp) determined by the output period determination unit 103, and also has a function of determining whether the output purpose has been accomplished.

[0116] Next, a control flow relating to the generation of information executed by the information generating device 100 will be described with reference to Fig. 12. Note that this description is based on the above-mentioned description of the information generating device 100, and is therefore a rough description.

[0117] In step S10, the information generating device 100 starts an application that generates information under the driving state of the vehicle. Furthermore, this application repeatedly executes steps S11 to S26 until the application is terminated in step S27.

[0118] In step S11, the situation information acquisition unit 101 acquires information about the current situation inside and outside the vehicle from vehicle sensors and road sensors. Next, in step S12, the output purpose determination unit 102 calculates the deviation (Da) between the ideal driving and the current actual driving from the situation information. Next, in step S13, the output purpose determination unit 102 determines the degree of deviation (Da) of the calculated deviation (Da). If it is determined that the deviation (Da) is smaller than a predetermined determination threshold (Th) (No determination), the process proceeds to step S21.

[0119] On the other hand, if it is determined that the deviation amount (Da) is greater than the predetermined threshold value (Th) (Yes determination), the process proceeds to step S14. In step S14, the output purpose determination unit 102 determines the output purpose (setting of the urgency (Ur)) based on the magnitude of the deviation amount (Da), for example, in three stages. This corresponds to the processing in Figs. 3 and 4. Once the output purpose is determined, in step S15, the output purpose determination unit 102 determines whether another output purpose is defined. This determines whether the output purposes conflict. If it is determined that there is no conflict (No determination), the process proceeds to step S16. In step S16, the information output period corresponding to the current output purpose is determined, and the process proceeds to step S19. On the other hand, if it is determined that another output purpose is defined (Yes determination), the process proceeds to step S17.

[0120] In step S17, the output purpose determination unit 102 determines whether the current output purpose (urgency (Ur)) has a higher priority than the previously set existing output purpose (urgency (Ur)). If it is determined that the existing output purpose (urgency (Ur)) has a higher priority (No determination), the process proceeds to step S16. Step S16 is as described above.

[0121] On the other hand, if it is determined that the current output purpose (urgency (Ur)) has a higher priority (Yes determination), the process proceeds to step S18. In step S18, the output period determination unit 103 determines the current information output period (tp) and changes the existing information output period (tp). This corresponds to the processing in, for example, Figures 7 and 8A.

[0122] Here, as mentioned above, the information output period (tp) includes an information output start period (ts) during which information output begins, an information comprehension period (tu), an operation execution period (to), and an information non-output period (tn) during which no information is output. Once the information output period (tp) is determined in steps S16 and S18, the process proceeds to step S19.

[0123] In step S19, the generation method determination unit 104 determines an information generation method corresponding to the information output period (tp). In addition, in step S20, the information generation unit 105 generates output information using the determined information generation method. This output information is sent to the output device 108 and is announced as information from a display or speaker. This corresponds to the processing in Figures 9 and 10.

[0124] When the output information is generated in step S20, the process proceeds to step S21. In step S21, the output information control unit 111 determines whether the information output period (tp) has ended. If it is determined that the information output period (tp) has not ended (No determination), the process proceeds to step S25. On the other hand, if it is determined that the information output period (tp) has ended (Yes determination), the process proceeds to step S22.

[0125] In step S22, the output of information is stopped, and the process proceeds to step S23. In step S23, the output information control unit 111 determines whether the output purpose has been completed, and if it determines that it has not been completed (No determination), the process proceeds to step S24. In step S24, the output purpose determination unit 102 sets an output purpose for review and performs the review as described above. On the other hand, if it determines that the output purpose has been completed (Yes determination), the process proceeds to step S25.

[0126] In step S25, the output information control unit 111 determines whether it is the information output start period (ts) of existing information. This is because it has been determined in step S18 that the priority is low. If it is determined in this step that it is not the information output start period (ts) of existing information (No determination), the process proceeds to before step S27. On the other hand, if it is determined that it is the information output start period (ts) of existing information (Yes determination), the process proceeds to step S26.

[0127] In step S26, the information generation unit 105 generates output information using the determined information generation method. This output information is sent to the output device 108 via the output information control unit 111, and is notified as information from a display or speaker.

[0128] While the vehicle is traveling, the above steps S11 to S26 are repeatedly executed, and when the vehicle is stopped, the process proceeds to step S27 and this application is terminated.

[0129] As described above, this embodiment includes an output purpose determination unit that determines an output purpose for outputting information to the driver using operation information of at least one of the driver and the vehicle, an output period determination unit that determines an output period for the information using the output purpose determined by the output purpose determination unit, a generation method determination unit that determines a generation method for generating information according to the output period, an information generation unit that generates information corresponding to the output purpose based on the generation method determined by the generation method determination unit, and an output unit that outputs the generated information according to the output period, and the output period is set to a period during which the driver can recognize the information output from the output unit.

[0130] This provides an advantage that an information generating device that generates necessary information in real time can output necessary information at the required timing without delay. Note that the necessary information is an amount of information that fits within the information comprehension period. [Example]

[0131] In the above-described first embodiment, the generation method determination unit 104 selects the generation method with the highest priority among the generation methods that can output information within the information output start period (ts). In contrast, in the second embodiment, the generation method determination unit 104 is characterized in that it determines the generation method and generated data to be output that can be in time for the information output start period (ts) of each generation method. Only the differences from the first embodiment will be described below.

[0132] 13 shows the overall configuration of the information generating device according to the embodiment 2. Similar to the situation information acquiring unit 101 according to the embodiment 1, the situation information acquiring unit 120 acquires information on the operation of the driver and the vehicle, which is a motivation for generating information, and information on the surrounding situation from sensors, maps, input data from the driver, etc.

[0133] Similar to the output purpose determination unit 102 in the first embodiment, the output purpose determination unit 121 calculates ideal driving from the information acquired by the situation information acquisition unit 120, and determines the output purpose for generating information based on the deviation between this ideal driving and the driver's actual driving.

[0134] The output period determination unit 122, similar to the output period determination unit 103 in the first embodiment, sets the information output period (tp) for outputting information for each generation method according to the output purpose determined by the output purpose determination unit 121, including information regarding urgency (Ur).

[0135] The information generation unit 123 starts generating information by each generation method using information from the output purpose determination unit 121, in parallel with the output period determination unit 122. The generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer, (2) a second generation method that uses motion information, (3) a third generation method that uses output generated in a similar situation in the past, (4) a fourth generation method that generates information by modifying part of the output generated in a similar situation in the past, (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements, and (6) a sixth generation method that inputs motion information into a generative model to generate all the information to be generated.

[0136] The generation method determination unit 124 has an output priority for each of the generation methods described above, and adopts the generation method with the highest priority (largest amount of information) that can be generated within the information output start period (ts).

[0137] The output unit 125 outputs the information generated by the generation method adopted by the generation method determination unit 124. With this configuration, it is possible to start generating information early, and therefore it is possible to adopt a generation method with higher priority.

[0138] Since an increase in the computational load on the server side and the edge device side is expected, the information generating device of Example 2 may be used in combination with the information generating device of Example 1 when the computational load is tolerable. [Example]

[0139] Next, another embodiment of the present invention will be described. Below, we will use workers in the manufacturing, quality assurance, and maintenance processes as an example to explain how this embodiment operates when a danger or procedural error occurs during manufacturing or inspection work, and when a warning or correction method is presented. Below, we will mainly explain the differences from the embodiments described so far.

[0140] In this embodiment, the driver described in the first embodiment is basically replaced with a "worker" and the vehicle is replaced with a "work object." In the configuration of the information generating device shown in Fig. 1, this embodiment operates as follows.

[0141] The situation information acquisition unit 101 acquires information on the operation of the worker and the work object (such as equipment used for manufacturing or inspection, equipment to be manufactured or inspected, equipment for recording manufacturing or inspection, etc.) that motivates the generation of information, as well as information on the surrounding situation, from input data from sensors and the worker.

[0142] As shown in Figure 14, sensors include sensors 301 built into or attached to the work object, sensors 302 attached to the work environment, wearable sensors 303 worn by the worker, and information on work plans and workers on a server obtained via the Internet, etc., and information 304 disclosed by the government, etc.

[0143] Specific information acquired includes, for example, information about the operation of the work object (direction, amount, and speed of movement of the moving parts, flow velocity of the fluid passage parts, current and voltage, etc.), information about the condition of the work object (temperature, presence or absence of malfunctions, years of operation, etc.), information about the work content and work method of the worker (type of work, work procedure, posture, etc.), information about the worker's work experience, usual work tendencies, physical condition, presence or absence of impatience, etc. (presence or absence of delays from the work plan, pulse rate, etc.), information about the work environment (illumination, temperature, humidity, noise, etc.), etc.

[0144] The output purpose determination unit 102 calculates an "ideal task" from the information acquired by the situation information acquisition unit 101, and determines the output purpose for generating information based on the deviation between this ideal task and the worker's actual task. The ideal task is defined based on the work procedure, and the current ideal task is calculated in light of the current task progress and the work procedure.

[0145] Furthermore, because there are some individual differences in work, ideal work can be defined with a range. If there are large differences in work methods between individuals or teams, work procedures can be adapted to each individual or team to absorb the differences.

[0146] The output purposes for generating information include, for example, emergency notifications about dangerous situations, warnings about deviations in work (manufacturing, inspection, etc.) methods (no danger occurs, but there is a problem with the work results), and suggestions for improving efficiency (the results are correct, but the methods have quirks, or there is a better way).

[0147] If a problem occurs during a manufacturing line process or while working in separate teams, it may affect the work of other workers depending on the extent of the dangerous incident or rework. For this reason, it may be possible to set up output purposes to notify other workers on the line and supervisors (notifying the current work progress of other workers and changes to the work plan).

[0148] The classification of the purpose can be set in advance, for example, according to the type and degree of deviation between the ideal work and the actual work for each worker. The output purpose also includes information about urgency, as in the first embodiment.

[0149] The output period determination unit 103 sets an information output period for outputting information for each generation method in accordance with the output purpose including information on urgency determined by the output purpose determination unit 102 .

[0150] The generation method determination unit 104 determines a generation method for generating information based on the output purpose and urgency determined by the output purpose determination unit 102, the information output period determined by the output period determination unit 103, and the estimated time required for each generation method.

[0151] The information generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer; (2) a second generation method that uses motion information; (3) a third generation method that uses output generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the output generated in a similar situation in the past; (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements; and (6) a sixth generation method that inputs motion information into a generative model to generate all the information that needs to be generated.

[0152] Furthermore, the generation methods are prioritized, and the generation method determination unit 104 selects the generation method with the highest priority from among the generation methods that can output within the information output start period. Note that, when the time available for generation is short, in a method that uses past output as is or with partial modifications, or in a method that uses a generative model for generation, the length of the sentence to be generated may be specified to be short, that is, a part of the past output may be output, or the number of words in the sentence to be generated may be set to be small.

[0153] In addition, the expected time required for each generation method may be set by measuring the average or maximum value in advance for each generation method, or, since communication speed and processing speed with the server change as needed, dummy information (previous information) may be generated as needed when the possibility of an output generation request is low (for example, between tasks) and used to make an estimate.

[0154] The information generation unit 105 generates information using the generation method determined by the generation method determination unit 104 (it may also use information from the situation information acquisition unit, etc.). The output unit 106 outputs the generated information to the worker. The output method may be a method using a display or speaker provided on the work object, or a method using a display or speaker on a smartphone, work tablet, or the like held by the worker.

[0155] Next, a control flow relating to the generation of information executed by the information generating device 100 according to this embodiment will be described with reference to FIG.

[0156] Note that the explanation here is substantially the same as that of the information generating device 100 in the first embodiment described above, and therefore the step reference numbers have an "A" suffix. Therefore, explanations of the same steps will be omitted. Note that while the first embodiment describes information targeted at vehicle driving operations, the present embodiment describes information targeted at work performed by a work machine.

[0157] 12 for the first embodiment, steps S28, S29, and S30 have been added, with the aim of providing information to individuals and teams.

[0158] In step S28, the output purpose determination unit 102 performs individual adaptation of the ideal work. When there are large differences in work methods between individuals or teams, the work procedure manual is adapted for each individual or team so that the differences can be absorbed. This allows the ideal work to be set for each individual or team.

[0159] Furthermore, steps S29 and S30 for executing a repeat loop are inserted after steps S14A and S26A, respectively. As a result, in cases where the information is output to other destinations in addition to the worker himself (workers on the same line, workers in the same factory, supervisors of the line or factory, workers on a team working together, etc.), steps S15A to S26A are executed for each destination. [Example]

[0160] Next, another embodiment of the present invention will be described. The example described below is an example in which, when a certain work time is set, the number of times or length of information output does not take up too much of the work time.

[0161] For example, we will explain how this embodiment works when tracking the progress of a student in an educational setting and providing some kind of notification to the student or educator, but below we will focus on the differences from the embodiments described so far.

[0162] In this embodiment, the driver described in the first embodiment is basically replaced with a "student" and the vehicle is replaced with an "education-related device." In the configuration of the information generating device 100 shown in Fig. 1, this embodiment operates as follows.

[0163] The situation information acquisition unit 101 acquires information on the operation of the student and education-related devices (equipment operated by the student or equipment operating for education, etc.), which motivates the output of information, as well as information on the surrounding situation, from sensors, input data from the student, etc.

[0164] As shown in the configuration in Figure 16, the sensors include sensors 401 built into or attached to education-related devices (in Figure 16, a bat operated by the student or a pitching machine that throws balls for the student's batting practice), sensors 402 attached to the educational environment, wearable sensors 403 worn by the student, and information on educational plans and progress on a server obtained via the Internet, etc., and information 404 disclosed by the government, etc.

[0165] Specific information acquired includes, for example, information regarding the operation of education-related devices, information regarding the status of education-related devices, information regarding the education plan (schedule, unit information, progress, remaining education time, etc.), information regarding the behavior of the student, information regarding the student's education experience, usual behavioral tendencies, physical condition, whether or not they are anxious, etc. (information on time limits, exams, pulse rate, etc.), and information regarding the educational environment (weather, illuminance, temperature, humidity, noise, etc.).

[0166] The output purpose determination unit 102 calculates an ideal movement from the information acquired by the situation information acquisition unit 101, and determines an output purpose based on the deviation between this ideal movement and the actual movement of the trainee.

[0167] Here, the ideal motion is defined based on the educational objectives, the progress of the trainee, the level of proficiency, etc. For example, if the trainee is a beginner at baseball, the ideal motion is first defined as being able to hold the bat in the correct way, and if the trainee has some baseball experience, the ideal motion is defined as being able to swing the bat without wavering. Such ideal motions may be defined in advance, or may be defined statistically by collecting information on many trainees.

[0168] Examples of output purposes include emergency notifications regarding dangerous behavior, warnings about deviations in behavior (which do not pose a danger but are not good behaviors), instructions or suggestions for the next behavior (next unit, break, etc.), and notifications to educators (for example, current progress, the status of the student, future plans, and notifications of dangerous events).

[0169] The classification of output purposes is set in advance, for example, according to the type and degree of discrepancy between the ideal movement and the trainee's actual movement. The output purpose also includes information on the urgency of the output. Urgency can be, for example, most importantly warning about dangerous movements, followed by correcting the trainee's bad habits, and these can be set in advance for each item.

[0170] The output period determination unit 103 sets an information output period for outputting information for each generation method in accordance with the output purpose, including information regarding urgency, determined by the output purpose determination unit 102. Note that in output for educational purposes, output before an event is not necessarily desirable for all output purposes, since the output is intended to develop the trainee's own thinking ability and trial and error ability, and depending on the output purpose, it may also be effective to set an information output period after the event.

[0171] For example, if the trainee is standing too close to the base and there is a risk of the ball hitting them, the system will output information urging them to correct their position in advance.On the other hand, if the trainee is standing too far from the base and it is predicted that their form will be disrupted when they hit the ball, the system can output advice after the trainee has tried batting from that position at least once.

[0172] Alternatively, multiple output purposes and information output periods (for example, periods for outputting brief information before an operation, detailed explanatory information immediately after an operation, review information during a review period, etc.) may be set for one event, such as briefly explaining the "problem operation → improvement method" before the operation and then providing a detailed explanation after the operation is completed.

[0173] Furthermore, the output period determination unit 103 may predict that executing all outputs may cause a delay in the progress of the education unit or that the planned education content may not be able to be executed within the remaining time, taking into account the progress of the education unit, the remaining time of the day, etc. In this case, the information output period for information with low urgency may be set after the end of the education time for that day.

[0174] The generation method determination unit 104 determines a method for generating information based on the output purpose and urgency determined by the output purpose determination unit 102, the information output period determined by the output period determination unit 103, and the estimated time required for each generation method.

[0175] The information generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer; (2) a second generation method that uses motion information; (3) a third generation method that uses output generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the output generated in a similar situation in the past; (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements; and (6) a sixth generation method that inputs motion information into a generative model to generate all the information that needs to be generated.

[0176] Here, the generation methods are prioritized, and the generation method determination unit 104 selects the method with the highest priority as the generation method from among the generation methods that can be output within the information output start period.

[0177] If the time available for generating information is short, a generation method can be adopted that uses past output as is or with some modifications.Furthermore, in a generation method that uses a generative model, the length of the generated sentence can be specified to be short, that is, a part of the past information can be output, or the number of words in the generated sentence can be set to be small.

[0178] The expected time required for each generation method may be set by measuring the average or maximum value in advance for each generation method, or, since communication speed and processing speed with the server change as appropriate, dummy information may be generated as appropriate when the possibility of a request for information generation is low (for example, during breaks or the time between certain actions).

[0179] The information generating unit 105 generates information to be output using the generation method determined by the generation method determining unit 104 (it may also use information from the situation information acquiring unit, etc.). The output unit 106 outputs the generated information to the student. The output method may be a method using a display or speaker provided in the educational environment or education-related equipment, or a method using a smartphone or earphones owned by the student.

[0180] Next, a control flow relating to the generation of information executed by the information generating device 100 according to this embodiment will be described with reference to FIG.

[0181] Since the explanation here is substantially the same as that of the information generating device 100 in the first embodiment, the step reference numbers are suffixed with "B." Therefore, explanations of the same steps will be omitted. Note that while the first embodiment explains information related to vehicle driving operations, the present embodiment explains information related to the trainee's movements.

[0182] 12 for the first embodiment, steps S40 to S48 are added. These steps are intended to provide information to the trainee.

[0183] A difference from the flowchart in the first embodiment is that in step S40, the output purpose determination unit 102 sets a threshold value of urgency (Sh) based on the remaining time of the training. The threshold value of urgency corresponds to, for example, the progress level relative to the remaining time of the training, and the lower the progress level relative to the remaining time, the higher the threshold value of urgency.

[0184] In step S41, it is determined whether the urgency exceeds a threshold value. In step S41, the output purpose determination unit 102 determines the degree of urgency of the current output purpose relative to the urgency threshold value (Sh) set in step S40.

[0185] If it is determined in step S41 that the urgency is equal to or greater than the threshold, the steps from step S42 onward are executed to set the information output period, which is set based on the output purpose in step S14B.

[0186] On the other hand, if it is determined that the urgency is below the threshold (Sh), the process proceeds to step S43, where the output purpose determination unit 102 changes the output purpose to "review" and sets the information output period to "review period." The review period is a preset period just before or after the end of the education session, or a period when the trainee inputs a command to check the review output.

[0187] Here, steps S42 to S44 are executed in a repeated loop for each output destination (for example, for each trainee).

[0188] In steps S45 to S47, the output information control unit 111 outputs the review information generated in steps S19B and S20B while determining that the period is a review period. The urgency threshold (Sh) may be set to be higher in inverse proportion to the remaining time of the education, or may be set in stages according to the remaining time of the education.

[0189] In the example of Figure 17, review information is generated each time and stored on a server or edge device, and is displayed during the review period. However, if there is too much review information, it may not be possible to review it all during the review period. Therefore, the review information may be displayed in order of urgency. Also, the review information may be re-entered into the generative model, and the main points may be summarized and the amount of information reduced before being notified. [Example]

[0190] Next, another embodiment of the present invention will be described. Below, we will explain how this embodiment operates when another model is the output target and information (in this case, output information) is output to this model. Below, we will mainly explain the points that differ from the embodiments described so far. Case 1 In Case 1, we will explain the behavior of a generative model (Image to text, video to text, data to text, etc.) that allows a subsequent generation device (here, Text-to-Text Transfer Transformer) 110 to understand a scene.

[0191] The following example explains how to generate information for a generative model that generates output information for the driver from images from external cameras and vehicle sensor information mounted on the vehicle. In this example, we assume that the system can grasp the external world using point cloud information from LiDAR, etc., in addition to the images input to the generative model.

[0192] In the output purpose determination unit 102 in Case 1, since the scene has not yet been understood, it is difficult to determine a high-level output purpose as explained in Examples 1 to 4. Therefore, the output purpose and urgency are determined for each object detected by the sensor using a rule base or the like.

[0193] For example, based on information on the positions and relative speeds of features and other vehicles, the system determines the level of urgency by obtaining information on features close to the vehicle, information on features in the direction the vehicle is traveling, information on deviations in driving operations, etc., and sets a higher level of urgency for items with a higher level of danger.

[0194] In case 1, the output period determination unit 103 predicts a dangerous event and determines a data output period for each detected object on a rule-based basis. For example, it calculates the future positions of the host vehicle and the oncoming vehicle from the trajectories and speeds of the host vehicle and the oncoming vehicle, calculates the predicted collision time when a dangerous event, such as a collision, will occur, and determines the data output period and data output start period to be a preset time before the predicted collision time.

[0195] In case 1, the generation method determination unit 104 determines the amount of information to be output to the subsequent generation device 110 according to the length of time until the earliest output start period. If the length of time until the output start period is sufficiently long, all information for the object is generated and output, and if the length of time is short, only information with high urgency is selected, generated, and output.

[0196] In Case 1, the output destination, unlike in the first to fourth embodiments described above, is not a display or a speaker, but the subsequent generation device 110. Therefore, the output purpose determination unit 102 functions as a recognition unit.

[0197] Furthermore, if a sensor such as LiDAR is not installed and the output purpose and urgency cannot be grasped on a rule-based basis when determining the output purpose, a request for the generation speed (corresponding to the time length of the data output period) may be obtained from the subsequent generation device 110 or the generation device control system 10, and if the requested generation speed is short, only output information related to a predetermined object may be generated.

[0198] In other words, if the response of the server, which is the subsequent generation device 110, is slow and the required generation speed is short, only text related to objects with high urgency will be generated based on a list that matches the object types and urgency that is stored in advance.

[0199] As a result, if no object is specified, text is generated that includes the driving environment, etc. However, if an object is specified, only text related to surrounding vehicles is generated, which shortens the time required to complete generation compared to generating text that includes all information.

[0200] It can also be applied to a model that generates text that understands a scene in order to generate robot behavior (text to X). The following explains the case of a robot.

[0201] In this case, the output purpose determination unit 102 determines the urgency based on information necessary for avoiding danger (position and speed information of surrounding objects, including the ground) and information necessary for generating behavior (facial expressions and speech of nearby people, and the state of the object of the behavior), and sets a higher urgency for higher danger.

[0202] Furthermore, the output period determination unit 103 cooperates with the subsequent generation device 110 to set the timing at which the next information is required based on the robot's action plan.

[0203] Furthermore, the generation method determination unit 104 generates all output information if the time length until the earliest dangerous event is long, and generates only output information with high urgency if the time length is short.

[0204] From this perspective, the following flowchart is executed: Figure 18 shows a flowchart outlining this embodiment.

[0205] Note that the explanation here is similar to that of the information generating device 100 of the first embodiment described above, and therefore similar steps are designated by the suffixes "C" and "D" to the reference numbers of the steps. Therefore, explanations of the same steps may be omitted.

[0206] Furthermore, in this embodiment, the target is not viewable information presented to drivers, workers, etc., but data forming information.

[0207] In step S50, the output purpose determination unit 102 determines whether an output purpose can be determined for each object. A situation in which an output purpose can be determined for each object refers to a situation in which a sensor such as LiDAR is available and information on the characteristics (position, speed, etc.) of each object can be obtained, or a situation in which objects to be noted are defined in advance in an application such as a test witness application.

[0208] If it is determined that the output purpose can be determined for each object (Yes determination), the following steps are executed for each object in step S51.

[0209] In step S14C, the output purpose determination unit 102 determines the output purpose and urgency for each object. The output purpose determined in this step includes the output purpose completion condition. Next, in step S15C, if it is determined that an output purpose other than the object has been defined (Yes determination), the process proceeds to step S17C.

[0210] In step S17C, the output purpose determination unit 102 compares the urgency of the current output purpose with the urgency of the existing output purpose, and if it determines that the current urgency is higher (Yes determination), in step S18C, the output period determination unit 103 determines the current data output period and changes the existing data output period.

[0211] On the other hand, if it is determined in step S15C that there are no settings other than the current output purpose (No determination), or if it is determined in step S16C that the current urgency is equal to or less than the existing urgency (No determination), the current data output period is set in step S17C.If it is determined in step S17C that the current urgency is greater than the existing urgency (Yes determination), the current data output period is set and the existing output period is changed in step S18C.

[0212] In step S19C, the generation method determination unit 104 determines a generation method for the current output purpose in accordance with the data output period. Then, in step S20C, the information generation unit 105 generates output information using the determined generation method. Next, in step S21C, if the output information control unit 111 determines that any data output period related to the object has ended (Yes determination), in step S22C, the output information is stopped.

[0213] Next, in step S23C, the output information control unit 111 determines whether the output purpose of the stopped output information has been completed, and if it determines that it has not been completed (No determination), it proceeds to step S24C to set a re-output purpose for this output information, and returns to step S15C to execute again.

[0214] In addition, the situation in which the output purpose is completed is, for example, a situation in which all output information is passed to the subsequent generation device 110, or there is no further generation request from the subsequent generation device 110, and is the condition determined when determining the output purpose in step S14C.

[0215] In the first to fourth embodiments, the output target for outputting information was a person such as a driver, worker, or trainee, and therefore the completion of the output purpose could be determined, for example, by whether the output was continued for a predetermined period of time or more, whether the time set for the person to understand the output content and start the next action had passed, whether the person had taken an action that indicated that the output content had been understood, etc. In contrast, in this embodiment, the output target is the subsequent-stage generation device 110.

[0216] In step S23C, if the output information control unit 111 determines that the output purpose of the stopped output information has been accomplished (Yes determination), it proceeds to step S25C and determines whether any output start period related to the object in question has begun.

[0217] If it is determined that the output start period for any of the objects has begun (Yes), the process proceeds to step S26C and starts outputting the output information. Once the output information has been output in step S26C, the process proceeds to step S52, where processing for each object ends.

[0218] On the other hand, returning to step S50, if the output purpose determination unit 102 determines that it is not possible to determine an output purpose for each object (No determination), the process proceeds to step S53, where a generation speed request is obtained from the supervisory system of the subsequent generation device 110. Based on this information, the output purpose determination unit 102 determines an output purpose in step S14D, and the output period determination unit 103 determines a data output period in step S15D. Steps S15D and after operate in the same way as steps S15C and after.

[0219] If the output information control unit 111 determines in step S23D that the output purpose has not been accomplished, the process returns to step S53, a generation speed request is made, and step S14C and subsequent steps are repeatedly executed. <Case 2> In Case 2, we will explain an example of an application that uses multiple generative models to draw a 3D space from sensor data (e.g., images).

[0220] When constructing a 3D model from an image, it is necessary to estimate multiple pieces of information, such as the three-dimensional shape, parts not visible in the image, and color when the light hits it differently. In addition, when drawing, it is necessary to calculate how it will look based on the viewpoint and light source settings, and if the model is designed to allow people to interact with it, it is necessary to calculate and draw changes in the shape and appearance of the 3D model in response to people's movements.

[0221] For this reason, it is possible to combine generative models specialized for each function, or to combine multiple models specialized for each function within a generative model to render 3D space.

[0222] In particular, in applications that are expected to render real space in high real-time, it is important to reduce the time required for generation for each generative model or generative function.

[0223] In the following, digital twins created with high real-time capabilities will be referred to as real-time twins for convenience. Possible applications include real-time twins for test observations and real-time twins for tourism. The rendering speed is determined by factors such as the speed of movement within the metaverse space.

[0224] The output purpose determination unit in Case 2 is set according to the purpose of generating a real-time twin. For example, if the purpose is to witness a test, the requirement for test equipment is the highest, and the requirement for generating surrounding people, buildings, etc. is low.

[0225] In addition, the output period determination unit in Case 2 determines the generation period in which each generative model or generative function can be used by calculating backward from the overall output period. The ratio of the generation periods of each generative model or generative function may be determined from the ratio of the average values ​​of each generative model or generative function, or the weight of each generative model or generative function may be changed depending on the object to be drawn.

[0226] For example, when witnessing a car crash test, three-dimensional information such as the deformation of the car at the time of collision is more important than the color, etc. For this reason, the output period of the generative model or generative function that estimates the three-dimensional shape is made longer, and the output period of the generative model or generative function that estimates the color is made shorter. The weighting may be set in advance on a rule-based basis according to the purpose of generating the real-time twin, or may be determined by receiving a request from the user.

[0227] In addition, the generation method determination unit in Case 2 can select from (1) a detailed generation method, (2) a generation method with reduced resolution, or (3) a generation method that reuses past information for each object depending on the output purpose, urgency, and output period. For example, similar information such as traffic lights can use past data.

[0228] In Case 2, due to the nature of real-time spatial rendering, when a new output purpose is set for an object for which an output purpose has already been defined, it is highly likely that the existing output purpose will become unnecessary. In such a case, "changing the existing output period" in steps S18C and S18D is synonymous with "discarding the existing output period."

[0229] The present invention is not limited to the above-described embodiments, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment.

[0230] Furthermore, the above-described configurations, functions, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD. [Explanation of symbols]

[0231] 101... situation information acquisition unit, 102... output purpose determination unit, 103... output period determination unit, 104... generation method determination unit, 105... information generation unit, 106... output unit.

Claims

1. An information generation device that generates information to be output to a worker, an output purpose determination unit that determines an output purpose, which is a purpose for outputting the information to the worker, using motion information of at least one of the worker and the work object; an output period determination unit that determines an output period of the information using the output purpose determined by the output purpose determination unit; a generation method determination unit that determines a generation method for generating the information in accordance with the output period; an information generation unit that generates the information corresponding to the output purpose based on the generation method determined by the generation method determination unit; an output unit that outputs the information generated by the information generation unit during the output period; The output period determination unit sets an adjustable period required for the worker to recognize the information as the output period. An information generating device characterized by:

2. 2. The information generating device according to claim 1, The information generating device is characterized in that the information includes one or more of characters, videos, images, sounds, lights, vibrations, reaction forces, shapes, and temperatures.

3. 2. The information generating device according to claim 1, The output purpose corresponds to the information required by the worker or the information that is determined to need to be output to the worker, which is defined based on the operation information, and is an information generating device characterized by including information on urgency.

4. 2. The information generating device according to claim 1, The output period is an information generation device characterized in that it is a period obtained by adding together at least an output start period indicating the period when the output of the information should begin, an information understanding period required for the worker to recognize the information, and an operation execution period required for the worker to operate the information.

5. 2. The information generating device according to claim 1, An information generating device characterized in that the output period is defined as a period obtained by adding together at least an output start period indicating the period when output of the information should begin, an information understanding period required for the worker to recognize the information, an operation execution period required for the worker to operate, and an information non-output period set following the end of the operation execution period during which the information is not output.

6. 6. The information generating device according to claim 5, The information generating device is characterized in that the output period determination unit sets a review output start period and a review output period after the information non-output period.

7. 6. The information generating device according to claim 5, An information generating device characterized in that the output period determination unit starts outputting the information during the output start period and ends outputting the information at the end of the operation execution period or the start of the information non-output period.

8. 5. The information generating device according to claim 4, The information generating device is characterized in that the output period determination unit adjusts the time length of the output period depending on the amount of information of the information.

9. 9. The information generating device according to claim 8, The information generating device is characterized in that the generation method determination unit selects a generation method for generating the information within a range of the time length of the output period and generates the information.

10. 2. The information generating device according to claim 1, An information generation device characterized in that, if there is no generation method that can output the information within the output period, the output period determination unit switches the output purpose to a review purpose and determines the output period and the generation method.

11. 9. The information generating device according to claim 8, The generation method determined by the generation method determination unit is A first generation technique that generates a simple notification. A second generation method using motion information; a third generation method using the information generated in a similar situation in the past; a fourth generation method for generating the information by modifying a part of the information generated in a similar situation in the past; A fifth generation method in which motion information is input into a generative model and generated using high-priority elements; and a sixth generation method in which motion information is input into a generation model to generate all information.

12. The information generating device according to claim 11, An information generation device characterized in that the third generation method or the fourth generation method determined by the generation method determination unit divides the information to be generated into elements and generates it using information with a high priority.

13. The information generating device according to claim 11, The information generation device is characterized in that the generation method determination unit has the function of dividing the sentence generated by the fifth generation method into specific elements, assigning a search label to each element, and storing the elements in a database.

14. 14. The information generating device according to claim 13, The information generation device is characterized in that the third generation method or the fourth generation method determined by the generation method determination unit generates the information by searching the elements stored in the database with reference to the search label.

15. The information generating device according to claim 11, An information generation device characterized in that, when the time from the current time to the end time of the output start period is short, the generation method determination unit determines that the input information for generating information using the fifth generation method or the sixth generation method is the input information with higher importance.

16. The information generating device according to claim 11, An information generation device characterized in that the generation method determination unit specifies the length of the information generated using the fifth generation method or the sixth generation method to be short when the time from the current time to the end time of the output start period is short.

17. The information generating device according to claim 11, An information generation device characterized in that, when estimating the required time for the fifth generation method or the sixth generation method, the generation method determination unit generates the information using the fifth generation method or the sixth generation method at a time when there is no request to generate the information.

18. 2. The information generating device according to claim 1, When a new output purpose is determined, the information generating device outputs the information corresponding to the output purpose with a high degree of urgency with priority.

19. 2. The information generating device according to claim 1, An information generating device characterized in that the information that is currently or may be generated in the near future is generated in advance and stored on an edge device side.

20. An information generation method for generating information to be output to a worker, comprising: an output purpose determination step of determining an output purpose, which is a purpose for outputting the information to the worker, using motion information of at least one of the worker and the work object; an output period determination step of determining an output period of the information using the output purpose determined in the output purpose determination step; a generation method determination step of determining a generation method for generating the information in accordance with the output period; an information generating step of generating the information corresponding to the output purpose based on the generation method determined in the generation method determining step; an output step of outputting the information generated in the information generation step during the output period; In the output period determination step, an adjustable period required for the worker to recognize the information is set as the output period. An information generating method comprising:

21. A vehicle equipped with an information generating device that generates information to be output to a driver via a display, A vehicle, wherein the information generating device is the information generating device according to any one of claims 1 to 19.

22. A display equipped with the information generating device according to claim 1.

23. A speaker equipped with the information generating device according to claim 1.

Images