Information generation device, information generation method, vehicle, display, and speaker

Abstract

Provided are an information generation device, an information generation method, and a vehicle with which it is possible to output necessary information without delay.　The present invention is characterized by comprising: an output purpose determination unit 102 that uses operation information pertaining to a worker and / or a workpiece to determine an output purpose for outputting information to the worker; an output period determination unit 103 that determines an output period for the information by using the output purpose determined by the output purpose determination unit; a generation process determination unit 104 that determines a generation process for generating information in accordance with the output period; an information generation unit 105 that generates information corresponding to the output purpose on the basis of the generation process determined by the generation process determination unit; and an output unit 106 that outputs information generated so as to correspond to the output period, wherein the output period is set to a period during which the worker can recognize the information output from the output unit.

Description

Information generating device, information generating method, vehicle, display, and speaker

[0001] The present invention relates to an information generating device that generates output information for a worker, an information generating method, a vehicle, a display, and a speaker.

[0002] Patent Document 1 describes a technique for displaying information at an appropriate time.

[0003] Patent Document 1 states that "the driving plan information distribution system includes a driving plan information distribution device having a position information / road shape information ahead acquisition unit that acquires vehicle position information and road shape information ahead, a sensor information acquisition unit that acquires output information from a group of on-board sensors, a driving plan information creation unit that creates driving plan information based on the position information / road shape information ahead acquisition unit and the output information of the sensor information acquisition unit, a driving plan information notification unit that supplies the output information of the driving plan information creation unit to a driving plan information display unit for display, a wireless communication device that performs vehicle-to-vehicle communication with surrounding vehicles, and a driving plan information distribution control unit that controls the operation of each of these units at a predetermined timing, and when there is a change in the driving plan information for a certain time ahead from the current time that is created by the driving plan information creation unit, the driving plan information distribution control unit outputs the changed driving plan information to the wireless communication device before the start time of the driving plan by a time that is sufficient for surrounding vehicles or the driver to respond, and notifies surrounding vehicles."

[0004] Republished Patent No. 2019 / 234915

[0005] The process of generating information requires a generation processing time for various calculations, etc. This generation processing time increases or decreases depending on many factors, such as the size of the input information and output information, the communication speed with the generation model, the processing speed on the server side, and the generation means. Patent Document 1 does not take into consideration the generation processing time until the information is generated or the generation means that generates the information.

[0006] An object of the present invention is to provide an information generating device, an information generating method, and a vehicle that output necessary information without delay.

[0007] The present invention is characterized by comprising an output purpose determination unit that determines an output purpose for outputting information to a worker using motion information of at least one of the worker and the work object, an output period determination unit that determines an output period for the information using the output purpose determined by the output purpose determination unit, a generation method determination unit that determines a generation method for generating information with content corresponding to the output period, an information generation unit that generates information corresponding to the output purpose based on the generation method determined by the generation method determination unit, and an output unit that outputs the generated information corresponding to the output period, wherein the output period is set to a period during which the worker can recognize the information output from the output unit.

[0008] According to the present invention, it is possible to output necessary information at the necessary timing without delay.

[0009] 1 is a block diagram showing the overall configuration of an information generating device according to an embodiment of the present invention. FIG. 1 is a configuration diagram showing an example of the configuration of a sensor used in the information generating device shown in FIG. 1. FIG. 2 is a flowchart for explaining the operation of the output purpose determining unit shown in FIG. 1. FIG. 3 is a flowchart for explaining specific operations for determining an output purpose in the flowchart shown in FIG. 3. FIG. 1 is an explanatory diagram for explaining the operation of the output period determining unit shown in FIG. 1. FIG. 5 is an explanatory diagram for explaining the relationship between the output period and the information generation method shown in FIG. 5. FIG. 6 is an explanatory diagram for explaining the relationship between each output period for events with different priorities. FIG. 7 is an explanatory diagram for explaining a first relationship between each output period for a plurality of events. FIG. 8 is an explanatory diagram for explaining a second relationship between each output period for a plurality of events. FIG. 9 is an explanatory diagram for explaining a first display example of information displayed on a display. FIG. 10 is an explanatory diagram for explaining a second display example of information displayed on a display. FIG. 11 is an explanatory diagram for explaining a data format in which an information type, a search label, and an information importance level are assigned to sentences constituting generated information. FIG. 12 is a configuration diagram showing the overall configuration of a vehicle control system incorporating the information generating device shown in FIG. 1. FIG. 13 is a flowchart for explaining the overall operation of the information generating device shown in FIG. 1. FIG. 14 is a configuration diagram showing a modified example of the information generating device shown in FIG. 1. Fig. 15 is a configuration diagram showing an example of the configuration of sensors used in an information generating device (work machine) according to another embodiment of the present invention. Fig. 16 is a flowchart illustrating the overall operation of the information generating device in the work machine shown in Fig. 14. Fig. 17 is a configuration diagram showing an example of the configuration of sensors used in an information generating device (training device) according to yet another embodiment of the present invention. Fig. 18 is a flowchart illustrating the overall operation of the information generating device in the training device shown in Fig. 16. Fig. 19 is a flowchart illustrating the overall operation of an information generating device used in a subsequent generation model according to yet another embodiment of the present invention.

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the present invention is not limited to the following embodiments, and various modifications and application examples within the technical concept of the present invention are also included within the scope of the present invention. Note that the embodiments described below are merely examples for explaining the present invention, and some details have been omitted or simplified as appropriate for clarity of explanation. The present invention can also be embodied in various other forms.

[0011] 1 is a block diagram showing the overall configuration of an information generating device according to a first embodiment. This embodiment takes the case of driving an automobile (an example of a task) as an example, and is directed to a device that generates information in a timely manner to provide guidance on the driving operation to be improved when an operation that should be improved is predicted in the driving operation of a driver (an example of a worker). The configuration and operation of the information generating device 100 according to this embodiment will be described below.

[0012] The situation information acquisition unit 101 acquires operation information of the driver and the automobile (an example of a work object, hereinafter referred to as the vehicle), as well as surrounding information regarding the surrounding situation, from various sensors, maps, and input information from the driver.

[0013] As shown in the example of Fig. 2, the various sensors include an on-board sensor 201 attached to the vehicle, a wearable sensor 202 worn by the driver, sensors 203 attached to infrastructure such as roads and surrounding facilities, and information on a server acquired via the Internet or information 204 disclosed by government agencies, etc. Information on a server includes, for example, the driver's driving history and driving tendencies, and information on roads frequently traveled by the driver. Note that this information may be stored in an edge device (on the vehicle) rather than in a server.

[0014] Specific information obtained from various sensors includes, for example, the vehicle's speed and acceleration, lateral position within the lane, distance to intersections and stop lines, vehicle condition (whether or not there is skidding, whether or not there is a malfunction, etc.), the driver's line of sight, the position and content of signs, road surface conditions, the position and relative speed of other vehicles, the position and relative speed of other traffic participants (pedestrians, motorcycles, etc.), weather, road type, congestion information, the driver's driving experience and usual driving tendencies, physical condition, whether or not there is impatience, etc. (whether or not there is a delay from the driving plan, pulse rate, etc.).

[0015] The output purpose determination unit 102 calculates "ideal driving" from the information acquired by the situation information acquisition unit 101, and determines "output purpose for generating information" based on the deviation between this ideal driving and the driver's actual driving.

[0016] "Ideal driving" refers to the conditions for ideal driving, defined by the vehicle's desired speed, acceleration, driving position, safety confirmation method, etc. Of course, other conditions can also be set.

[0017] In addition, the "output purpose of generating information" is the purpose of generating information required for foreseen events, such as "notification of the urgency of a dangerous situation," "suggestions for correcting usual driving habits," and "suggestions for driving plans."

[0018] The information used to define the ideal driving described above is calculated based on a "driving policy" set according to the purpose of use of the information generating device, "information on the traffic environment around the vehicle," and the like.

[0019] Here, the "driving policy" refers to, for example, the driving method that should be considered safe driving, the driving method that should be considered eco-driving, the driving method that should be considered commercial driving (for example, a driving method that is stricter than general safe driving and specifies upper limits on how to check for safety around the vehicle and on acceleration when starting and stopping), etc. Furthermore, ideal driving may be defined based on multiple driving policies.

[0020] Furthermore, "information about the traffic environment around the vehicle" refers to the positional relationship with other vehicles and pedestrians, road shape and type, speed limit, vehicle traffic division such as one-way streets, the obligation to stop near stop lines and pedestrian crossings, the positional relationship with traffic lights and the color of the traffic lights, etc. Note that, since there is a certain degree of individual difference in vehicle driving, the definition of ideal driving may be given a certain degree of flexibility.

[0021] The classification of such purposes is set in advance according to, for example, the type and degree of deviation between ideal driving and actual driving by the driver. A detailed method for determining the "output purpose for generating information" will be described later with reference to FIGS.

[0022] The output period determination unit 103 determines an output period for outputting information in accordance with the "output purpose for generating information" including information regarding urgency determined by the output purpose determination unit 102. This output period is determined for each information generation method, and the details of the method will be described later with reference to FIG.

[0023] The generation method determination unit 104 determines a generation method for generating information based on the output purpose determined by the output purpose determination unit 102, the urgency, the output period determined by the output period determination unit 103, and the expected generation processing time for each generation method.

[0024] The information generation methods include one or more of the following: (1) a first generation method that generates simple notifications such as buzzers; (2) a second generation method that generates short notification sentences based on rules from action information; (3) a third generation method that uses information generated in similar situations in the past; (4) a fourth generation method that generates information by modifying part of information generated in similar situations in the past (for example, changing "Checking behind the left is insufficient" to "Checking behind the right is insufficient"); (5) a fifth generation method that inputs action information into a generative model to generate only high-priority elements; and (6) a sixth generation method that inputs action information into a generative model to generate all the information that should be generated.

[0025] Here, the above-mentioned generation methods are prioritized, and the generation method determination unit 104 selects the method with the highest priority as the generation method among the generation methods that can output within the output period. If the expected generation processing time required to generate information is short, the amount of information to be generated may be specified to be small (for example, the length of the sentence (number of words, etc.) may be set short) in a method that outputs past information as is, or outputs information after partially modifying it, or in a method that generates information using a generative model. In other words, if the time from the current time to the end time of the output start period is short, the length of the information to be generated using the fifth or sixth generation method may be specified to be short.

[0026] The expected generation processing time for each generation method may be set by measuring the average or maximum value in advance for each generation method. Furthermore, since the communication speed and processing speed with the server vary depending on the vehicle's location and time of day, the expected generation processing time can be set in association with location information and time information. Furthermore, dummy information (previous information) may be generated as appropriate for situations where the possibility of an information generation request is low (e.g., when the vehicle is stopped) and used for estimation.

[0027] The information generation unit 105 generates information to be output using the generation method determined by the generation method determination unit 104. The information generated by the information generation unit 105 is sent to the output unit 106 via the output information control unit 111, and the output unit 106 outputs the generated information to the driver. The function of the output information control unit 111 will be described later with reference to FIG. 11.

[0028] The method of outputting information may be a method using a display (image), speaker (audio), steering wheel (vibration), pedal (reaction force), etc. installed in the vehicle, or a method using a display or speaker on a smartphone or various tablets held by the driver that are not installed in the vehicle.

[0029] The information generating device is, for example, a computer, and includes a processor such as a CPU (Central Processing Unit), and a storage device such as a memory and an HDD (Hard Disk Drive).

[0030] Next, a method for determining an output purpose performed by the output purpose determination unit 102 will be described. Fig. 3 shows an example of a flowchart for determining an output purpose. The method for determining an output purpose is set in advance based on, for example, the amount of deviation between ideal driving and the actual driving by the driver. Here, a plurality of predetermined thresholds Th for determining the amount of deviation (Da) between ideal driving and the actual driving by the driver are set, and for example, the relationship is "Th1>Th2>Th3."

[0031] In step S101, the output purpose determination unit 102 determines whether the deviation (Da) between the ideal driving and the actual driving of the driver is equal to or greater than a first determination threshold (Th1). If it is determined in step S101 that the deviation (Da) is equal to or greater than the first determination threshold, the process proceeds to step S102. In step S102, the output purpose determination unit 102 sets the output purpose to "emergency notification regarding a dangerous situation (risk level (dr): high)." Examples of indicators of the determination threshold include the speed and acceleration of the driven vehicle. Furthermore, examples of situations where the deviation is equal to or greater than the first determination threshold (Th1) include a situation where the driven vehicle is moving toward an obstacle without decelerating. In such a situation, there is a risk that the driven vehicle will collide with the obstacle.

[0032] In step S103, after determining that the deviation (Da) between the ideal driving and the driver's actual driving is less than the first judgment threshold (Th1), the output purpose determination unit 102 determines whether the deviation is equal to or greater than the second judgment threshold (Th2). If it is determined in step S103 that the deviation is equal to or greater than the second judgment threshold (Th2), the process proceeds to step S104. In step S104, the output purpose determination unit 102 sets the output purpose to "Suggestion regarding correction of usual driving habits (risk level (dr): medium)." An example of a situation where the deviation is equal to or greater than the second judgment threshold is when the driver is decelerating toward an obstacle, but the deceleration is insufficient in light of the driving policy that defines ideal driving. In such a situation, there is little risk of a collision, but greater consideration must be given to safety.

[0033] In step S105, after determining that the deviation is less than the first and second determination thresholds, the output purpose determination unit 102 determines whether the deviation between the ideal driving and the driver's driving plan is equal to or greater than a third determination threshold (Th3). If it is determined in step S105 that the deviation is equal to or greater than the third determination threshold (Th3), the process proceeds to step S106. In step S106, the output purpose determination unit 102 sets the output purpose to "suggestion regarding driving plan (risk level (dr): low)."

[0034] Examples of situations that exceed the third threshold include when a traffic jam is detected on the route set by the driver and it is predicted that taking an alternative route will result in a faster arrival at the destination, or when a dangerous road surface condition (flooding, freezing, etc.) is predicted or detected on the driver's usual route and it is considered desirable to change the route, cancel the trip, or issue a danger warning. When the deviation amount is equal to or exceeds the third threshold, it is desirable to suggest a change to the driving plan, although this is less urgent than when there is an immediate risk of a collision or the like.

[0035] Next, we will explain how to set the output objective and the corresponding urgency (Ur) based on the predicted time until a certain assumed situation (sometimes referred to as an assumed event) occurs, which is caused by a discrepancy between ideal driving and the driver's actual driving.

[0036] 4 shows how the urgency (Ur) is set according to the risk of an assumed event occurring due to deviation from ideal driving and the predicted time until the assumed event occurs. For example, the higher the risk (dr) and the shorter the event occurrence time (te), the higher the urgency (Ur) is set.

[0037] In Fig. 4, in step S201, the output purpose determination unit 102 calculates the risk (dr) of an anticipated event resulting from the deviation. This is based on the risk calculation flow in Fig. 3. Next, in step S202, the output purpose determination unit 102 predicts the event occurrence time (te) from the current time to the occurrence of a dangerous anticipated event resulting from the deviation from ideal driving. This predicts the event occurrence time (te) until a collision, for example, based on the position, traveling direction, and speed of the subject vehicle and the positions, traveling direction, and speed of other vehicles and pedestrians.

[0038] Once the risk level (dr) and the event occurrence time (te) are determined, the urgency level (Ur) is determined in step S203. The urgency level (Ur) is determined by a map determined from the risk level (dr) and the event occurrence time (te).

[0039] In Figure 4, as an example, the horizontal axis indicates the risk level (dr), which is set in four stages: (1) a situation with a high risk level, for example, "a situation where a collision is predicted"..."High"; (2) a situation with a moderately high risk level, for example, "a situation where proximity to others (vehicles, pedestrians, etc.) is predicted (the distance to others is less than a specified value)"..."Medium"; (3) a situation with a low risk level, for example, "a situation where there are no others nearby but deviation from the speed limit is predicted"..."Low"; and (4) a situation with no risk level, for example, "a situation where traffic congestion is detected on the route"..."None".

[0040] The vertical axis indicates the time to event occurrence (te), which is set in three stages: (1) te≦t1, (2) t1<te≦t2, and (3) t2<te. Here, t1 and t2 are predetermined judgment times, and the longer the time, the more time there is until the anticipated event occurs.

[0041] In this way, the urgency (Ur) is calculated from the combination of the risk (dr) and the event occurrence time (te). In this embodiment, the urgency levels are set from 1 to 6, and the higher the risk (dr) and the shorter the event occurrence time (te), the higher the urgency (Ur).

[0042] The value relating to the urgency (Ur) may be calculated using the following formula: In other words, the severity of the deviation results is weighted, and then multiplied by a value inversely proportional to the time until the occurrence. The value relating to the severity of the deviation results can be calculated by preparing values ​​corresponding to various dangerous events in a database in advance.

[0043] The urgency (Ur) can be calculated using the following formula (1): Ur = k (r / te) (1) where "r" is the severity of the deviation, "k" is a coefficient, and "te" is the time it takes for a dangerous event to occur.

[0044] Next, the concept of the output period (corresponding to time) for outputting information, which is determined by the output period determination unit 103, will be described with reference to FIG.

[0045] 5, the event occurrence time 501 (shown by a black triangle) at which the anticipated event is predicted is the reference time for various times and periods described below. As described above, this event occurrence time 501 is the reference time calculated based on the event occurrence time (te) from the current time to the occurrence of the dangerous event.

[0046] It is desirable that times and periods other than the event occurrence time 501 be set for each generation method determined by the generation method determination unit 104. The reason for this is that the amount of information output differs depending on the generation method, as will be described later with reference to FIG.

[0047] In FIG. 5, the output period determination unit 103 sets, based on the event occurrence time 501, an operation start time 502 at which the driver should first understand the content of the output information and then start the required driving operation.

[0048] This operation start time 502 can be obtained by calculating backward an operation execution period (to) during which the required driving operation can be performed from the event occurrence time 501. The operation execution period (to) may be a predetermined time length, or may be determined based on the content of the driving operation.

[0049] Next, an information comprehension period (tu) corresponding to the time required for the driver to understand the content of the output information is calculated, and the information output time 503 is set by counting backward from the operation start time 502. The information comprehension period (tu) may be set by preparing a database of estimated required times for anticipated events and driving deviations in advance. Furthermore, the information comprehension period (tu) may be set longer as the generation method or the amount of information (number of characters, length of audio, number of elements in images or videos, etc.) is predicted to be large.

[0050] Furthermore, if possible, information on the information processing ability (such as text reading speed) of each driver may be acquired, and the time may be shortened for drivers with better information processing ability. Information processing ability can be estimated, for example, from the time it takes for the driver to perform some operation (driving operation or pressing a button such as "OK" to end the output) in response to the presented output information.

[0051] Here, the information output period (tp) is defined as at least the period (tu+to) obtained by adding the information comprehension period (tu) and the operation execution period (to), or the period (tu+to+Δt) obtained by adding a short period (Δt) after the event occurrence time 501 to this added period (tu+to).

[0052] Next, an information output start period (ts) for starting the output of information is set, and an information output start time 504 is set by counting backward from the information output time 503. Here, the information output start period (ts) is set for the purpose of ensuring an information understanding period (tu) for the driver to understand the information with ample time to spare.

[0053] In other words, it is desirable to start outputting information immediately after the information output start time 504, which allows the information comprehension period (tu) to be set long. Therefore, it indicates that output should start no later than the end time of the information output start period (ts), in other words, by the information output time 503. In this way, by having time for the information output start period (ts), it is possible to provide the driver with ample time for understanding the information (tu).

[0054] The end time of the information output start period (ts) (which corresponds to the information output time 503) is the time calculated backward from the information understanding period (tu) required for the driver to understand the information, and the information output period (tp) for outputting information generated for an anticipated event is set from this information output time 503.

[0055] If the information output can be started during the output start period (ts), the information output continues throughout the information output period (tp). In this case, time is secured to generate the necessary information. On the other hand, if the information output cannot be started during the information output start period (ts) and is delayed into the information output period (tp), the information output using the generation method corresponding to the anticipated event at that time is set not to be started for the anticipated event.

[0056] The information output period (tp) may be set to end simultaneously with the event occurrence time 501 of the anticipated event, or may be set to end a little later (after Δt has elapsed) after the event occurrence time 501. The information non-output period (tn), which is set after Δt has elapsed from the event occurrence time 501, is set immediately after the end of the information output period (tp).

[0057] This is because the period immediately after the occurrence of an anticipated event is a period in which the driver should pay attention to the anticipated event and the surrounding situation. Also, this is a period in which the output of information for review, which will be described below, should not be performed, and during this period, no output is performed regarding the anticipated event, including the output of information for the purpose of review, which will be described below. Here, review refers to the action of reconfirming the information output in the anticipated event.

[0058] When generating an output of information for the purpose of looking back on an anticipated event, the output period determination unit 103 sets a look-back output start time 505 and a look-back output start period (tr), and a look-back output start time 506 and a look-back information output period (tb). The look-back information output period (tb) may be set immediately after the end of the information non-output period (tn), or may be set to wait until the driving operation has settled down (for example, until the next traffic light), or may be set after the driving operation has ended, such as when the driver has arrived at the destination.

[0059] The length of the review information output period (tb) is preferably set in the same manner as the information output period (tp), that is, depending on the amount of information and the driver's information processing ability. Furthermore, the method of determining whether or not to output the review information may be to always output the information for an anticipated event, or to output the information when a generation method for generating all the information that needs to be generated is not yet available and another method is used.

[0060] Furthermore, it may be configured to output when neither output is in time. In other words, if there is no generation method that can output information within the output period, the output period determination unit can switch the output purpose to review and determine the output period and generation method.

[0061] Alternatively, a function for determining whether the output purpose has been achieved may be provided. As will be described later, if a high-priority output request for another anticipated event occurs and the information output cannot be continued within the scheduled length of time, it may be determined that the output purpose has not been achieved, and information for review may be generated and output. Alternatively, if the driver is repeatedly driving in the same manner, information for review may be output.

[0062] Next, examples of the information output period (tp) determined by the output period determination unit 103 for each information generation method will be described with reference to Fig. 6. In this example, the information output period (tp) based on three generation methods is compared: a generation method (method A) that generates information including all information that should be generated for a certain anticipated event, a generation method (method B) that generates only information with high priority, and a generation method (method C) that generates only information with a simple notification such as a buzzer.

[0063] Here, the amount of information to be output is predicted to be in the following relationship: generation method A > generation method B > generation method C, so the information comprehension period (tu) required for the driver to understand the information in generation method A is longer than the information comprehension period (tu) required for the driver to understand the information in generation method B. As a result, the information output start period (ts) at which output of information in generation method A should start starts earlier than the information output start period (ts) at which output of information in generation method B should start. In other words, the amount of information can be said to be the amount of information that fits within the information comprehension period (tu).

[0064] As described above, because the start times of the information output start periods (ts) are different, the information output start period (ts) of generation method B is later than that of generation method A. The end time of the information non-output period (tn), during which output should not start, is set to the same time regardless of the generation method. This is because it is believed that the time during which the driver should pay attention to anticipated events and the surrounding situation remains the same regardless of the generation method.

[0065] On the other hand, in generation method C, there is no information understanding period (tu) required for the driver to understand the output, but rather an information output start period (ts) is set up until just before the occurrence of the anticipated event, and depending on the design policy, an information output period (tp) is set up until a little after the occurrence of the anticipated event, and a non-output period (tn) is set up during which no information should be output for a certain period of time.

[0066] The reason for this is that simple notifications such as buzzers contain less information and place a low cognitive load on the driver's information processing ability, making them less likely to hinder the driver's response to anticipated events, and are thought to have the effect of prioritizing notifications until just before an anticipated event occurs.

[0067] Next, a case where a request for outputting information on a certain anticipated event occurs during a period in which information on another anticipated event is being output will be described with reference to FIG.

[0068] For example, if the vehicle's trajectory sags toward the center of the lane while turning left at an intersection, the system outputs information to advise drivers to be careful with their driving. However, if the system detects a motorcycle approaching from the front while this information is being output, it determines that there is a possibility of a collision with the motorcycle based on the relative speed between the vehicle and the motorcycle and the driver's line of sight, and generates a request to output information urging drivers to be careful of the motorcycle.

[0069] In this situation, the hypothetical event of the trajectory widening during a left turn (called hypothetical event A) alone does not pose an immediate risk of a collision, so the urgency is not high. In contrast, the event of a collision with a motorcycle (called hypothetical event B) is more urgent. For this reason, the priority of information output is higher for warning motorcycles. We will explain how to output information in this situation.

[0070] In Figure 7, for an event occurrence time 701 when the occurrence of anticipated event A is predicted, the following are set using the method described in Figures 5 and 6: an operation start time 702 when the driver should understand the content of the information and begin acting; an information understanding period (tu) required for the driver to understand the output; an information output start period (ts) when output of information begins; an information output period (tp) when the generated information is output; and an information non-output period (tn) when no information is output.

[0071] Next, when the event occurrence time 707 at which the expected event B is predicted to occur is calculated, the operation start time 708 at which the driver should understand the content of the output and begin to act, the information understanding period (tu) required for the driver to understand the output, the information output start period (ts) during which the output of information begins, the information output period (tp) during which the generated information is output, and the information non-output period (tn) during which no information is output are set.

[0072] Actual information output for the anticipated event A is executed during the period (tA) from the information output start time 709 for the anticipated event A to the information output start time 710 for the anticipated event A.

[0073] On the other hand, after the information output start time 710 for the anticipated event B, the output of information for the anticipated event B is executed over a period (tB). Here, the information is output over a period including the information output start time 710 and the information non-output period (tn). However, as mentioned above, the information may be output up to the event occurrence time 707.

[0074] Furthermore, if it is determined that there is a high possibility that information regarding anticipated event B will be requested before information regarding anticipated event A is output or before the information is generated, the output period for information regarding anticipated event A may be changed, or information regarding anticipated event A and anticipated event B may be output in combination.

[0075] Next, a method for changing the information output period (tp) for anticipated event A will be described with reference to FIG. 8A.

[0076] For the event occurrence time 801 at which the occurrence of anticipated event B is predicted, the operation start time 802 at which the driver should understand the content of the output and begin to act, the information understanding period (tu) required for the driver to understand the output, the information output start period (ts) during which the output of information begins, the information output period (tp) during which the generated information is output, and the information non-output period (tn) during which no information is output are set using the method described in Figures 5 and 6.

[0077] Next, a new operation start time 803 at which the driver should understand the content of the information output for anticipated event A and start the required driving operation is set to the earlier of the original operation start time 804 for anticipated event A shown by the dashed line and an information output start time 805 of the information output start period (ts) at which the output of information for anticipated event B starts. In this case, the operation start time 803 is set to the information output start time 805.

[0078] 8A , the information output start time 805 of the information output start period (ts) at which the output of information for anticipated event B starts is earlier than the original operation start time 804 for anticipated event A. Therefore, the original operation start time 804 at which the driver should understand the content of the information for anticipated event A and start driving operation is set to the information output start time 803 of the information output start period (ts) at which the output of information for anticipated event B starts, and the operation start time 804 is brought forward.

[0079] Next, the information comprehension period (tu) required for the driver to understand information about anticipated event A is used to set the information output start period (ts) for starting to output information about anticipated event A. Then, the period from the end time of the information output start period (ts) for anticipated event A to the end time of the information output start period (ts) for anticipated event B is defined as the information output period (tp) for anticipated event A.

[0080] Furthermore, the non-information output period (tn) during which information regarding anticipated event A should not be output is set to a range from the end time of the information output period (tp) for anticipated event A to a predetermined time that is slightly beyond the event occurrence time 806 of anticipated event A, similar to the method described in Figures 5 and 6.

[0081] As a result, the information that is actually output is the period from the information output start time 807 for the anticipated event A to the information output start time 808 for the anticipated event B, and during this period (tA), output of information for the anticipated event A is executed. On the other hand, during the period (tB) from the information output start time 808 for the anticipated event B to the end time of the information output period (tp) for the anticipated event B, output of information for the anticipated event B is executed.

[0082] By using the above method, once the driver has understood the content of the information regarding anticipated event A and is in a state where he or she can perform driving operations, the driver can recognize the content of the information regarding anticipated event B, thereby enabling the driver to respond to both anticipated event A and anticipated event B.

[0083] 7 and 8A illustrate a case where anticipated event B, which occurs later in time, is more urgent than anticipated event A, which occurs earlier. On the other hand, if anticipated event A, which occurs earlier, is more urgent, or if anticipated event A and anticipated event B are equally urgent, it is possible to start outputting information about anticipated event B after outputting information about anticipated event A is completed.

[0084] The determination of completion of output of information on anticipated event A may be made on the condition that the output period set by the output period determination unit 103 has ended, or the time of occurrence of anticipated event A may be set as the completion time of information on anticipated event A even before the end of the output period.

[0085] In addition, on the condition that driving operations have begun in accordance with the information on anticipated event A, it may be determined that the gist of the information on anticipated event A has been understood and the purpose of outputting the information on anticipated event A has been achieved, and the output period for the information on anticipated event A may be terminated early.

[0086] Next, a method for outputting information combining expected events A and B will be described with reference to FIG. 8B.

[0087] First, an operation start time 813 at which the driver should understand the content of the information and start driving operation, and an information comprehension period (tu) required for the driver to understand the information are set relative to the earlier of the event occurrence time 811 of anticipated event A and the occurrence time 812 of anticipated event B. Furthermore, an information output start period (ts) at which the output of information starts, an information output period (tp) at which the generated information is output, and an information non-output period (tn) at which no information is output are set.

[0088] The end times of the information output period (tp) during which the generated information is output and the information non-output period (tn) are set to coincide with the occurrence time of the event that occurs later (in FIG. 8B, the occurrence time of expected event B (812)).

[0089] This method makes it possible to output information about anticipated events A and B with sufficient time to spare, regardless of the urgency of anticipated events A and B, or the interval between the occurrence of anticipated events A and B. This allows the driver to respond to both events with sufficient information.

[0090] The generation method determination unit 104 and the information generation unit 105 create information as shown in Figures 9A to 9C. The examples shown in Figures 9A to 9C show output examples of three methods when a method of displaying text and images on a display mounted on a vehicle is used: a generation method (A) that generates all information to be generated, a generation method (B) that generates only high-priority elements, and a generation method (C) that uses motion information.

[0091] 9A shows a display 901 on which information generated by a generation method for generating all information to be generated is displayed. Text 902 and video 903 are displayed on the display 901.

[0092] Text 902 is made up of two sentences, and points out problems with the current driving and provides detailed instructions on how to improve it. Video 903 shows the positions of the vehicle 904 and the parked vehicle 905, as well as the current vehicle trajectory 906 (dotted arrow) and the ideal trajectory 907 (solid arrow), explaining the degree of deviation.

[0093] The example in Figure 9B shows a display 908 displaying information generated using a generation method that generates only high-priority elements. Text 909 consists of a single "sentence" and succinctly instructs how to improve the situation. Image 910 shows only the position of a parked vehicle 911 and an ideal trajectory 912, with fewer elements drawn.

[0094] The example in Figure 9C shows a display 913 displaying information generated using a generation method that uses motion information. Text 914 consists of a single short "sentence" and succinctly instructs on a generalized improvement method. Image 915 is a simple pictogram 916 (a steering wheel pictogram) indicating that the operation to be improved is steering.

[0095] Here, in the generation method using motion information in Figure 9C, information to be output is generated using a rule base or the like from information obtained from a sensor, so there is no need to communicate with a generative model or server, and information can be generated in a short time.

[0096] In this way, by using these generation methods depending on the grace period between when a demand for information generation arises and when output begins, educational guidance as shown in Figure 9A can be provided when there is grace period, and even when there is little grace period, a generation method that provides as much information as possible as shown in Figures 9B and 9C can be selected to provide information to the driver.

[0097] Although these examples show text and images displayed on a display, the method of outputting information is not limited to a display, and any method that can be perceived by the driver may be used. For example, information may be output as audio, such as text being read aloud, or as music, or may be output as a light that lights up or flashes, or as a color that has meaning. Alternatively, information may be output using vibrations, reaction forces, shape changes, temperature changes, etc., applied by the steering wheel, pedals, seat, etc. that the driver is touching.

[0098] Furthermore, the input information for generating the information may be current information from various sensors, etc., but when a change in the situation is predicted, predicted values ​​may be used as input information. For example, if the current speed and acceleration are not a problem in the moment, but the acceleration is being maintained and it is predicted that the speed will deviate from the appropriate speed in the near future, the timing at which the vehicle speed will deviate may be predicted, and the generation of information to warn about the vehicle speed may be started accordingly.

[0099] This increases the likelihood that information generation can be completed in advance, making it easier to output messages to the driver at the appropriate time. Furthermore, using the results of situation predictions, multiple pieces of information may be generated in advance in preparation for cases where it is predicted that there will be a high need to output information, and the information may be stored appropriately in the memory on the edge device side. In other words, information that may be generated now or in the near future can be generated in advance and stored on the edge device side. This makes it easier to output information appropriate to the situation, even if communication with the server becomes poor.

[0100] Next, using FIG. 10 as an example of information consisting of multiple "sentences" (in this case, a sentence), we will explain a method of dividing the generated information into elements in a database by adding information on "search labels" and "importance of information," and storing it as a data format.

[0101] The target information is an example of a generated sentence such as the sentence 1001 and the sentence 1002 .

[0102] These sentences 1001 and 1002 are stored in a database as a data format 1003. The data format 1003 is assigned the following items: "element," "information type," "search label," and "information importance" of the sentence. As shown in the data format 1003, the generated sentences 1001 and 1002 are divided into, for example, "sentences" (or phrases), which are defined as elements, and the information type, search label, and information importance are associated with these and stored.

[0103] Here, the information type, search label, and information importance may be assigned by analyzing the generated text using a text analysis method, or, for example, when generating texts 1001 and 1002, the structure of the text may be specified (problem operation information in the first sentence, cause information in the second sentence, improvement method in the third sentence, etc.), and that information and information on urgency at the time of generation may be assigned on a rule-based basis.

[0104] Furthermore, by starting the search from elements with high information importance when generating data using past output, it is possible to extract important elements first, making it possible to respond even when the output start period is short.

[0105] In this way, the generation method determined by the generation method determination unit 104 is one of the following: (1) a first generation method that generates a simple notification; (2) a second generation method that uses motion information; (3) a third generation method that uses information generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the information generated in a similar situation in the past; (5) a fifth generation method that inputs motion information into a generation model and generates it using high-priority elements; or (6) a sixth generation method that inputs motion information into a generation model and generates all the information.

[0106] Furthermore, the third generation method or the fourth generation method determined by the generation method determination unit 104 divides the information to be generated into elements and generates it using information with high priority in order to shorten the information generation time. Furthermore, the third generation method or the fourth generation method determined by the generation method determination unit 104 generates information by searching elements stored in a database with reference to search labels. Furthermore, the generation method determination unit 104 has a function of dividing the sentence generated by the fifth generation method into specific elements, assigning search labels to each element, and storing the elements in a database.

[0107] In addition, when the time from the current time to the end time of the output start period is short, the generation method determination unit 104 determines that the input information for generating information using the fifth generation method or the sixth generation method is input information of high importance.

[0108] Furthermore, if the time from the current time to the end time of the output start period is short, the length of the information to be generated by the fifth generation method or the sixth generation method can be specified to be short. Furthermore, when estimating the time required for the fifth generation method or the sixth generation method, the generation method determination unit 104 can generate information by the fifth generation method or the sixth generation method at a time when there is no request to generate information.

[0109] Next, a specific system configuration and a flowchart of this embodiment will be described with reference to FIGS.

[0110] 11 shows the configuration of an information generating device 100 related to this embodiment in a generating device control system 10 mounted on a vehicle. This generating device control system 10 includes an information generating device 100. The information generating device 100 has a so-called generating function, and is provided with a function for outputting necessary information at appropriate timing.

[0111] As described above, the information generating device 100 includes a situation information acquiring unit 101, an output purpose determining unit 102, an output period determining unit 103, a generation method determining unit 104, an information generating unit 105, and an output unit 106. The functions and actions of these units have already been described, so a description thereof will be omitted here.

[0112] Here, information from various vehicle sensors 107A mounted on the vehicle and road sensors 107A arranged on the road is input to the situation information acquisition unit 101. Furthermore, various information is also input from the cloud server 107B, such as information on the driver's driving history and driving tendencies, information on roads frequently traveled by the driver, and information disclosed by the government, etc.

[0113] In addition, the information generation unit 105 is connected to the generative model server 109 via a communication network (wired or wireless), and can transmit the generated information to the generative model server 109, and conversely, receive necessary data from the generative model server 109.

[0114] Furthermore, the output information from the output unit 106 is sent to an output device 108 such as a vehicle display or speaker after the output timing is controlled by an output information control unit 111, and is converted into information such as an image or sound and notified. The output information from the output unit 106 is also sent to a post-stage generation device 110, where it can be used for another model calculation. The output information is information generated in accordance with an output purpose.

[0115] 9A to 9C, the output information control unit 111 controls the output timing by selecting information to be presented or terminated on the output device 108 in accordance with the information output period (tp) determined by the output period determination unit 103, and also has a function of determining whether the output purpose has been accomplished.

[0116] Next, a control flow relating to the generation of information executed by the information generating device 100 will be described with reference to Fig. 12. Note that this description is based on the above-mentioned description of the information generating device 100, and is therefore a rough description.

[0117] In step S10, the information generating device 100 starts an application that generates information under the driving conditions of the vehicle. Furthermore, this application repeatedly executes steps S11 to S26 until the application is terminated in step S27.

[0118] In step S11, the situation information acquisition unit 101 acquires current situation information inside and outside the vehicle from vehicle sensors and road sensors. Next, in step S12, the output purpose determination unit 102 calculates the deviation (Da) between the ideal driving and the current actual driving from the situation information. Next, in step S13, the output purpose determination unit 102 determines the degree of deviation (Da) of the calculated deviation (Da). If it is determined that the deviation (Da) is smaller than a predetermined determination threshold (Th) (No determination), the process proceeds to step S21.

[0119] On the other hand, if it is determined that the deviation amount (Da) is greater than the predetermined determination threshold (Th) (Yes determination), the process proceeds to step S14. In step S14, the output purpose determination unit 102 determines the output purpose (setting of the urgency (Ur)) based on the magnitude of the deviation amount (Da), for example, in three stages. This corresponds to the processing in Figures 3 and 4. Once the output purpose is determined, in step S15, the output purpose determination unit 102 determines whether another output purpose has been defined. This determines whether the output purposes conflict. If it is determined that there is no conflict (No determination), the process proceeds to step S16. In step S16, an information output period corresponding to the current output purpose is determined, and the process proceeds to step S19. On the other hand, if it is determined that another output purpose has been defined (Yes determination), the process proceeds to step S17.

[0120] In step S17, the output purpose determination unit 102 determines whether the current output purpose (urgency (Ur)) has a higher priority than the previously set existing output purpose (urgency (Ur)). If it is determined that the existing output purpose (urgency (Ur)) has a higher priority (No determination), the process proceeds to step S16. Step S16 is as described above.

[0121] On the other hand, if it is determined that the current output purpose (urgency (Ur)) has a higher priority (Yes determination), the process proceeds to step S18. In step S18, the output period determination unit 103 determines the current information output period (tp) and changes the existing information output period (tp). This corresponds to the processes in, for example, Figures 7 and 8A.

[0122] As described above, the information output period (tp) includes an information output start period (ts) during which information output begins, an information comprehension period (tu), an operation execution period (to), and an information non-output period (tn) during which no information is output. Once the information output period (tp) is determined in steps S16 and S18, the process proceeds to step S19.

[0123] In step S19, the generation method determination unit 104 determines an information generation method corresponding to the information output period (tp). In step S20, the information generation unit 105 generates output information using the determined information generation method. This output information is sent to the output device 108 and is notified as information from a display or speaker. This corresponds to the processing in FIGS. 9 and 10.

[0124] When the output information is generated in step S20, the process proceeds to step S21. In step S21, the output information control unit 111 determines whether the information output period (tp) has ended. If it is determined that the information output period (tp) has not ended (No determination), the process proceeds to step S25. On the other hand, if it is determined that the information output period (tp) has ended (Yes determination), the process proceeds to step S22.

[0125] In step S22, the output of information is stopped, and the process proceeds to step S23. In step S23, the output information control unit 111 determines whether the output purpose has been accomplished, and if it determines that it has not been accomplished (No determination), the process proceeds to step S24. In step S24, the output purpose determination unit 102 sets an output purpose for review and performs the review as described above. On the other hand, if it determines that the output purpose has been accomplished (Yes determination), the process proceeds to step S25.

[0126] In step S25, the output information control unit 111 determines whether it is the information output start period (ts) of existing information. This is because it has been determined in step S18 that the priority is low. If it is determined in this step that it is not the information output start period (ts) of existing information (No determination), the process proceeds to step S27. On the other hand, if it is determined that it is the information output start period (ts) of existing information (Yes determination), the process proceeds to step S26.

[0127] In step S26, the information generation unit 105 generates output information using the determined information generation method. This output information is sent to the output device 108 via the output information control unit 111, and is notified as information from a display or speaker.

[0128] While the vehicle is running, the above steps S11 to S26 are repeatedly executed, and when the vehicle is stopped, the process proceeds to step S27 and this application is terminated.

[0129] As described above, this embodiment includes an output purpose determination unit that determines an output purpose for outputting information to the driver using operation information of at least one of the driver and the vehicle, an output period determination unit that determines an output period for the information using the output purpose determined by the output purpose determination unit, a generation method determination unit that determines a generation method for generating information according to the output period, an information generation unit that generates information corresponding to the output purpose based on the generation method determined by the generation method determination unit, and an output unit that outputs the generated information according to the output period, and the output period is set to a period during which the driver can recognize the information output from the output unit.

[0130] This provides an advantage that an information generating device that generates necessary information in real time can output necessary information at the required timing without delay. Note that the necessary information is an amount of information that fits within the information comprehension period.

[0131] In the first embodiment described above, the generation method determination unit 104 selects the generation method with the highest priority from among the generation methods that can output information within the information output start period (ts). In contrast, in the second embodiment, the generation method determination unit 104 determines the generation method and generated data to be output that can be in time for the information output start period (ts) of each generation method. Only the differences from the first embodiment will be described below.

[0132] 13 shows the overall configuration of the information generating device according to Example 2. Similar to the situation information acquiring unit 101 according to Example 1, the situation information acquiring unit 120 acquires information on the operation of the driver and the vehicle, which is a motivation for generating information, and information on the surrounding situation from sensors, maps, input data from the driver, etc.

[0133] Similar to the output purpose determination unit 102 in Example 1, the output purpose determination unit 121 calculates ideal driving from the information acquired by the situation information acquisition unit 120, and determines the output purpose for generating information based on the deviation between this ideal driving and the driver's actual driving.

[0134] Similar to the output period determination unit 103 in Example 1, the output period determination unit 122 sets the information output period (tp) for outputting information for each generation method according to the output purpose determined by the output purpose determination unit 121, which includes information regarding urgency (Ur).

[0135] The information generation unit 123 starts generating information by each generation method using information from the output purpose determination unit 121, in parallel with the output period determination unit 122. The generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer, (2) a second generation method that uses motion information, (3) a third generation method that uses output generated in a similar situation in the past, (4) a fourth generation method that generates information by modifying part of output generated in a similar situation in the past, (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements, and (6) a sixth generation method that inputs motion information into a generative model to generate all information to be generated.

[0136] The generation method determination unit 124 has an output priority for each of the generation methods described above, and adopts the generation method with the highest priority (largest amount of information) that can be completed within the information output start period (ts).

[0137] The output unit 125 outputs the information generated by the generation method adopted by the generation method determination unit 124. With this configuration, it is possible to start generating information early, and therefore it is possible to adopt a generation method with a higher priority.

[0138] Since an increase in the computational load on the server side and the edge device side is expected, the information generating device of Example 2 may be used in combination with the information generating device of Example 1 when the computational load is tolerable.

[0139] Next, another embodiment of the present invention will be described. Below, we will use workers in the manufacturing, quality assurance, and maintenance processes as an example to explain how this embodiment operates when a danger or procedural error occurs during manufacturing or inspection work, and when a warning or correction method is presented. The following will focus on differences from the embodiments described so far.

[0140] In this embodiment, the driver described in the first embodiment is basically replaced with a "worker" and the vehicle is replaced with a "work object." In the configuration of the information generating device shown in FIG. 1, this embodiment operates as follows.

[0141] The situation information acquisition unit 101 acquires information on the operation of the worker and the work object (such as equipment used for manufacturing or inspection, equipment to be manufactured or inspected, equipment for recording manufacturing or inspection, etc.) that motivates the generation of information, as well as information on the surrounding situation, from input data from sensors and the worker.

[0142] As shown in FIG. 14, sensors include sensors 301 built into or attached to the work object, sensors 302 attached to the work environment, wearable sensors 303 worn by the worker, and information on work plans and workers on a server obtained via the Internet or the like, and information 304 disclosed by the government or the like.

[0143] Specific information acquired includes, for example, information about the operation of the work object (direction, amount, and speed of movement of the moving parts, flow velocity of the fluid passage parts, current and voltage, etc.), information about the condition of the work object (temperature, presence or absence of malfunctions, years of operation, etc.), information about the work content and work method of the worker (type of work, work procedure, posture, etc.), information about the worker's work experience, usual work tendencies, physical condition, presence or absence of impatience, etc. (presence or absence of delays from the work plan, pulse rate, etc.), information about the work environment (illumination, temperature, humidity, noise, etc.), etc.

[0144] The output purpose determination unit 102 calculates an "ideal task" from the information acquired by the situation information acquisition unit 101, and determines the output purpose for generating information based on the deviation between this ideal task and the worker's actual task. The ideal task is defined based on the work procedure, and the current ideal task is calculated in light of the current task progress and the work procedure.

[0145] Furthermore, because there are some individual differences in work, ideal work can be defined with a range. If there are large differences in work methods between individuals or teams, work procedures can be adapted to each individual or team to absorb the differences.

[0146] The output purposes for generating information include, for example, emergency notifications about dangerous situations, warnings about deviations in work (manufacturing, inspection, etc.) methods (no danger occurs, but there is a problem with the work results), and suggestions for improving efficiency (the results are correct, but the methods have quirks, or there is a better way).

[0147] If a problem occurs during a manufacturing line process or while working in separate teams, it may affect the work of other workers depending on the extent of the dangerous incident or rework. For this reason, it may be possible to set up output purposes to notify other workers on the line and supervisors (notifying the current work progress of other workers and changes to the work plan).

[0148] The classification of the purpose can be set in advance, for example, according to the type and degree of deviation between the ideal work and the actual work for each worker. The output purpose also includes information about urgency, as in the first embodiment.

[0149] The output period determination unit 103 sets an information output period for outputting information for each generation method in accordance with the output purpose including information on urgency determined by the output purpose determination unit 102 .

[0150] The generation method determination unit 104 determines a generation method for generating information based on the output purpose and urgency determined by the output purpose determination unit 102, the information output period determined by the output period determination unit 103, and the estimated time required for each generation method.

[0151] The information generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer; (2) a second generation method that uses motion information; (3) a third generation method that uses output generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the output generated in a similar situation in the past; (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements; and (6) a sixth generation method that inputs motion information into a generative model to generate all the information that should be generated.

[0152] Furthermore, the generation methods are prioritized, and the generation method determination unit 104 selects the generation method with the highest priority from among the generation methods that can output within the information output start period. Note that, when the time available for generation is short, in a method that uses past output as is or with partial modification, or in a method that uses a generative model for generation, the length of the sentence to be generated may be specified to be short, that is, a part of the past output may be output, or the number of words in the sentence to be generated may be set to be small.

[0153] In addition, the expected time required for each generation method may be set by measuring the average or maximum value in advance for each generation method, or, since communication speed and processing speed with the server change as needed, dummy information (previous information) may be generated as needed when the possibility of an output generation request is low (for example, between tasks) and used to make an estimate.

[0154] The information generation unit 105 generates information using the generation method determined by the generation method determination unit 104 (it may also use information from the situation information acquisition unit, etc.). The output unit 106 outputs the generated information to the worker. The output method may be a method using a display or speaker provided on the work object, or a method using a display or speaker on a smartphone, work tablet, or the like owned by the worker.

[0155] Next, a control flow relating to the generation of information executed by the information generating device 100 according to this embodiment will be described with reference to FIG.

[0156] Note that the explanation here is substantially the same as that of the information generating device 100 in the first embodiment described above, and therefore the step reference numbers have an "A" suffix. Therefore, explanations of the same steps will be omitted. Note that while the first embodiment describes information targeted at vehicle driving operations, the present embodiment describes information targeted at work performed by a work machine.

[0157] 12 for the first embodiment, steps S28, S29, and S30 have been added, with the aim of providing information to individuals and teams.

[0158] In step S28, the output purpose determination unit 102 performs individual adaptation of the ideal work. If there are large differences in work methods between individuals or teams, the work procedure manual is adapted for each individual or team to absorb the differences. This allows the ideal work to be set for each individual or team.

[0159] Furthermore, steps S29 and S30, which execute a repeat loop, are inserted after steps S14A and S26A, respectively. As a result, in cases where the information is output to destinations other than the worker himself (workers on the same line, workers in the same factory, supervisors of the line or factory, workers on a team working in cooperation with each other, etc.), steps S15A to S26A are executed for each destination.

[0160] Next, another embodiment of the present invention will be described. The example described below is an example in which, when a certain work time is set, the number of times or length of information output does not take up too much of the work time.

[0161] For example, we will explain how this embodiment works when tracking the progress of a student in an educational setting and providing some kind of notification to the student or educator, but below we will focus on the differences from the embodiments described so far.

[0162] In this embodiment, the driver described in the first embodiment is basically replaced with a "student" and the vehicle is replaced with an "education-related device." In the configuration of the information generating device 100 shown in FIG. 1, this embodiment operates as follows.

[0163] The situation information acquisition unit 101 acquires information on the operation of the student and education-related devices (equipment operated by the student, or equipment operating for education, etc.) that motivates the output of information, as well as information on the surrounding situation, from sensors, input data from the student, etc.

[0164] As shown in the configuration in Figure 16, the sensors include sensors 401 built into or attached to education-related devices (in Figure 16, a bat operated by the student or a pitching machine that throws balls for the student's batting practice), sensors 402 attached to the educational environment, wearable sensors 403 worn by the student, and information on educational plans and progress on a server obtained via the Internet, etc., and information 404 disclosed by the government, etc.

[0165] Specific information acquired includes, for example, information regarding the operation of education-related devices, information regarding the status of education-related devices, information regarding the education plan (schedule, unit information, progress, remaining education time, etc.), information regarding the behavior of the student, information regarding the student's education experience, usual behavioral tendencies, physical condition, whether or not they are anxious, etc. (information on time limits, exams, pulse rate, etc.), and information regarding the educational environment (weather, illuminance, temperature, humidity, noise, etc.).

[0166] The output purpose determination unit 102 calculates an ideal movement from the information acquired by the situation information acquisition unit 101, and determines an output purpose based on the deviation between this ideal movement and the actual movement of the trainee.

[0167] Here, the ideal motion is defined based on the educational objectives, the progress of the trainee, the level of proficiency, etc. For example, if the trainee is a beginner at baseball, the ideal motion is first defined as being able to hold the bat in the correct way, and if the trainee has some baseball experience, the ideal motion is defined as being able to swing the bat without wavering. Such ideal motions may be defined in advance, or may be defined statistically by collecting information on many trainees.

[0168] Examples of output purposes include emergency notifications regarding dangerous behavior, warnings about deviations in behavior (which do not pose a danger but are not good behaviors), instructions or suggestions for the next behavior (next unit, break, etc.), and notifications to educators (for example, current progress, the status of the student, future plans, and notifications of dangerous events).

[0169] The classification of output purposes is set in advance, for example, according to the type and degree of discrepancy between the ideal movement and the trainee's actual movement. The output purpose also includes information on the urgency of the output. Urgency can be, for example, most importantly warning about dangerous movements, followed by correcting the trainee's bad habits, and these can be set in advance for each item.

[0170] The output period determination unit 103 sets an information output period for outputting information for each generation method in accordance with the output purpose, including information regarding urgency, determined by the output purpose determination unit 102. Note that in output for educational purposes, output before an event is not necessarily desirable for all output purposes, since the output is intended to develop the trainee's own thinking ability and trial and error ability, and depending on the output purpose, it may also be effective to set an information output period after the event.

[0171] For example, if the trainee is standing too close to the base and there is a risk of the ball hitting them, the system will output information urging them to correct their position in advance.On the other hand, if the trainee is standing too far from the base and it is predicted that their form will be disrupted when they hit the ball, the system can output advice after the trainee has tried batting from that position at least once.

[0172] Alternatively, multiple output purposes and information output periods (for example, periods for outputting brief information before an operation, detailed explanatory information immediately after an operation, review information during a review period, etc.) may be set for one event, such as briefly explaining the "problem operation → improvement method" before the operation and then providing a detailed explanation after the operation is completed.

[0173] Furthermore, the output period determination unit 103 may predict that executing all outputs may cause a delay in the progress of the education unit or that the planned education content may not be able to be executed within the remaining time, taking into account the progress of the education unit, the remaining time of the day, etc. In this case, the information output period for information with low urgency may be set after the end of the education time for that day.

[0174] The generation method determination unit 104 determines a method for generating information based on the output purpose and urgency determined by the output purpose determination unit 102, the information output period determined by the output period determination unit 103, and the estimated time required for each generation method.

[0175] The information generation methods include one or more of the following: (1) a first generation method that generates a simple notification such as a buzzer; (2) a second generation method that uses motion information; (3) a third generation method that uses output generated in a similar situation in the past; (4) a fourth generation method that generates information by modifying part of the output generated in a similar situation in the past; (5) a fifth generation method that inputs motion information into a generative model to generate only high-priority elements; and (6) a sixth generation method that inputs motion information into a generative model to generate all the information that should be generated.

[0176] Here, the generation methods are prioritized, and the generation method determination unit 104 selects the method with the highest priority as the generation method from among the generation methods that can be output within the information output start period.

[0177] If the time available for generating information is short, a generation method can be adopted that uses past output as is or with some modification.Furthermore, in a generation method that uses a generative model, the length of the generated sentence can be specified to be short, that is, a part of the past information can be output, or the number of words in the generated sentence can be reduced.

[0178] The expected time required for each generation method may be set by measuring the average or maximum value in advance for each generation method, or, since communication speed and processing speed with the server change as appropriate, dummy information may be generated as appropriate when the possibility of a request for information generation is low (for example, during breaks or the time between certain actions).

[0179] The information generating unit 105 generates information to be output using the generation method determined by the generation method determining unit 104 (it may also use information from the situation information acquiring unit, etc.). The output unit 106 outputs the generated information to the student. The output method may be a method using a display or speaker provided in the educational environment or education-related device, or a method using a smartphone or earphones owned by the student.

[0180] Next, a control flow relating to the generation of information executed by the information generating device 100 according to this embodiment will be described with reference to FIG.

[0181] Since the explanation here is substantially the same as that of the information generating device 100 in the first embodiment, the step reference numbers are suffixed with "B." Therefore, explanations of the same steps will be omitted. Note that while the first embodiment explains information related to vehicle driving operations, the present embodiment explains information related to the movements of the trainee.

[0182] 12 for the first embodiment, steps S40 to S48 have been added. These steps are intended to provide information to the trainee.

[0183] As a difference from the flowchart of the first embodiment, in step S40, the output purpose determination unit 102 sets a threshold value of urgency (Sh) based on the remaining time of the training. The threshold value of urgency corresponds to, for example, the progress level relative to the remaining time of the training, and the lower the progress level relative to the remaining time, the higher the threshold value of urgency.

[0184] In step S41, the output purpose determination unit 102 determines whether the urgency exceeds a threshold value. In step S41, the output purpose determination unit 102 determines the degree of urgency of the current output purpose relative to the urgency threshold value (Sh) set in step S40.

[0185] If it is determined in step S41 that the urgency is equal to or greater than the threshold, the steps from step S42 onward are executed to set the information output period, which is set based on the output purpose in step S14B.

[0186] On the other hand, if it is determined that the urgency is below the threshold (Sh), the process proceeds to step S43, where the output purpose determination unit 102 changes the output purpose to "review" and sets the information output period to "review period." The review period is a preset period just before or after the end of the education period, or a period during which the trainee inputs a command to check the review output.

[0187] Here, steps S42 to S44 are executed in a repeated loop for each output destination (for example, for each trainee).

[0188] In steps S45 to S47, the output information control unit 111 outputs the review information generated in steps S19B and S20B while determining that the period is a review period. The urgency threshold (Sh) may be set to be higher in inverse proportion to the remaining time of the education, or may be set in stages according to the remaining time of the education.

[0189] In the example of Figure 17, review information is generated each time and stored on a server or edge device, and is displayed during the review period. However, if there is too much review information, it may not be possible to review it all during the review period. Therefore, the review information may be displayed in order of urgency. Also, the review information may be re-entered into the generative model, and the key points may be summarized and the amount of information reduced before being notified.

[0190] Next, another embodiment of the present invention will be described. Below, we will explain how this embodiment operates when another model is the output target and information (in this case, output information) is output to this model. Below, we will mainly explain the differences from the embodiments described so far. <Case 1> In Case 1, we will explain the behavior of a generative model (Image to text, video to text, data to text, etc.) that allows a subsequent generation device (here, a Text-to-Text Transfer Transformer) 110 to understand a scene.

[0191] The following example explains how to generate information for a generative model that generates output information for the driver from images from external cameras and vehicle sensor information mounted on the vehicle. In this example, we assume that the system can grasp the external world using point cloud information from LiDAR, etc., in addition to the images input to the generative model.

[0192] In the output purpose determination unit 102 in Case 1, since the scene has not yet been understood, it is difficult to determine a high-level output purpose as described in Examples 1 to 4. Therefore, the output purpose and urgency are determined for each object detected by the sensor using a rule base or the like.

[0193] For example, based on information on the positions and relative speeds of features and other vehicles, the system determines the level of urgency by obtaining information on features close to the vehicle, information on features in the direction the vehicle is traveling, information on deviations in driving operations, etc., and sets a higher level of urgency for items with a higher level of danger.

[0194] The output period determination unit 103 in Case 1 predicts dangerous events and determines a data output period for each detected object on a rule-based basis. For example, it calculates the future positions of the host vehicle and the oncoming vehicle from the trajectories and speeds of the host vehicle and the oncoming vehicle, calculates the predicted collision time when a dangerous event, such as a collision, will occur, and determines the data output period and data output start period, etc., to be a preset time before the predicted collision time.

[0195] The generation method determination unit 104 in Case 1 determines the amount of information to be output to the subsequent generation device 110 according to the length of time until the earliest output start period. If the length of time until the output start period is sufficiently long, all information for the object is generated and output, and if the length of time is short, only information with high urgency is selected, generated, and output.

[0196] In Case 1, the output destination, unlike in the first to fourth embodiments described above, is not a display or speaker, but a subsequent generation device 110. Therefore, the output purpose determination unit 102 functions as a recognition unit.

[0197] Furthermore, if a sensor such as LiDAR is not installed and the output purpose and urgency cannot be grasped on a rule-based basis when determining the output purpose, a request for the generation speed (corresponding to the time length of the data output period) may be obtained from the subsequent generation device 110 or the generation device control system 10, and if the requested generation speed is short, only output information related to a predetermined object may be generated.

[0198] In other words, if the response of the server, which is the subsequent generation device 110, is slow and the required generation speed is short, only text related to objects of a high urgency type is generated based on a list that matches the object type and urgency that is stored in advance.

[0199] As a result, if no object is specified, text is generated that includes the driving environment, etc. However, if an object is specified, only text related to surrounding vehicles is generated, which shortens the time required to complete generation compared to generating text that includes all information.

[0200] It can also be applied to a model that generates text that understands a scene in order to generate robot behavior (text to X). The following explains the case of a robot.

[0201] In this case, the output purpose determination unit 102 determines the urgency based on information necessary for danger avoidance (position and speed information of surrounding objects, including the ground) and information necessary for action generation (facial expressions and speech of nearby people, and the state of the object of the action), and sets a higher urgency for an action with a higher degree of danger.

[0202] Furthermore, the output period determination unit 103 cooperates with the subsequent generation device 110 to set the timing at which the next information is required based on the robot's action plan.

[0203] Furthermore, the generation method determination unit 104 generates all output information if the time length until the earliest dangerous event is long, and generates only output information with high urgency if the time length is short.

[0204] From this perspective, the following flowchart is executed: Figure 18 shows a flowchart outlining this embodiment.

[0205] Note that the explanation here is similar to that of the information generating device 100 in the first embodiment described above, and therefore similar steps are denoted by the suffixes "C" and "D" to the reference numbers of the steps. Therefore, explanations of the same steps may be omitted.

[0206] Furthermore, in this embodiment, the target is not viewable information presented to drivers, workers, etc., but data forming information.

[0207] In step S50, the output purpose determination unit 102 determines whether an output purpose can be determined for each object. A situation in which an output purpose can be determined for each object refers to a situation in which a sensor such as LiDAR is available and information on the characteristics (position, speed, etc.) of each object can be obtained, or a situation in which objects to be noted are predefined in an application such as a test witness application.

[0208] If it is determined that the output purpose can be determined for each object (Yes determination), the following steps are executed for each object in step S51.

[0209] In step S14C, the output purpose determination unit 102 determines the output purpose and urgency for each object. The output purpose determined in this step includes the output purpose completion condition. Next, if it is determined in step S15C that an output purpose other than the object has already been defined (Yes determination), the process proceeds to step S17C.

[0210] In step S17C, the output purpose determination unit 102 compares the urgency of the current output purpose with the urgency of the existing output purpose, and if it determines that the current urgency is higher (Yes determination), in step S18C, the output period determination unit 103 determines the current data output period and changes the existing data output period.

[0211] On the other hand, if it is determined in step S15C that there is no setting other than the current output purpose (No), or if it is determined in step S16C that the current urgency is equal to or less than the existing urgency (No), the current data output period is set in step S17C.If it is determined in step S17C that the current urgency is greater than the existing urgency (Yes), the current data output period is set and the existing output period is changed in step S18C.

[0212] In step S19C, the generation method determination unit 104 determines a generation method for the current output purpose corresponding to the data output period. Then, in step S20C, the information generation unit 105 generates output information using the determined generation method. Next, in step S21C, if the output information control unit 111 determines that any data output period for the corresponding object has ended (Yes determination), then in step S22C, the output information is stopped.

[0213] Next, in step S23C, the output information control unit 111 determines whether the output purpose of the stopped output information has been completed, and if it determines that it has not been completed (No determination), proceeds to step S24C to set a re-output purpose for this output information, and returns to step S15C to execute it again.

[0214] In addition, a situation in which the output purpose is completed is, for example, a situation in which all output information is passed to the subsequent generation device 110, or in which there is no further generation request from the subsequent generation device 110, and is a condition determined when determining the output purpose in step S14C.

[0215] In the first to fourth embodiments, the output target for outputting information was a person such as a driver, worker, or trainee, and therefore the completion of the output purpose could be determined, for example, by whether the output was continued for a predetermined period of time or more, whether the time set for the person to understand the output content and start the next action had passed, whether the person had taken an action that indicated that the output content had been understood, etc. In contrast, in this embodiment, the output target is the subsequent-stage generation device 110.

[0216] In step S23C, if the output information control unit 111 determines that the output purpose of the stopped output information has been accomplished (Yes determination), it proceeds to step S25C and determines whether any output start period for the object in question has begun.

[0217] If it is determined that the output start period for any of the objects has begun (Yes), the process proceeds to step S26C and starts outputting the output information. Once the output information has been output in step S26C, the process proceeds to step S52, where processing for each object is completed.

[0218] On the other hand, returning to step S50, if the output purpose determination unit 102 determines that it is not possible to determine an output purpose for each object (No determination), the process proceeds to step S53, where a generation speed request is obtained from the integrated system of the subsequent generation device 110. Based on this information, the output purpose determination unit 102 determines an output purpose in step S14D, and the output period determination unit 103 determines a data output period in step S15D. Steps S15D and after operate in the same way as steps S15C and after.

[0219] Furthermore, if the output information control unit 111 determines in step S23D that the output purpose has not been accomplished, the process returns to step S53, a generation speed request is made, and step S14C and subsequent steps are repeated. <Case 2> In case 2, an application that draws a 3D space using multiple generation models from sensor data (for example, an image) will be described as an example.

[0220] When constructing a 3D model from an image, it is necessary to estimate multiple pieces of information, such as the three-dimensional shape, parts not visible in the image, and color when the light hits it differently. In addition, when drawing, it is necessary to calculate how it will look based on the viewpoint and light source settings, and if the model is designed to allow people to interact with it, it is necessary to calculate and draw changes in the shape and appearance of the 3D model in response to people's movements.

[0221] For this reason, it is conceivable to combine generative models specialized for each function, or to combine multiple models specialized for each function within a generative model, to render a 3D space.

[0222] In particular, in applications that are expected to render real space in high real-time, it is important to reduce the time required for generation for each generative model or generative function.

[0223] In the following, digital twins created with high real-time capabilities will be referred to as real-time twins for convenience. Possible applications include real-time twins for test observations and real-time twins for tourism. The rendering speed is determined by factors such as the speed of movement within the metaverse space.

[0224] The output purpose determination unit in Case 2 is set according to the purpose of generating a real-time twin. For example, if the purpose is to witness a test, the requirement for the test equipment is the highest, and the requirement for generating surrounding people, buildings, etc. is low.

[0225] In addition, the output period determination unit in Case 2 determines the generation period in which each generative model or generative function can be used by calculating backward from the overall output period. The ratio of the generation periods of each generative model or generative function may be determined from the ratio of the average values ​​of each generative model or generative function, or the weight of each generative model or generative function may be changed depending on the object to be drawn.

[0226] For example, when witnessing a car crash test, three-dimensional information such as the deformation of the car at the time of collision is more important than the color, etc. For this reason, the output period of the generative model or generative function that estimates the three-dimensional shape is made longer, and the output period of the generative model or generative function that estimates the color is made shorter. The weighting may be set in advance on a rule-based basis according to the purpose of generating the real-time twin, or may be determined by receiving a request from the user.

[0227] Furthermore, the generation method determination unit in Case 2 can select from (1) a detailed generation method, (2) a generation method with reduced resolution, or (3) a generation method that reuses past information for each object depending on the output purpose, urgency, and output period. For example, past data can be reused for similar information such as signals.

[0228] In Case 2, due to the nature of real-time spatial rendering, when a new output purpose is set for an object for which an output purpose has already been defined, it is highly likely that the existing output purpose will become unnecessary. In such a case, "changing the existing output period" in steps S18C and S18D is synonymous with "discarding the existing output period."

[0229] The present invention is not limited to the above-described embodiments, but includes various modifications. The above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace other configurations with respect to the configuration of each embodiment.

[0230] Furthermore, some or all of the above-described configurations, functions, etc. may be implemented in hardware, for example, by designing them as integrated circuits. Furthermore, the above-described configurations, functions, etc. may be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD.

[0231] 101: situation information acquisition unit, 102: output purpose determination unit, 103: output period determination unit, 104: generation method determination unit, 105: information generation unit, 106: output unit.

Claims

1. An information generation device that generates information to be output to a worker, comprising: an output goal determination unit that determines an output goal, which is a goal for outputting the information to the worker, using motion information of at least one of the worker and a work object; an output period determination unit that determines an output period for the information using the output goal determined by the output goal determination unit; a generation method determination unit that determines a generation method for generating the information corresponding to the output period; an information generation unit that generates the information corresponding to the output goal based on the generation method determined by the generation method determination unit; and an output unit that outputs the information generated by the information generation unit during the output period, wherein the output period determination unit sets an adjustable period necessary for the worker to recognize the information as the output period.

2. An information generating device according to claim 1, characterized in that the information includes one or more of text, video, images, sound, light, vibration, reaction force, shape, and temperature.

3. An information generating device as described in claim 1, wherein the output purpose corresponds to the information required by the worker, or the information determined to need to be output to the worker, as defined based on the operation information, and includes information on urgency.

4. An information generating device as described in claim 1, characterized in that the output period is a period obtained by adding together at least an output start period indicating the period during which output of the information should begin, an information understanding period required for the worker to recognize the information, and an operation execution period required for the worker to operate the information.

5. An information generating device according to claim 1, wherein the output period is defined as a period obtained by adding together at least an output start period indicating the period during which output of the information should begin, an information understanding period required for the worker to recognize the information, an operation execution period required for the worker to operate, and an information non-output period set following the end of the operation execution period during which the information is not output.

6. An information generating device according to claim 5, wherein the output period determination unit sets a review output start period and a review output period after the information non-output period.

7. An information generating device according to claim 5, characterized in that the output period determination unit starts outputting the information during the output start period and ends outputting the information at the end of the operation execution period or the start of the information non-output period.

8. An information generating device according to claim 4, wherein the output period determining section adjusts the time length of the output period in accordance with the amount of information of the information.

9. An information generating device according to claim 8, wherein the generation method determination unit selects a generation method for generating the information within the range of the time length of the output period, and generates the information.

10. An information generation device as described in claim 1, characterized in that, if there is no generation method that can output the information within the output period, the output period determination unit switches the output purpose to a review purpose and determines the output period and the generation method.

11. An information generation device as described in claim 8, wherein the generation method determined by the generation method determination unit is one of: a first generation method for generating a simple notification; a second generation method using motion information; a third generation method using the information generated in a similar situation in the past; a fourth generation method for generating information by modifying part of the information generated in a similar situation in the past; a fifth generation method for inputting motion information into a generation model and generating information using high-priority elements; and a sixth generation method for inputting motion information into a generation model and generating all information.

12. An information generation device as described in claim 11, characterized in that the third generation method or the fourth generation method determined by the generation method determination unit divides the information to be generated into elements and generates it using information with a higher priority.

13. An information generation device as described in claim 11, characterized in that the generation method determination unit has the function of dividing the sentence generated by the fifth generation method into specific elements, assigning a search label to each of the elements, and storing them in a database.

14. An information generation device as described in claim 13, characterized in that the third generation method or the fourth generation method determined by the generation method determination unit generates the information by searching the elements stored in the database with reference to the search label.

15. An information generation device as described in claim 11, characterized in that when the time from the current time to the end time of the output start period is short, the generation method determination unit determines that the input information for generating information using the fifth generation method or the sixth generation method is the input information with higher importance.

16. An information generation device as described in claim 11, characterized in that the generation method determination unit specifies the length of the information generated by the fifth generation method or the sixth generation method to be short when the time from the current time to the end time of the output start period is short.

17. An information generation device as described in claim 11, characterized in that, when estimating the time required for the fifth generation method or the sixth generation method, the generation method determination unit generates the information using the fifth generation method or the sixth generation method at a time when there is no request to generate the information.

18. An information generating device as described in claim 1, characterized in that when a new output purpose is determined, the information corresponding to the output purpose with the highest urgency is output with priority.

19. An information generating device according to claim 1, characterized in that the information that is currently or may be generated in the near future is generated in advance and stored on the edge device side.

20. An information generation method for generating information to be output to a worker, comprising: an output purpose determination step for determining an output purpose, which is a purpose for outputting the information to the worker, using motion information of at least one of the worker and a work object; an output period determination step for determining an output period for the information using the output purpose determined in the output purpose determination step; a generation method determination step for determining a generation method for generating the information corresponding to the output period; an information generation step for generating the information corresponding to the output purpose based on the generation method determined in the generation method determination step; and an output step for outputting the information generated in the information generation step during the output period, wherein in the output period determination step, an adjustable period necessary for the worker to recognize the information is set as the output period.

21. A vehicle equipped with an information generating device that generates information to be output to a driver via a display, wherein the information generating device is an information generating device described in any one of claims 1 to 19.

22. A display equipped with the information generating device according to claim 1.

23. A speaker equipped with the information generating device according to claim 1.

Images