Method for operating an assistance system, assistance system and means of transport therewith

Abstract

The invention relates to a method for operating an assistance system (1) in a means of transport (2), in particular in a motor vehicle, with camera-based detection of an activity of a user (4) of the means of transport (2) in a passenger compartment (5) of the means of transport (2) on the basis of image data of a camera unit (3), by a computing unit (6); with estimation of a period of time for the detected activity by the computing unit (6); with delivery of activity data, which specify the detected activity and the period of time estimated for the detected activity, to at least one functional unit (7); and with performance of at least one function by the functional unit (7) according to the period associated with a specified activity in the activity data.

Description

[0001] METHOD FOR OPERATING AN ASSISTANCE SYSTEM, ASSISTANCE SYSTEM AND MEANS OF TRANSPORT THEREOF

[0002] The disclosure relates to an assistance system in a means of transport, in particular in a motor vehicle, with camera-based recognition of an activity of a user of the means of transport in a passenger compartment of the means of transport based on image data from a camera unit and an adapted execution of a further function by a functional unit. In the interior of a means of transport, i.e. a passenger compartment of a means of transport, such as in a passenger car or a truck, but also in a drone taxi or a gondola, it is becoming increasingly important to identify the activities, in particular activities for which assistance is available from the means of transport and / or activities that are not permitted when using the means of transport in the respective role of the user, and possible

[0003] To detect distractions of users, i.e. the vehicle occupants and in particular of a user responsible for driving the vehicle. This makes it possible, for example, to optimize the handover of driving responsibility from a navigation and / or control unit of a (partially) automated vehicle back to the driver, to adapt assistance systems of the vehicle to the respective activity of the user, for example by proactively offering functions, or to take targeted safety measures, such as deploying an airbag adapted to a respective detected body pose. In this way, driving functions, comfort functions and safety functions in the vehicle can be improved based on a respective detected activity. The comfort functions can in particular include entertainment functions such as gaming functions and / or communication functions and / or work functions.The safety functions may in particular include a monitoring function to ensure appropriate behavior when there is no person with driving or management responsibility, driver and / or no staff in an autonomous vehicle.

[0004] A known example is body pose activity recognition, in which one or more cameras in the passenger compartment detect people in the image data generated by the cameras, and for each detected person, a configuration of their skeleton is recognized in three dimensions, i.e., calculated according to a predefined anatomical model. In particular, the passenger compartment can refer to a closed space intended for accommodating users, such as the passenger compartment of a sedan, and / or a partially open space intended for accommodating users, such as the passenger compartment of a convertible, and / or an open space intended for accommodating users, such as the passenger compartment of a motorcycle. The respective activity is then recognized or classified from the movement of the limbs, the position of known objects or devices in the passenger compartment, such as the steering wheel, and possibly other features, i.e.recognized as belonging to a type of activity. For example, classes such as "opens bottle," "drinks," "uses phone," "types on laptop," "sleeps," and the like can be specified for this classification or recognition. The recognition can also include a hierarchical class model, in which classes can each have one or more subclasses. For example, the class "uses phone" can include the subclasses "uses phone with mobile phone to ear" and "uses phone with hands-free device." As an alternative to this type of activity recognition, other approaches can also be used, such as end-to-end recognition, in which the image data from the cameras is fed directly into a classifier, and activities are recognized accordingly without intermediate steps such as the described analysis of the body pose.

[0005] Based on the known systems for detecting a user's activity and adapting a function of the means of transport to a detected activity, the task is to improve the comfort and safety of the means of transport.

[0006] This object is achieved by the subject matter of the independent claims. Advantageous embodiments emerge from the dependent claims, the description, and the figure.

[0007] One aspect relates to a method for operating an assistance system in a means of transport, in particular a driver assistance system in a means of transport. The means of transport can be an airborne means of transport such as a drone taxi or a gondola and / or a water-based means of transport such as a ship and / or a land-based means of transport such as a motor vehicle, i.e. a passenger car and / or a truck and / or a bus. One method step is a camera-based detection of an activity of a user of the means of transport, in particular of a user who at least temporarily bears the driving responsibility for the means of transport, in a passenger compartment of the means of transport. This detection is carried out by a computing unit based on image data from a camera unit. The activity can also be used, at least implicitly, to detect distractions of the occupants of the means of transport from traffic and / or another activity.The computing unit may comprise a central computer and / or several distributed computers, for example in the form of respective control units ("controllers"). The camera unit may comprise one or more cameras.

[0008] A further method step is an estimation of a temporal duration for the recognized activity, i.e. an estimation of the length of time in which the user pursues the recognized activity according to a respective prediction, by the computing unit. This can be done, for example, empirically based on activities specifically recognized for the user in the past and / or according to a stored specification. The estimation of the temporal duration can also be based on additional information provided to the computing unit; for example, in many articles provided on the Internet, an expected reading time is stated as additional information. Estimation can therefore be understood as a calculation under certain assumptions.Alternatively or in addition to the duration, the mental strain associated with the activity, which can also be understood in the sense of "cognitive involvement," can also be estimated, and / or a fallback time, i.e., the time a person needs to transition from the recognized activity to another activity. The other activity is or includes, in particular, assuming leadership responsibility, whether with or without control intervention in the means of transport. The fallback time can therefore be understood as the time required for a transition from the recognized activity to the other activity after a warning from the means of transport. Mental strain and fallback time can be estimated independently of one another or together, for example, the fallback time can be estimated based on the mental strain.Relevant models and / or empirical data can be used for estimation.

[0009] A further method step is providing activity data, which specify the detected activity and the estimated duration of the detected activity and the duration assigned to the activity in the activity data, to at least one functional unit. Alternatively or additionally, the activity data can specify the estimated mental strain assigned to the respective detected activity and / or the relapse time assigned to the respective detected activity. The functional unit then performs a respective function depending on the detected activity specified in the activity data and / or the associated duration and / or estimated mental strain and / or estimated relapse time assigned to the detected activity.

[0010] For example, a camera can detect that the user is reading. If reading is estimated to take a longer time, such as reading an article, a book, or emails, the lighting function of an interior lighting unit can be adjusted as an additional functional unit, for example, by increasing the brightness. However, if the activity—in this case reading—is estimated to take a shorter time, such as reading a text message, the lighting can be omitted to prevent the eyes from adapting to reading and thus potentially increase safety.The execution of the function by the functional unit depending on the specified activity and / or the associated duration can therefore include checking the length of the duration, whereby in particular the function of the further functional unit is then adapted, preferably automatically adapted, if the estimated duration exceeds a predetermined limit duration. This limit can be at least 15 seconds, for example.

[0011] This has the advantage that assistance systems can adapt better than before to the user's respective activity and therefore also makes it possible to proactively offer functions or to select the right time for proactive messages, in particular requests for action for the user. In particular, a possible handover of driving responsibility from a (partially) automated vehicle back to the driver can be optimized in this way, as will be explained in more detail below. The effectiveness of safety measures can also be improved in this way, for example by taking into account a body posture typical for a certain activity. The detected activity can therefore serve as the basis for many use cases. If the user wants to read a book, for example, this can be recognized and a reading light can be switched on automatically, or the reading intention can be taken into account when dimming the interior lighting.The user to whom driving responsibility is to be transferred is on the phone, the navigation unit must provide a timely warning or wait until the call is ended. If a particular activity requiring high concentration is detected for the user in the driver and / or front passenger seat, it may be preferable, for example, for a fully automated vehicle to avoid choosing a dynamically intensive route that could irritate or distract the user, or even trigger motion sickness. By estimating the duration of detected activities, settings can be made automatically that were previously made manually by the driver or another user. Accordingly, manual interventions in the vehicle's functions are reduced.This not only contributes to increased comfort, but can also significantly increase safety, as the assistance system can take more information into account when making the appropriate settings and, in particular when planning a route, can take a longer time horizon into account than is possible for a human user.

[0012] The approach described here is therefore based on a paradigm shift: Previously available assistance systems in the interior of vehicles or for (partially) automated driving require the driver or passengers to adapt to the assistance systems or configure them to meet their needs. With the help of the solution presented here, which can be summarized as an "activity manager for adaptive vehicle assistance functions," the systems in the vehicle can adapt to the occupants and the prevailing interior situation, as well as plan ahead. This increases comfort and safety for the occupants. The more predictive and thus more intuitively understandable "behavior" of the systems also increases society's acceptance of the increasing mechanization and automation of vehicles.

[0013] In a further embodiment, the detected activity is stored in an activity database by a storage unit, along with the estimated and / or otherwise recorded associated time duration and / or a specification of the mental strain associated with the detected activity and / or the relapse time associated with the detected activity. The activity database can also contain further information, for example typical start times or locations for the activities. This can be done, for example, using so-called artificial intelligence (AI), i.e. a trained or learning algorithm which recognizes recurring patterns, in particular characteristics such as a typical time duration of recurring patterns of detected activities. In particular, the AI ​​can comprise a neural network or be such a neural network.The duration is estimated based on identical and / or similar activities according to a predefined similarity criterion and the duration stored for the identical or similar activity. Especially when using a CL, especially a CL with a neural network, the similarity criterion can be implicitly specified by the structure of the learned algorithm, particularly by the neural connections of the neural network.

[0014] The activity database can also contain sequences of different activities that are to be expected with a corresponding probability. This can also be done on a user-specific basis. This improves the estimation of the time duration, particularly on a user-specific basis, and also enables further planning, for example in the form of to-do lists with a series of activities and with the durations of the different activities in the means of transport for the respective users. The activity manager can thus be enabled to recognize not only for the users, i.e. occupants of the means of transport, which activity they are pursuing and how long they will pursue this activity, but also which activity they are most likely to pursue afterwards and how long the subsequent activities will approximately, i.e. estimate, take.The activity manager can accordingly predict user behavior ("behavior prediction"), i.e. extend activity recognition to intention recognition.

[0015] In a further embodiment, it is provided that one of the at least one functional unit is a navigation unit. Performing the function of the navigation unit comprises at least partially autonomous control, i.e., semi-autonomous or fully autonomous control, of a driving function of the vehicle according to a predetermined route, in particular longitudinal guidance such as a speed setting. The at least partially autonomous control of the driving function can here take place in particular in the form of controlling a driving function according to SAE levels 1-5 according to J3016 of January 16, 2014, updated as of April 30, 2021, or can comprise such control of a driving function according to the respective SAE level. Accordingly, partially autonomous driving also includes so-called assisted driving within the meaning of the SAE levels.In particular, the driving function of the vehicle can be adapted depending on the detected activity and / or the associated duration and / or the associated mental strain and / or the recovery time of the provided activity data (based on a previous specification, for example, before the activity was detected and / or before the duration / mental strain / recovery time of the activity was estimated) in such a way that the user is permitted to complete the detected activity in accordance with a stored legal or other, for example physiologically conditioned, specification for semi-autonomous control. This can be understood here as the functional unit, in this case the navigation unit, maintaining the boundary conditions within its sphere of influence that permit the user's activity, i.e., are a prerequisite for the user's activity (within specified limits).The user is therefore not requested or forced to interrupt his activity in accordance with the stored specification by carrying out the function, in particular by controlling it at least partially autonomously when following the specified route.

[0016] For example, in the case of autonomous control of a vehicle, for example on a motorway, the driving function of the vehicle can be adapted, i.e., reduced, to the detected activity by adjusting the speed, for example, reading an article or processing emails on a predetermined route, in such a way that the detected activity is expected to be completed, i.e., according to the estimated duration, upon reaching the end of the motorway, i.e., at the end of a section of the route that runs on the motorway. At the end of the motorway, depending on the local conditions (urban traffic) and the properties of the navigation unit (certified for autonomous driving according to SAE Level 3), it could, for example, be legally required that the user resume driving responsibility.Even if this is not the case, and the navigation unit is also authorized to autonomously control the driving functions of the vehicle for a subsequent route section, which, for example, does not run on a motorway but on a winding country road, slowing down on the motorway and then completing the activity, for example, based on another requirement such as a physiological requirement or simply a stored preference of the user, can prevent the onset of nausea, which often occurs when reading on winding roads. This further improves comfort and safety.

[0017] In a further embodiment, it is provided that carrying out the function comprises specifying the route for the at least partially autonomous control of the driving function of the means of transport. Specifying the route also comprises specifying a previously specified route, i.e., adapting a previously specified route, an original route. The specification can be fully automatic or semi-automatic, for example, after the user has released an original route as the then specified route. This is also known from known route guidance systems, in which a user is suggested different route alternatives from which the user can select one, or in which one of the routes is selected automatically, for example, after a specified time has elapsed. In known systems, too, original routes can be specified again, i.e., adapted, for example, to changed traffic conditions.

[0018] The route is specified, in particular, adapted, depending on the detected activity and / or the associated duration and / or mental strain and / or the recovery time of the provided activity data, so that the user is permitted to complete the detected activity in accordance with the stored legal or other requirements for semi-autonomous control. The route can also be adapted to changed activities, for example, if a user does not undertake the planned activity described below or the activity expected by the activity manager, but rather a different one.For example, if the user's activity is detected as processing an email, which is estimated to take a long time, a slower route via a motorway due to traffic jams can be chosen rather than a faster route through a town where, however, fully autonomous driving and thus no email processing is possible due to the increased demands on the vehicle's control. In this sense, an existing or originally taken route can also be redefined, i.e., adjusted, if an activity such as processing an email is detected unexpectedly, i.e., contrary to a prediction or plan. Here, too, comfort and safety in the vehicle are improved.

[0019] In a further embodiment, at least one activity planned by the user, in particular for the time of use of the means of transport (the journey), is recorded with an estimation of the respectively associated time duration and / or the associated mental strain and / or the fallback time for the planned, i.e. expected future activity. The recording is carried out via a recording unit, in particular using the activity database. The recording unit can comprise an input unit for an input from the user and / or an interface unit, for example to access the user's appointment calendar, email program or similar information sources, or can be coupled to the corresponding units. Preferably, the recording unit records the one or more planned activities from a prediction unit which is designed to predict the at least one activity planned by the user.For example, the prediction unit can use an AI-based algorithm that, based on the activity database, makes a prediction for the user's expected activity for the trip, in particular for a specific route and / or specific period of time. The user can also provide a corresponding to-do list of planned activities to the recording unit via the input unit or the interface unit.

[0020] The route is specified depending on the activity planned by the user and / or the respective associated duration and / or the associated mental strain and / or the recovery time. For example, the scenario introduced above can be continued in this way: A user can plan to work on their way to work as an activity. This can be recorded by the recording unit if the user explicitly enters it, for example via a to-do list, or via the prediction unit, which knows, for example, that the user usually works on their way to work on weekdays. Accordingly, a route can be specified from the outset that takes them via the motorway, thus enabling fully automatic control, for example even though this would be slower than an alternative route through a built-up area due to traffic jams.In this scenario, however, the motorway route is preferred and suggested to the user, if not automatically enabled for route guidance, allowing the user to pursue and complete their usual planned activity. This also increases comfort and safety in the vehicle.

[0021] In a further embodiment, specifying the route comprises identifying at least one route section with constant or substantially constant requirements for the user, iethe same or essentially constant boundary conditions for the activity, such as a provisional travel duration assigned to the respective route section, comparing the at least one time duration assigned to the planned activity(ies) with the at least one provisional travel duration, assigning the one or more planned activities to one or more respective route sections on the basis of a comparison result for the time duration and the travel duration (which can be the provisional travel duration or, as explained below, an adjusted travel duration), and displaying the predetermined route with the at least one route section depending on the at least one assigned activity, in particular with an identification of the at least one route section that is specific to the assigned activity.In particular, the predefined route for the at least partially autonomous control can be selected automatically, for example by means of a timer such as for example the fastest route is known, or manually by the user. The constant or essentially constant requirements, i.e. requirements that remain the same apart from a predefined deviation, can, as described above, correspond in particular to a stored legal and / or physiological requirement and preferably relate to a requirement at least specified by the user responsible for driving the vehicle. Alternatively or additionally, the requirements can include a (particularly user-specific) predefined maximum permissible lateral and / or longitudinal acceleration. A lateral and / or longitudinal acceleration to be expected on the respective route section can, for example, be calculated from a known road course using the respective assumed travel speeds.In the example scenario above, the user could be going on a business trip, i.e., not wanting to take the usual route to work in the morning, but instead wanting to drive to another destination, such as a customer. Nevertheless, the user could still plan to check their emails on the way. A route to a customer, for example, provided by conventional route planning, can be specified as the original route and then adapted depending on the planned activity(ies), in this case the planned or expected email processing. The new route may have many distractions in the first section of the route, because the user does not drive onto the nearby motorway as usual, but instead requires a longer journey to another motorway.Accordingly, after identifying the respective route sections - in this case an initial route section with many distractions and a later route section with fewer distractions due, for example, to longer motorway usage - the estimated duration for email processing is compared with the respective travel times on the corresponding route sections and the corresponding user requirements. For example, it may be impossible for a user to check their emails while driving through a town or on a country road because, for example, the navigation unit there is not certified for SAE Level 3 compliant driving. Accordingly, email processing is then assigned to the later route section, and the corresponding route with the later email processing is displayed to the user. The user can therefore see where and when on the planned route they can check their emails.This will further improve safety and comfort in the means of transport.

[0022] In a further embodiment, when assigning the one or more planned activities to one or more respective route sections, a stored urgency of the respective activity, i.e. an explicit or implicit appointment requirement which is linked to the respective activity, for example in the activity database, can be taken into account. In particular, this can be done by assigning more urgent activities to route sections which, within the route specified by the navigation unit, are arranged before route sections which are assigned to less urgent activities. In the above example scenario, for example, the user can plan a business telephone call with high urgency on the day of travel. This is then assigned, for example, to a route section which, without the telephone call, would be assigned to email processing.However, a private phone call without urgency will be assigned to a route section after email processing if the email processing is assigned a higher (but not high) urgency. This also contributes to improved time utilization and clearer focus, thus improving comfort and safety on the train.

[0023] In a further embodiment, when assigning the one or more planned activities to one or more respective route sections, the provisional travel duration of one or more route sections can be adapted to the respective duration of one or more activities if adjusting the provisional travel duration leads to an improved assignment result in terms of a predetermined optimality criterion. This can be done in particular by adjusting a speed on the corresponding route section. For example, if a route section which is suitable for a planned activity except for a deviation in the duration which does not exceed a predetermined limit, can be made suitable, i.e. adapted to the activity, by reducing the speed.In return, for example, a speed can be preferentially increased on another route section, so that the total travel time can remain unchanged. However, even if the total travel time is increased, the result can be improved in terms of the specified optimality criterion, since the planned activity is still carried out and completed on the means of transport, so that the total time spent on the journey plus the activity is less than if the activity were carried out after the journey. For example, adjusting the provisional travel time can also be specified so that this is only carried out for activities of a specified urgency, which is particularly advantageous.

[0024] In the example scenario already used several times, experience has shown that there may be enough time on the journey to work, for example due to traffic jams, to process emails on the motorway section of the route. If, for example, the journey time on the motorway is shortened because there is, for example, no traffic jam, the activity—in this case, processing emails—could be assigned to a different route section. If this is not possible or, for example, not optimal due to other activities planned for the route, the provisional journey time can be adjusted accordingly, for example by reducing the vehicle's speed on the motorway section of the route, thus creating sufficient time to process emails while driving on the motorway.While the total travel time may be increased, the overall experience is improved because the activities planned for the trip can be completed. Overall, time is used more efficiently, effectively saving time. This also contributes to increased relaxation and thus comfort and safety on the train.

[0025] In a further embodiment, the route specification includes specifying at least two different alternative routes, with the identification, comparison, assignment, and display being performed for each of the different alternative routes. In particular, the different alternative routes have a different chronological sequence of the planned activities. Analogous to alternative routes in known systems, where, for example, a choice can be made between routes of different durations, the user can decide whether they want the faster route with email processing at the beginning or the slower route, in which the business phone call is scheduled first.The selection can also be made automatically, for example, based on the detected activity, so that the route matches the detected activity. For example, if the user starts working on emails, the faster alternative route is automatically selected in this example; if they start making a phone call, the slower alternative route is automatically selected. This also leads to a more intuitive assistance function, thus increasing comfort and safety in the vehicle.

[0026] In a further embodiment, the activity data is provided to a plurality of independently functioning functional units. This implements a central activity management system or a central activity manager, through which independently functioning functional units react coherently to the behavior of the user(s). The activity data can be sent and distributed in various ways, e.g. via a multi-broadcast to the entire vehicle, i.e. many or all functional units of the vehicle, or via a publish / subscribe process to selected individual functional units. The CAN bus as well as equivalents or alternatives such as Ethernet can be used as the infrastructure. The respective assistance systems with their functional units can accordingly access the activity data and react to it accordingly.Route planners can use the activity data to adapt driving to the interior activities, both from a comfort perspective (cf. "motion sickness" and similar) and from a safety perspective, particularly level-compliant behavior of the occupants. Interior safety systems can use the activity data, for example, to be able to deploy airbags adaptively; lighting assistance systems can take the respective activities of the occupants into account when changing the lighting; voice-based assistance systems can avoid interrupting ongoing conversations or phone calls between occupants and plan dialogues in advance, for example, when announcements or queries from the navigation unit occur.

[0027] Accordingly, the provision can be made to a functional unit for a voice-based assistance function, which can also be or include the navigation unit, and a voice output function can be the or one of the functions to be performed or performed, with a delay depending on the activity specified in the activity data and / or the associated duration, in order not to interrupt conversations or telephone calls of the occupants. The provision can also be made to at least one functional unit for a safety function, in particular for an interior safety function such as an airbag deployment function, and / or at least one functional unit for a comfort function, in particular an interior lighting function such as a reading light function. All of this contributes to increased comfort and improved safety in the vehicle.

[0028] Alternatively, the activity manager or activity management could not operate centrally, but be installed separately in the various assistance systems, i.e. be installed independently of one another or be instantiated in software and only act as required for the functionality of the respective individual assistance systems, i.e. the proposed method would be executed in independent instances. The navigation unit would thus be a self-contained system that detects and processes the activities of the occupants with the help of the cameras in the vehicle. The other systems, for example functional units for an airbag deployment function, a reading light function and a voice-based assistance function, would do the same. What all systems would have in common, however, is that the respective activities, including the associated duration, are detected or estimated in order to not only check for the presence of activities, but also to be able to plan ahead.

[0029] A further aspect relates to an assistance system, in particular a driver assistance system, for a means of transport, in particular for a motor vehicle. The assistance system comprises a camera unit with one or more cameras for generating image data of an activity of a user of the means of transport in a passenger compartment of the means of transport, as well as a computing unit with one or more computers for detecting the user's activity in the passenger compartment based on the generated image data. The computing unit is designed to estimate a time duration for the detected activity and to provide activity data, which specify the detected activity and the time duration estimated for the detected activity, to at least one functional unit.The functional unit, which may or may not be part of the assistance system, is designed to perform at least one function depending on the activity specified in the activity data and / or the associated duration.

[0030] Advantages and advantageous embodiments of the assistance system correspond to advantages and advantageous embodiments of the described method.

[0031] The features and feature combinations mentioned above in the description, including in the introductory part, as well as the features and feature combinations mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the respective combination specified, but also in other combinations without departing from the scope of the invention. Thus, embodiments are also to be considered encompassed and disclosed by the invention that are not explicitly shown and explained in the figures, but which emerge and can be produced by separate feature combinations from the explained embodiments. Embodiments and feature combinations are also to be considered disclosed that therefore do not have all the features of an originally formulated independent claim.Furthermore, embodiments and combinations of features are to be regarded as disclosed, in particular by the embodiments set out above, which go beyond or deviate from the combinations of features set out in the reliances of the claims.

[0032] Fig. 1 shows an exemplary embodiment of an assistance system in a means of transport. The assistance system 1 is presently installed in a motor vehicle as the means of transport 2. It comprises a camera unit 3 for generating image data of an activity of a user 4 of the means of transport 2 in a passenger compartment 5 of the means of transport 2. Furthermore, the assistance system 1 has a computing unit 6 for detecting the activity of the user in the passenger compartment 5 based on the generated image data. The computing unit 6 is designed to estimate a time duration for the detected activity and to provide activity data which specify the detected activity and the time duration estimated for the detected activity to at least one functional unit 7. The further functional unit 7 is designed to perform a function depending on the activity specified in the activity data and the associated duration.In the present case, the functional unit 7 is designed as a navigation unit and is coupled to a steering unit 8 for steering the vehicle 2, a drive unit 10 for driving the vehicle 2, a braking unit 11 for braking the vehicle 2, and an interactive display unit 9 for displaying a route and receiving input from the user 2. The computing unit 6 is additionally coupled to a storage unit 12, which is designed to store the detected activity with the estimated and / or a recorded associated time duration in an activity database. Accordingly, the estimation of the time duration is carried out based on identical and / or similar activities according to a predetermined similarity criterion and the time duration stored for the identical or similar activity in the storage unit 12.The assessment of similarity can, for example, be carried out in a class-based manner, i.e. based on a learned or trained algorithm.

[0033] The assistance system thus serves as an activity manager, which is able to recognize for the user(s) 4, i.e. occupants of the means of transport 2, which activity they are pursuing, and preferably also which subsequent activity they are most likely to pursue after this and approximately how long the respective activities will last. Optionally, the activity manager can also be used to assign a to-do list of the user 2, in particular a driver, to the most suitable route or route sections. This is particularly relevant for truck drivers and other drivers who are supervised by a dispatcher. When planning and assigning the activities to the respective route sections, the level of automation possible on the respective route section, i.e.The SAE level possible according to the means of transport 2 or the navigation unit must be taken into account, as well as, for example, the road surface condition and / or dynamic driving requirements of the route, such as the frequency and / or tightness of curves. In principle, the assistance system 1 shown thus enables the inclusion of a to-do list in the route planning for completing the activities listed on the to-do list during the journey. This can be displayed, for example, on a route plan of the navigation unit, similar to how traffic jams are displayed previously.

[0034] The Activity Manager can provide interfaces to all common vehicle systems, such as CAN bus, Ethernet, and the like, to act as a central element for recording the activities of the user(s) 4 and making them available to all other connected systems with their corresponding functional units. This enables a (partially) automated vehicle to adapt the route to the current state of the user and plan ahead, for example, to avoid motion sickness. The navigation unit can synchronize route sections with the activities of the user(s) 4 in the interior 5 and adjust the route if necessary. Furthermore, the navigation unit can compare route sections with specifications for SAE-level-compliant behavior and check whether all users, especially the driver, are adhering to these specifications.For example, it may be planned that the (partially) automated vehicle 2 should assume driving responsibility on a specific route section. However, the route section requires a certain behavior from user 2 so that they behave in accordance with the automation level and the transfer of driving responsibility to motor vehicle 2 is permitted. If user 4 does not behave accordingly, the navigation unit or its control function can intervene, warn the user in a timely manner, or select an alternative route.

[0035] The goal here is to establish a match between the activities of user 4 and the requirements of motor vehicle 2 and route. The recognition of the activity and the comparison with level-compliant behavior can also be used for safety purposes in, for example, automated taxis, Uber services, automated long-distance transport, or similar. Due to the storage unit 12, the activity manager can also save the recognized activities and, if necessary, provide a list of past activities for each user 4. This can, for example, be advantageous for a voice assistance function, which could, for example, proactively address user 4 when approaching a rest area and suggest that they take a break after a journey of a predetermined minimum length. However, recently, i.e.If it were detected within a specified period of time that the driver had slept and / or eaten, such a suggestion would only be irritating, which is why this can be excluded with a corresponding setting for the voice assistance function. This also makes it possible to create an overview in which each detected activity can be shown as a proportion of the total time, such as the journey time. For example, the activity data might show that user 4 spent 20% of their driving time with a passenger or interacting with an infotainment system for 10% of the driving time.

[0036] The basis of the approach presented here is therefore the automated recognition of user activities, preferably all users of a means of transport, as well as the creation of correlated durations of the respective activities for planning in the respective case of need of the assistance system and, if necessary, an additional necessary prediction of the subsequent activity or several probable subsequent activities. The recognition of activities is preferably carried out centrally for all users 4, ideally stored in a corresponding activity database and transferred via appropriate interfaces to the

[0037] Functional units of various interior assistance systems communicate. This leads to improved comfort and safety in the vehicle.

Claims

Claims 1. A method for operating an assistance system (1) in a means of transport (2), in particular in a motor vehicle, comprising the following method steps: - camera-based recognition of an activity of a user (4) of the means of transport (2) in a passenger compartment (5) of the means of transport (2) based on image data from a camera unit (3), by a computing unit (6); - estimating a time duration for the detected activity by the computing unit (6); - Providing activity data specifying the detected activity and the estimated duration of the detected activity to at least one functional unit (7); - performing at least one function by the functional unit (7) depending on the duration associated with a specified activity in the activity data.

2. Method according to the preceding claim, wherein the time duration corresponds to a length of a period in which the user pursues the recognized activity in accordance with a respective prediction.

3. Method according to claim 1 or 2, characterized by a - storing the detected activity with the estimated and / or a recorded associated time duration in an activity database, by a storage unit (12); wherein - the estimation of the duration is based on the same and / or similar activities according to a given similarity criterion and the duration stored for the same or similar activity.

4. Method according to one of the preceding claims, characterized in that - one of the at least one further functional unit (7) is a navigation unit; and - the execution of the function comprises an at least partially autonomous control of a driving function of the means of transport (2) according to a predetermined route, in particular a longitudinal guidance such as a speed specification; wherein in particular - the driving function of the means of transport (2) is adapted depending on the detected activity and / or the associated duration of the provided activity data in such a way that the user (4) is allowed to complete the detected activity in accordance with a stored specification for the semi-autonomous control.

5. Method according to the preceding claim, characterized in that - performing the function comprises specifying the route for the at least partially autonomous control of the driving function of the means of transport (2); and - the route is specified depending on the recognized activity and / or the associated duration of the provided activity data, in particular is adapted, so that the user (4) is allowed to complete the recognized activity in accordance with the stored specification for the semi-autonomous control.

6. Method according to the preceding claim, characterized by a - Recording of at least one activity planned by the user (4) with an estimation of the respective associated duration, in particular using the activity database, by a recording unit; preferably with a prediction of the at least one activity planned by the user (4), in particular using the activity database; and with a - Specifying the route depending on the activity planned by the user (4) and / or the respective associated duration.

7. Method according to the preceding claim, characterized in that specifying the route comprises: - identifying at least one route section with constant or essentially constant requirements for the user (4) and a provisional travel time assigned to the respective route section; - comparing the duration allocated to at least one of the planned activities with the at least one provisional travel duration; - Assigning the one or more planned activities to one or more respective route sections based on a comparison result for the time duration and travel time; - Displaying the specified route with the at least one route section depending on the at least one assigned activity, in particular with an identifier of the at least one route section that is specific to the assigned activity.

8. Method according to the preceding claim, characterized in that when assigning the one or more planned activities to one or more respective route sections, a stored urgency of the respective activity is taken into account, in particular by assigning more urgent activities to route sections which are arranged within the route before route sections which are assigned to less urgent activities.

9. Method according to one of the two preceding claims, characterized in that when assigning the one or more planned activities to one or more respective route sections, the provisional travel duration of one or more route sections is adapted to the respective duration of one or more respective activities if the adaptation of the provisional travel duration leads to an improved result of the assignment in the sense of a predetermined optimality criterion.

10. Method according to one of the three preceding claims, characterized in that the specification of the route comprises a specification of at least two alternative routes which differ from one another, wherein the identification, the comparison, the assignment and the display are carried out for the alternative routes in each case, in particular the alternative routes have a different chronological sequence of the planned activities.

11. Method according to one of the preceding claims, characterized in that the provision is made to several functional units (7) functioning independently of one another.

12. Method according to one of the preceding claims, characterized in that the provision to a functional unit (7) for a voice-based assistance function is carried out and a voice output function is carried out with a delay depending on the activity specified in the activity data and the associated duration.

13. Method according to one of the preceding claims, characterized in that the provision is made to at least one functional unit (7) for a safety function, in particular for an interior safety function such as an airbag triggering function, and / or to at least one functional unit (7) for a comfort function, in particular an interior lighting function such as a reading light function.

14. Assistance system (1) for a means of transport (2), in particular for a motor vehicle, comprising: - a camera unit (3) for generating image data of an activity of a user (4) of the means of transport (2) in a passenger compartment (5) of the means of transport (2); - a computing unit (6) for detecting the activity of the user (4) in the passenger compartment (5) based on the generated image data; characterized in that - the computing unit (6) is designed to estimate a time duration for the detected activity and to provide activity data, which specify the detected activity and the time duration estimated for the detected activity, to at least one further functional unit (7).

15. Assistance system (1) according to the preceding claim, characterized by the further functional unit (7) which is designed to perform a function depending on the activity specified in the activity data and the associated duration.

16. Means of transport (2), in particular a motor vehicle, with an assistance system (1) according to one of the two preceding claims.

Images