PREDICT-BASED DRIVING

Abstract

Method for operating weak artificial intelligence (AI) agents for at least partially autonomous driving, comprising: (a) receiving multi-domain information about elements relating to a vehicle by multiple perception modules, wherein each of the multiple perception modules is associated with a dedicated domain of the multi-domain information; (b) generating class signatures by the multiple perception modules that indicate classes of the elements of the multi-domain information; (c) determining a multi-domain identifier that identifies the generated class signatures of the multiple perception modules; and (d) identifying, based on the multi-domain identifier, one or more weak AI agents relevant for processing at least a portion of the multi-domain information.The identification triggers the execution of further processing of at least some of the multi-domain information by the identified weak AI agent(s) in order to make one or more weak AI driving-related decisions.

Description

BACKGROUND

[0001] Autonomous vehicles include a perception system that performs semantic decomposition (SD). There, the AV flow consists of three modules: the perception module, which forwards its output to the route planning module, which forwards its output to the actuation module (brake / acceleration / steering, etc.).

[0002] Since the route planning and actuation modules don't have access to raw sensor inputs, the role of perception in the SD architecture is to fully label the raw inputs from the vehicle's sensors, such as detecting cars and pedestrians, segmenting the free drivable space, detecting speed limits from road signs, and anything else in the scene that could potentially affect driving in that scene. Because so many objects / entities in the scene must be identified and accurately located, the SD perception module tends to be a complicated, computationally intensive model. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The embodiments of the disclosure will be better understood and appreciated from the following detailed description taken in conjunction with the drawings in which: Fig. 1 illustrates an example of a system; Fig. 2 illustrates an example of a system; Fig. 3 illustrates an example of a method; and Fig. 4 an example of a step of the process of Fig. 3 illustrated; Fig. 5A illustrates an example of a method; Fig. 5B illustrates an example of an implementation of the method; Fig. 5C illustrates an example of an image; Fig. 5D illustrates an example of a vehicle; Fig. 5E illustrates an example of an image; Fig. 5F illustrates an example of an image; Fig. 5G illustrates an example of an image; Fig. 6A illustrates an example of an image; Fig. Figure 6B illustrates an example of a prediction; Fig. 6C illustrates an example of different units; Fig. 7A illustrates an example of a unit; Fig. Figure 7B illustrates an example of an MDI statistic; Fig. 7C illustrates an example of a method; and Fig. 7D illustrates an example of various units. DESCRIPTION OF EMBODIMENTS

[0004] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, processes, and components have not been described in detail in order not to obscure the present invention.

[0005] The inventive subject matter is particularly pointed out and distinctly claimed in the concluding portion of the specification. However, the invention may best be understood by reference to the following detailed description, when read in conjunction with the accompanying drawings, as to both organization and operation, together with objects, features, and advantages.

[0006] For simplicity and clarity, the elements depicted in the figures are not necessarily drawn to scale. For example, the dimensions of some elements may be exaggerated compared to others for clarity. Furthermore, where appropriate, reference numerals may be repeated in the figures to indicate corresponding or analogous elements.

[0007] Since the illustrated embodiments of the present invention can largely be implemented with electronic components and circuits known to those skilled in the art, details will not be set forth in greater detail than is deemed necessary for an understanding and appreciation of the underlying concepts of the present invention, and in order not to obscure or detract from the teachings of the present invention.

[0008] Any reference in the description to a method should be applied mutatis mutandis to a device or system capable of carrying out the method and / or to a non-transitory computer-readable medium on which instructions for carrying out the method are stored.

[0009] Any reference in the specification to a system or device should be applied mutatis mutandis to a method capable of being performed by the system and / or may be applied mutatis mutandis to a non-transitory computer-readable medium on which instructions executable by the system are stored.

[0010] Any reference in the specification to a non-transitory computer-readable medium should be applied mutatis mutandis to a device or system capable of executing instructions stored in the non-transitory computer-readable medium and / or may be applied mutatis mutandis to a method of executing the instructions.

[0011] Any combination of modules or units listed in any of the figures, any part of the description and / or any of the claims may be provided.

[0012] Each of perception units, weak AI agents, driving decision units may be implemented in hardware and / or code, instructions and / or commands stored in a non-transitory computer-readable medium, may be contained in a vehicle, outside a vehicle, in a mobile device, in a server, and the like.

[0013] The vehicle can be any type of vehicle, i.e. a land vehicle, an aircraft and a watercraft.

[0014] The description and / or the drawings may refer to an image. An image is an example of a media unit. Any reference to an image may be applied mutatis mutandis to a media unit. A media unit may be an example of sensed information. Any reference to a media unit may be applied mutatis mutandis to any type of natural signal, such as a signal generated by nature, a signal representing human behavior, a signal representing stock market events, a medical signal, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, and the like. Any reference to a media unit may be applied mutatis mutandis to the sensed information. The sensed information may be of any type and may be sensed by any type of sensors, such as:a visual light camera, an audio sensor, a sensor capable of sensing infrared, radar images, ultrasound, electro-optics, radiography, LIDAR (Light Detection and Ranging), etc. Sensing may involve generating samples (e.g., pixels, audio signals) that represent the signal that was transmitted or otherwise reached the sensor.

[0015] The description and / or drawings may refer to a stress element. A stress element may be implemented in software or hardware. Different stress elements of a given iteration are configured to apply different mathematical functions to the inputs they receive. Non-limiting examples of the mathematical functions include filtering, although other functions may also be applied.

[0016] The description and / or drawings may refer to a conceptual structure. A conceptual structure may contain one or more clusters. Each cluster may contain signatures and associated metadata. Any reference to one or more clusters may be applicable to a reference to a conceptual structure.

[0017] The description and / or drawings may refer to a processor. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), fully customized integrated circuits, etc., or a combination of such integrated circuits.

[0018] Any combination of steps of a method illustrated in the description and / or drawings may be provided.

[0019] Any combination of subject-matter from any of the claims may be provided.

[0020] Any combination of the systems, units, components, processors, sensors shown in the description and / or drawings may be provided.

[0021] Any reference to an object can also be applied to a pattern. Accordingly, any reference to an object recognition is also applicable, mutatis mutandis, to a pattern recognition.

[0022] A situation can be a singular location or combination of properties at a specific point in time. A scenario is a series of events that logically follow one another within a causal framework. Any reference to a scenario should be applied mutatis mutandis to a situation.

[0023] The captured piece of information can be captured by one or more sensors of one or more types. The one or more sensors can belong to the same device or system—or to different devices or systems.

[0024] A perception unit may be provided, which may be preceded by one or more sensors and / or one or more interfaces for receiving one or more detected information units. The perception unit may be configured to receive a detected information unit from an I / O interface and / or from a sensor. The perception unit may be followed by several weak AI agents, also referred to as an ensemble of weak AI agents.

[0025] An artificial intelligence agent (AI agent) can refer to an autonomous entity that acts on an environment by directing its activity toward achieving goals (i.e., it is an agent) by observing the environment using sensors and taking action in response (i.e., it is intelligent). Intelligent agents can also learn or use knowledge to achieve their goals. They can be very simple or very complex. A reflexive machine, such as a thermostat, is considered an example of an intelligent agent (www.wikipedia.org).

[0026] A captured unit of information may, but does not have to, be processed before reaching the perceptual unit. Processing may be accomplished through filtering, noise reduction, and the like.

[0027] The number of weak AI agents can exceed 100, 500, 1,000, 10,000, 100,000, etc. A larger number of weak AI agents can enable more accurate driving decisions.

[0028] A weak AI agent is weak in the sense that it is not trained to respond to all possible (or all likely, or a majority of) scenarios that should be handled by the entire ensemble. For example, each AI agent may be trained to respond to a fraction (e.g., less than 1 percent) of the scenarios managed by the entire ensemble. A weak AI agent may be trained to respond only to some of the factors, elements, parameters, or variables that make up a scenario.

[0029] The weak AI agents may have the same complexity and / or the same parameters (depth, energy consumption, technology implementation), but at least some of the weak AI agents may differ from each other by at least one of the complexity and / or parameters.

[0030] The weak AI agents can be trained in a supervised manner and / or in an unsupervised manner.

[0031] One or more weak AI agents can be a neural network or different from a neural network.

[0032] The ensemble may include one or more sensors and any other unit for generating a sensed information unit and / or may receive (via an interface) one or more sensed information units from the one or more sensors.

[0033] The perception unit can process the one or more acquired information units and determine which weak AI agents are relevant for processing the one or more acquired information units.

[0034] An autonomous vehicle system can be deployed that can use the perception unit to classify the observed scene into several coarse-grained categories. The system can comprise an ensemble of weak-Ki agents (EoN).

[0035] The perception unit can receive and / or generate anchors that, once detected (by the perception unit), can influence the selection of weak AI agents. The number of anchors can be very large (e.g., over 100, 500, 1,000, 10,000, 20,000, 50,000, 100,000 anchors, and even more).

[0036] For a given scenario (which may be represented by one or more captured information units, such as one or more images), the perception unit may detect one or more anchors.

[0037] The detected anchors can provide sufficient contextual clues for the perceiving unit to determine which are the relevant weak Kl agents.

[0038] The contextual cue may be a high-level context of the acquired information unit. It is high-level in the sense that determining the contextual cue is less complex and / or requires fewer computational resources than performing object detection of a small object in a acquired information unit. A small object may have a minimal size to be detected, e.g., a size of a few tens of pixels, a size smaller than 0.1, 0.5, 1, 2, or 3 percent of the acquired information unit, and the like. For example, determining the contextual cue need not involve determining the precise positions of every object in the image—including the positions of objects that appear as a few tens of pixels in an image.

[0039] By searching for the context of a high-level information unit, the power consumption of the perception unit can be much lower (for example, even up to two orders of magnitude lower) than the power consumption of a state-of-the-art system designed to perform the entire process of object detection and determining which driving action to perform.

[0040] At least part of the energy savings can be attributed to the fact that the context of the high-level captured information item does not contain location information, that it is not necessary to determine whether objects of different sizes are the same type of objects, and the like.

[0041] A weak AI agent can receive inputs directly from the sensors (e.g., as output from the perception module) and provide as output a suggested behavior (desired vehicle control parameters—and desired vehicle behavior—also called driving decisions), e.g., angle of a steering wheel, acceleration / braking signal, or control of another aspect of driving, etc.

[0042] The outputs of the various selected weak AI agents are forwarded to a driving decision unit (also called a coordinator), which issues one or more driving decisions, such as one or more commands, requests, or recommendations, to various modules of a vehicle and / or to a driver.

[0043] The coordinator may use any method to generate one or more output driving decisions, such as one or more commands and requests, based on the outputs of the various selected weak AI agents.

[0044] These methods may include arbitration, competition, selecting a response based on a risk imposed by accepting an output from a weak AI agent, and the like.

[0045] Returning to the perceptual unit, non-limiting examples of anchors are listed below: • Approaching a pedestrian crossing. Detecting a pedestrian on the sidewalk triggers different behaviors depending on the proximity of a crosswalk. If the pedestrian is standing on the sidewalk next to a crosswalk, the car should slow down (in case the pedestrian begins to cross). In other areas, the same pedestrian on the sidewalk can be safely ignored. Therefore, different weak agents may be required to cross a crosswalk. • Approaching a pothole on a road. This may cause a car to slow down or change lanes. • Limited visibility and slippery road conditions may require a slower speed and / or a greater safety distance from the vehicle in front. • Encrypted navigation signal - When crossing an intersection, the weak agent should receive a signal indicating whether we should cross the intersection straight ahead, turn left, or turn right. This anchor / signal changes the meaning of the term "stay in lane." • Lane change - this anchor triggers a specific weak agent that is responsible for changing lanes to the left or right. • Approaching road junctions with different traffic signs – do I give way or do I have priority? • Presence of an unsafe driver nearby: tailgating, erratic driving, signs of road rage may cause you to move into the right lane, slow down, and let the dangerous car pass. • A police car is following you with its blue lights flashing: Move into the right lane and stop on the hard shoulder. • An oncoming car makes an emergency stop: A weak agent is activated, which either makes an emergency stop or swerves onto a hard shoulder and stops there with less deceleration.

[0046] Anchors can be selected, generated, and / or learned in various ways—manually, automatically, based on human labeling of inputs, based on autonomous labeling of inputs, based on manual identification, based on scenario labeling (e.g., approaching a roundabout, approaching a pedestrian crossing, environmental conditions, presence of a road user)—in which case, the corresponding desired behavior can be inferred from the recorded behavior of the human driver in that situation, identifying unknown or poorly predicted patterns of human driving. These patterns may indicate a state of the world not well covered by existing weak agents, necessitating the assignment of a new agent.

[0047] A weak AI agent can be or include a simple model (e.g., a neural network) that receives raw (or preprocessed in any way) sensor data as input, processes it internally, and outputs a suggested behavior.

[0048] Examples of weak AI agents are listed below: • Weak AI agent for "staying in lane" steering behavior when not at an intersection. The weak AI agent can be a convolutional neural network that maps an image of the road captured by a front-facing camera to a vehicle's steering angle. • Weak AI agent for the "stay in lane" steering behavior at a left-turn intersection. Note that "stay in lane" has a different meaning at an intersection when we are going straight or turning left. The weak agent for the left side receives images from the frontal and left-facing cameras and converts them into a steering angle. Depending on the definition, the information indicating that an agent is turning left instead of going straight is either received as input from the navigation block or encoded as an "anchor." This weak AI agent can be implemented as a CNN. • Weak AI agent implementing the "left lane change" behavior. This weak AI agent is given access to a left-facing camera and suggests actions necessary for the lane change, including deciding when to change lanes, accelerating before and during the lane change, and steering to execute the lane change. • Weak AI agent that steers when a vehicle is stopped at a red light. This weak AI agent is implemented as an "if" condition and simply keeps the steering angle constant. • Weak AI agent implementing the “keep distance” logic: This weak AI agent maps the images captured by the front and rear facing cameras to the acceleration / deceleration signal. • Weak AI agent implementing "give way at pedestrian crossing" logic: This weak AI agent stops when a pedestrian is in a crosswalk or slows down when the pedestrian is nearby but not in the crossing. It uses data from the front camera and maps it to the braking / acceleration signal.

[0049] Examples of how the coordinator can process output from one or more weak AI agents: • Combine independently. • The weak AI agent responsible for the steering angle that allows the vehicle to stay in its lane is largely independent of driving speed. Therefore, the behavior proposed by this agent can be independently combined with the acceleration / braking behavior of the weak "keep distance" agent. • Overwrite. • The vehicle should stay in its lane, but the car in front of us has suddenly stopped, and there isn't enough stopping distance to stop. In this case, the desire to avoid a collision takes precedence over staying in our lane, and we pull onto the shoulder and stop there. • Animal on the road - giving up lane keeping and evasive maneuvers. • “Modulate” • On slippery roads, the behaviors “stay in lane” and “keep distance” remain almost unchanged, but require slower speeds and greater safety distances. • optimize risk reduction.

[0050] The system is expected to be superior to SD systems in all aspects, be it behavioral accuracy, model size and complexity, or computational intensity.

[0051] The perception unit can operate more efficiently than conventional perception systems. Conventional perception systems analyze the scene completely, aiming to reconstruct it—that is, focusing on all the details of the scene, such as the precise analysis of all actors on the road (pedestrians, cars, bicycles, lanes, traffic signs, traffic lights, trees, obstacles, etc.) and their attributes (exact position, pose, distance, angle, etc.). This is a very energy- and resource-intensive process, as its purpose is to create a comprehensive environment model as input to the strategy module.

[0052] A perception unit doesn't need to focus on all the details and identify each agent—it only needs to analyze and classify the scene to activate the appropriate agent. This is a very lightweight process in terms of performance. Once activated, the relevant agent will analyze the relevant features of the input image.

[0053] It should be noted that even with a large number of weak AI agents, some can be stored in RAM and others in non-volatile memory (e.g., on hard disks, in cheap non-volatile memories, and the like) and can be retrieved when needed.

[0054] The perception module can be very fast, as it is tasked with detecting anchors and selecting the relevant weak AI agents based on them. Each weak AI agent can also be very fast, but we have a very large number of them. Fortunately, only a few of them actually need to be running at any given time, meaning the runtime at any given time is very short.

[0055] The proposed units can be executed or hosted by a processor. The processor can be a processing circuit. The processing circuit can be implemented as a central processing unit (CPU) and / or one or more other integrated circuits such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), fully customized integrated circuits, etc., or a combination of such integrated circuits.

[0056] Fig. 1 shows an example of a system 10.

[0057] The system 10 comprises a perception unit such as the perception router 30 (which receives a captured information unit which is an image 8 with pixels), an ensemble 40 of weak AI agents (for example, weak AI agents associated with a roundabout, with a pedestrian crossing a crosswalk, with a specific traffic sign or with a traffic jam) and a driving decision unit such as the coordinator 50. The driving decision unit 50 can control and / or communicate with a reaction unit 60.

[0058] In Fig. Figure 2 shows an example of a system 10'.

[0059] The system 10' includes a sensing unit 20' (for receiving one or more sensed information units 15), a perception unit 30', weak AI agents 40(1)-40(K), where K is the number of weak AI agents, and a driving decision unit 50'. The driving decision unit can control and / or communicate with a reaction unit 60', such as a brake control system, a vehicle computer, an autonomous driving module, an ADAS driving module, and the like.

[0060] Fig. 3 shows the method 300 for operating an ensemble of weak AI agents with respect to a vehicle.

[0061] In relation to a vehicle, the output of method 300 may be one or more driving decisions that, if implemented, may affect the driving of the vehicle.

[0062] The method may include various steps, some of which may involve providing desired driving decisions (e.g., during training of any of the entities used during method 300, where the entities may include a perception unit, weak AI agents, and a driving decision unit).

[0063] Additionally or alternatively, driving decisions associated with one of the captured information units fed to one of the entities may be provided, and the method may include determining which driving decisions were correct. For example, this may be determined using statistics—for example, by assuming the most frequent driving decision per situation and / or scenario or an equation thereof.

[0064] The method 300 may begin with an initialization step 310.

[0065] Step 310 may be obtaining a perception unit, weak AI agents, and a driving decision unit configured to perform various steps of method 300.

[0066] Obtaining may include receiving after being trained, and / or training, and / or receiving at any stage of the training process, downloading instructions, or otherwise configuring a computerized system to perform another step of method 300.

[0067] Step 310 may include at least one of the following steps: (a) training at least one of the perception unit, the weak AI agents, and the driving decision unit, (b) receiving at least one of the already trained perception unit, the weak AI agents, and the driving decision unit, (c) otherwise configuring at least one of the perception unit, the weak AI agents, and the driving decision unit.

[0068] Step 310 may be followed by step 320, in which one or more captured information units are obtained.

[0069] Obtaining may include sensing, receiving without sensing, preprocessing, and the like.

[0070] Step 320 may be followed by a step 330 in which a perception unit determines, based on the one or more acquired information units, one or more relevant weak AI agents of the ensemble that may be relevant for processing the one or more acquired information units. The entire ensemble may be relevant for a first plurality of scenarios.

[0071] Each relevant weak AI agent may be relevant to a specific class. The class may be associated with an anchor. Step 330 may involve searching for the anchor.

[0072] Each class may be defined by at least a portion of one or more scenarios, where the at least a portion of the one or more scenarios may be a fraction of the first plurality of scenarios.

[0073] The various weak AI agents can be trained to respond to various scenarios, which may be (or may include) a T-intersection, various road elements, a crosswalk, a roundabout, obstacles, various environmental conditions (rain, fog, night, a straight highway, an incline, a traffic jam, etc.). Examples of various obstacles and / or various road elements are presented in PCT patent application WO2020 / 079508, entitled "METHOD AND SYSTEM FOR OBSTACLE DETECTION," which is hereby incorporated in its entirety.

[0074] The different scenarios can be different situations or different from situations.

[0075] For example, a scenario may be at least one of the following: (a) a location of the vehicle, (b) one or more weather conditions, (c) one or more contextual parameters, (d) a road condition, (e) a traffic parameter.

[0076] Various examples of road conditions may include the roughness of the road, the state of maintenance of the road, the presence of potholes or other obstacles on the road, and whether the road is slippery, covered with snow or other particles.

[0077] Various examples of a traffic parameter and the one or more contextual parameters may be the time (hour, day, period or year, specific hours on specific days, and the like), a traffic load, a distribution of vehicles on the road, the behavior of one or more vehicles (aggressive, calm, predictable, unpredictable, and the like), the presence of pedestrians near the road, the presence of pedestrians near the vehicle, the presence of pedestrians away from the vehicle, the behavior of pedestrians (aggressive, calm, predictable, unpredictable, and the like), the presence near the road, the presence of pedestrians near the vehicle, the presence of pedestrians away from the vehicle, the behavior of pedestrians (aggressive, calm, predictable, unpredictable, and the like), the risk associated with driving in a vehicular environment,The complexity of driving a vehicle, the presence (near the vehicle) of at least one of the following: kindergarten, school, crowd, etc. A contextual parameter may refer to the context of the captured information—the context may depend on or relate to the circumstances that frame an event, statement, or idea.

[0078] A relevant weak AI agent can be trained to respond to one or more situations from a very large number of situations. Examples of situations and situation-based processing are presented in U.S. patent application 16 / 035,732, which is incorporated herein by reference.

[0079] Each class can be defined by an anchor, which can be a contextual clue. Step 320 can be performed without detecting objects that may be located below a predefined number of pixels. The predefined number of pixels does not exceed a few tens of pixels.

[0080] The weak Kl agents can be consistently weak Kl agents.

[0081] For at least some of the weak AI agents, the respective share may be less than one percent of the first set of scenarios.

[0082] Step 330 may be followed by step 340, in which the one or more captured information units are sent to the relevant weak AI agents.

[0083] It should be noted that once the relevant weak AI agent(s) are determined, they may be uploaded to a processor and / or storage device. This reduces the RAM or other storage resources required for storing and executing step 350.

[0084] Step 340 (or method 300) may include maintaining at least one irrelevant weak AI agent in a low-power mode (idle, inactivated, sleep, partially operational, and the like) in which the power consumption of the at least one irrelevant weak AI agent may be lower than the power consumption of a relevant weak AI agent.

[0085] Step 340 may be followed by step 350, in which the one or more acquired information units are processed by the one or more relevant weak AI agents to make one or more driving decisions for the weak AI agent. Each weak AI agent may be relevant to a corresponding fraction of the first plurality of scenarios managed by the entire ensemble.

[0086] A weak AI agent driving decision can be a command, a request, or a recommendation for autonomous control of a vehicle.

[0087] A weak AI agent driving decision can be an ADAS command, an ADAS request, or an ADAS recommendation.

[0088] Step 350 may be followed by a step 360 in which one or more weak AI agent driving decisions are processed by a driving decision unit to provide an output driving decision.

[0089] Step 360 may include averaging the one or more weak AI agent driving decisions or applying one or more features (e.g., predefined and / or learned and / or modified over time) and / or applying one or more policies to the one or more weak AI agent driving decisions. An example of a strategy attempting to reduce risk—for example, in the case of two weak AI agents traveling at different speeds—is selecting the lower speed.

[0090] The method 300, in particular steps 320-360, may be repeated multiple times—for example, multiple times per second, once per second, once every few seconds, continuously, or discontinuously.

[0091] In Fig. 4 shows an example of step 310.

[0092] Step 310 may include step 312 in which the perception unit is trained to classify the acquired information units into classes.

[0093] Each class may be at least a part of one or more scenarios, where the one or more scenarios may be a fraction of the first plurality of scenarios.

[0094] Each class can be assigned an anchor. The anchor can be used to assign a captured piece of information to the class.

[0095] Step 312 may involve the perception unit receiving a definition of at least some of the classes prior to training. This may include, for example, receiving labels or other class-defining information.

[0096] Step 312 may include defining at least some of the classes by the perception unit.

[0097] Defining can include, for example, generating signatures and clustering the signatures into conceptual structures such as clusters. Clustering virtually defines the classes.

[0098] The clusters can all belong to the same level or be arranged hierarchically. Clustering can respond to the statistics of contextual cues—more frequently occurring contexts can be divided into more clusters. Larger clusters can be divided into lower-level clusters in any way—for example, by cross-correlation between cluster members, by finding common signature parts and unique signature parts, and the like.

[0099] Step 312 may include performing unsupervised training.

[0100] At least one part of one or more scenarios may be at least one of (a) one or more factors of a scenario, (b) one or more elements of a scenario, (c) one or more parameters of a scenario, and (d) one or more variables of a scenario.

[0101] Step 312 may include feeding the perception unit with a first data set of detected information units.

[0102] Step 312 may be followed by step 314, in which the trained perception unit is used to classify detected information units of a second data set. These detected information units are referred to as second detected information units.

[0103] Step 314 may also include entering one or more driving decisions per class.

[0104] Each class includes several second recorded information units and one or more driving decisions.

[0105] Each weak AI agent can be assigned to a specific class. Step 314 can be followed by a step 316 in which each weak AI agent is trained to output a driving decision of the weak AI agent that is assigned to the corresponding class.

[0106] The training of step 316 may include feeding a weak AI agent associated with a particular class with second sensed information and one or more driving decisions of the class.

[0107] Step 316 may be performed in a supervised or unsupervised manner. Supervised training may involve providing one or more driving decisions as a requested output of the weak AI agent.

[0108] Step 316 may be followed by a step 318 in which the driving decision unit is trained to provide an initial driving decision based on one or more driving decisions of the weak AI agent.

[0109] Step 318 may include feeding sensed information units of the third data set (hereinafter third sensed information units) to the perception unit, enabling the perception unit to determine the relevant weak AI classes (based on the classes of the third sensed information units), enabling the relevant weak AI classes to output driving decisions of the weak AI agent, and feeding the driving decision unit with the driving decisions of the weak AI agent and the driving decisions associated with the third data set.

[0110] Each of the first, second, and third data sets may contain any number of captured information units generated in any way. They may contain randomly selected captured information units or any combination of captured information units.

[0111] A non-transitory computer-readable medium may be provided having stored instructions for operating an ensemble of weak AI agents, wherein operating may comprise obtaining one or more sensed pieces of information; determining, by a sensing unit and based on the one or more sensed pieces of information, one or more relevant weak AI agents of the ensemble that are relevant for processing the one or more sensed pieces of information; wherein the ensemble is relevant to a first plurality of scenarios; processing, by the one or more relevant weak AI agents, the one or more sensed pieces of information to provide one or more driving decisions of the weak AI agent; wherein each weak AI agent is relevant to a respective fraction of the first plurality of scenarios;and processing the one or more driving decisions of the weak AI agent by a driving decision unit to provide an output driving decision;

[0112] A system for determining driving decisions, the system comprising an ensemble of weak AI agents; a perception unit configured to: receive one or more acquired pieces of information; and, based on the one or more acquired pieces of information, determine one or more relevant weak AI agents of the ensemble relevant to processing the one or more acquired pieces of information; wherein the ensemble is relevant to a first plurality of scenarios; wherein the one or more relevant weak AI agents are configured to process the one or more acquired pieces of information to provide one or more driving decisions of the weak AI agent; wherein each weak AI agent is relevant to a respective portion of the first plurality of scenarios;and a driving decision unit configured to process the one or more driving decisions of the weak AI agent to provide an output driving decision.; Use of multiple perception modules

[0113] The elements that influence a vehicle can come from different domains. The inventors have found that assigning perception modules per domain instead of using a single perception unit offers several technical advantages, such as: • Reduction of the computational and / or memory resources allocated for the selection of weak AI agents, since the perception modules are overall smaller than a single perception unit used to manage all information domains. • Reduction of computational and / or memory resources for training - because each perception module is trained with much more compact information and has to cover far fewer classification scenarios. • More accurate selection of weak AI agents – a perception module assigned to a domain can be better tailored to the selection in that domain. This eliminates the need for the perception module to compromise between different information domains. • Simplifying adaptation to a new information domain - which requires training or adding an already trained perception module for the new information domain. • Dynamic change of the selection of perception modules (from a group of perception modules) when the information areas change. • Maintaining the group's unselected perception modules in an idle state - thereby reducing power consumption. • Retrieving information about the selected perception modules - while preventing the retrieval of information about non-selected perception modules - which reduces the power consumption and / or memory resources allocated for the selection of the weak AI agents.

[0114] Fig. 5A shows an example of a method 500 for operating weak artificial intelligence (AI) agents for at least partially autonomous driving.

[0115] According to one embodiment, the method 500 comprises the step 510 of automatically selecting a plurality of perception modules from a group of perception modules.

[0116] According to one embodiment, the selection is made based on domains associated with the multi-domain information.

[0117] According to one embodiment, the selection is based on a previously generated output of a perception module in the group of perception modules. For example, if—assumed—there is insufficient light (this can be determined by image processing, location, and time information—e.g., driving on an unlit road in an unlit environment during a moonless night), a perception module based on image processing will not be activated.

[0118] According to one embodiment, at least two different sensing modules are connected to two different vehicle sensors. The selection can be based on an operating state of the two different vehicle sensors. For example, assuming a vehicle sensor malfunctions or generates unreliable measurement data, a sensing module connected to that vehicle sensor can be deselected, kept in idle mode, and the like.

[0119] According to one embodiment, step 510 may include or follow maintaining sensing modules of the group that are different from the plurality of sensing modules in an idle mode.

[0120] According to one embodiment, the plurality of perception modules are a plurality of perception routers. According to one embodiment, the plurality of perception modules are a plurality of perception subrouters connected to a perception router. Examples of a perception router are shown in U.S. patent application Ser. No. 18 / 036,150, which is incorporated herein by reference.

[0121] According to one embodiment, step 510 is followed by step 520 of receiving multi-domain information about elements related to a vehicle by a plurality of perception modules. Each of the plurality of perception modules is associated with a specific domain of the multi-domain information.

[0122] According to one embodiment, step 520 is followed by step 530 of generating class signatures by the multiple perception modules that indicate classes of the elements of the multi-domain information.

[0123] According to one embodiment, a perception module is a classifier configured to classify information of the domain associated with the perception module. The classifier may implement a machine learning process, may be a deep neural network (DNN), may be a neural network other than a DNN, may be different from a neural network, and the like.

[0124] According to one embodiment, step 530 is followed by step 540 of determining a multi-domain identifier (MDI) that identifies the generated class signatures of the multiple perception modules. The MDI may be a concatenation of the class signatures or identify the generated class signatures in another way.

[0125] According to one embodiment, step 540 is followed by step 550, in which, based on the MDI, one or more weak AI agents are identified that are relevant for processing at least a portion of the MDI.

[0126] According to one embodiment, the one or more weak AI agents are associated with clusters of MDIs. For example, a first weak AI agent may be associated with a first set of one or more MDI clusters, and a second weak AI agent may be associated with a second set of one or more MDI clusters. The number of MDIs in the first set may be the same as the number of MDIs in the second set. The number of MDIs in the first set may be different from the number of MDIs in the second set. The first set and the second set may share one or more MDIs. The first set and the second set may not contain any common MDIs.

[0127] A weak AI agent or a cluster of MDIs can be associated with a skill. The selection of one or more weak AI agents can virtually involve the selection of one or more skills required to handle a situation or scene.

[0128] According to one embodiment, the multi-domain information is input by at least one of the following elements: • Road setting information relating to static objects in the vehicle's surroundings. Examples of static objects include road elements (lanes, curves, intersections, roundabouts), static elements other than road elements (houses, trees), and location information (city, countryside, urban environment, non-urban environment). • Road user information relating to moving road users in the surrounding area. Examples of moving road users include pedestrians and other vehicles. The road user information may include information about the location and / or behavior of road users. • Traffic rule display information related to visual traffic rule indicators in the surrounding area. Traffic rule indicators may include traffic signs, content written on the road, and traffic rule indicators (e.g., lane markings, crosswalks, arrows indicating the permitted direction of travel, text, symbols, and the like). • Regulatory information relating to legal restrictions related to the environment. Legal restrictions may include maximum speed limits, minimum speed limits, and the like. Regulatory information may be obtained from location-based regulatory information or other sources. • Information on environmental conditions relating to at least one of the elements weather and ambient light conditions. • Vehicle status information relating to a movement-related status of the vehicle.

[0129] According to one embodiment, the plurality of perception modules may comprise at least one of the following modules: • A road attitude perception module. The road attitude perception module may be configured to generate a class signature indicating one or more classes of one or more static objects in an environment of the vehicle. The class signature may indicate a scenario that includes all static objects that (i) may affect the vehicle and (ii) are located in the environment of the vehicle. • A road user perception module. The road user perception module may be configured to generate a class signature representative of a class of moving road users. The class signature may indicate a scenario encompassing all moving road users that (i) may impact the vehicle and (ii) are in the vehicle's vicinity. • A traffic rule perception module. The traffic rule perception module may be configured to generate a class signature indicating a class of traffic rules indicated by traffic signs or other elements captured in the image. The class signature may indicate a scenario containing all traffic rules that (i) may affect the vehicle and (ii) are indicated by traffic rule indicators located in the vehicle's environment. • A rule perception module. The rule perception module may be configured to generate a class signature representative of a class of legal restrictions. The class signature may indicate a scenario containing all legal restrictions that (i) may affect the vehicle and (ii) are applicable to the vehicle's environment. • An environmental condition perception module. The environmental condition perception module may be configured to generate a class signature indicating a class of environmental conditions in the vehicle's environment, such as at least one of environmental weather and ambient lighting conditions. • A vehicle state perception module. The vehicle state module may be configured to generate a class signature indicative of a class of vehicle states—in particular, a motion-related state of the vehicle. The motion-related state may include kinematic information, velocity information, acceleration information, and the like.

[0130] Note that any other perception modules may be provided, and that one or more of these perception modules may be selected at any time. There may be any number of domains and any number of perception modules associated with the domains.

[0131] According to one embodiment, the identification of step 550 triggers the execution of further processing of at least a portion of the multi-domain information by the identified weak AI agent(s) to make one or more weak AI driving-related decisions.

[0132] Fig. Figure 5B shows an example of different information sources, different perception modules, an MDI generator, a selection unit, weak AI agents, and a driving decision unit.

[0133] The various sources of information include: • Information 581(1) from one or more vehicle sensors about an environment of the vehicle. • Information 581(2) from one or more vehicle condition sensors. • Environmental information 581(3). • Legal Restrictions Database 581(4).

[0134] The perception modules include: • A module for perceiving road conditions – referred to as RS Perception Module 582(1). • A module for the perception of road users – referred to as RU Perception Module 582(2). • A module for perceiving traffic rules – referred to as TR Perception Module 582(3). • A module for the perception of regulations - referred to as REG Perception Module 582(4). • A module for perceiving environmental conditions - referred to as AMB Perception Module 582(5). • A module for perceiving the vehicle condition - referred to as VS Perception Module 582(6).

[0135] In Fig. 5B also shows an unselected perception module 582(6) which may be idle.

[0136] The selection unit 587 identifies one or more weak AI agents by comparing the MDI 585 with the MDI cluster signatures 586(1)-586(J) to find one or more matching MDI clusters associated with one or more weak AI agents to be selected.

[0137] The matching MDI clusters indicate the weak AI agents to be selected (labeled 588). Fig. Figure 5B also shows unselected weak Kl agents (indicated by an inner pattern with a dashed line).

[0138] The selected weak AI agents 588 output weak AI agent driving decisions 589.

[0139] A driving decision unit 590 is configured to receive the weak AI agent driving decisions 589 and generate and output an output driving decision 591 that can be executed by one or more units of the vehicle.

[0140] Fig. 5C shows an example of an image 610 captured by a vehicle sensor.

[0141] Image 610 shows two lanes 611 and 612 of a road, a right-hand building 613, a first vehicle 615, a second vehicle 616, a tree 618, a table 617, a left-hand building 621, a child 623, a ball 624, a traffic sign 630 indicating a maximum speed (e.g., 50 km / h), a direction arrow 634, a zebra crossing 633, and a crowd 628 about to reach the zebra crossing and cross it from right to left. Fig. also shows a progress (behavior) 641 of the child 623, a progress 642 of the crowd 628, a progress 645 of the first vehicle 615 and a progress 646 of the second vehicle 616.

[0142] Road user information may indicate moving road users such as the first vehicle 615, the second vehicle 616, the child 623, and the crowd 628.

[0143] Road setting information may point to static objects selected from lanes 611 and 612, road 610, tree 618, table 617, right building 613, and left building 621 - not all of these static objects can be considered because not all of these static objects affect the vehicle.

[0144] The information on traffic rules is a reference to traffic sign 630, direction arrow 634 and lane markings.

[0145] Information about the ambient conditions may indicate that the area surrounding the vehicle is illuminated by sunlight and not obscured by rain or other weather conditions.

[0146] Vehicle condition information can be obtained from information not contained in the image.

[0147] Control information can be obtained from information not contained in the image.

[0148] Fig. 5D shows an example of a vehicle 700. The vehicle 700 includes a vehicle transmitting unit 710, which may include one or more sensors such as vehicle sensors 712 and 714. The vehicle also includes one or more processing circuits labeled 720, a memory unit 730, a communication unit 740, and one or more vehicle units such as one or more vehicle computers, units controlled by the one or more vehicle units, engine units, chassis, wheels, and the like.

[0149] The one or more processing circuits are configured to perform method 600.

[0150] Fig. 5E shows an example of a method 800.

[0151] According to one embodiment, method 800 includes steps 510, 520, 530, 540, and 550.

[0152] According to one embodiment, method 800 comprises the step 860 of triggering further processing of the at least a portion of the multi-domain information by the one or more identified weak AI agents to make one or more weak AI driving-related decisions.

[0153] According to one embodiment, the method 800 comprises the step 870 of further processing at least a portion of the multi-domain information by the identified weak AI agent(s) to make one or more weak AI driving-related decisions.

[0154] According to one embodiment, method 800 includes step 360, in which one or more weak AI agent driving decisions are processed by a driving decision unit to provide an output driving decision.

[0155] It should be noted that method 800 may include only one or two of steps 860, 870, and 360.

[0156] In Fig. 5F shows an example of method 900.

[0157] According to one embodiment, method 900 includes the step 910 of receiving information from the one or more identified weak AI agents. The information may be the multidimensional information provided to the plurality of perception modules, one or more segments of the multidimensional information provided to the plurality of perception modules, or any representation or metadata about at least a portion of the plurality of perception modules.

[0158] According to one embodiment, the one or more identified weak KI agents were identified by performing method 500.

[0159] According to one embodiment, step 910 is followed by step 920 of processing information by the identified weak AI agent(s) to provide one or more weak AI driving-related decisions to be sent to a driving decision unit. The one or more weak AI driving-related decisions may trigger the generation of one or more output driving decisions by the driving decision unit. The one or more output driving decisions may be one or more commands, requests, or recommendations to various modules of a vehicle and / or to a driver.

[0160] Fig. 5G shows an example of method 1000.

[0161] According to one embodiment, the method 1000 comprises the step 1010 of receiving one or more weak AI driving-related decisions from one or more identified weak AI agents by a driving decision unit.

[0162] According to one embodiment, step 1010 is followed by step 1020 of determining one or more output driving decisions by the driving decision unit. The one or more output driving decisions may be one or more commands, requests, or recommendations to various modules of a vehicle and / or to a driver.

[0163] A non-transitory computer-readable medium may be provided for operating weak artificial intelligence (AI) agents for at least partially autonomous driving, the non-transitory computer-readable medium having stored instructions for: receiving, by a plurality of perception modules, multi-domain information about elements relating to a vehicle; wherein each of the plurality of perception modules is associated with a dedicated domain of the multi-domain information; generating, by the plurality of perception modules, class signatures indicating classes of the elements of the multi-domain information; determining a multi-domain identifier identifying the generated class signatures of the plurality of perception modules; and identifying one or more weak AI agents relevant to processing at least a portion of the multi-domain information based on the multi-domain identifier.The identification triggers the execution of further processing of at least a portion of the multi-domain information by the identified weak AI agent(s) to make one or more weak AI driving-related decisions.

[0164] According to one embodiment, the plurality of perception modules are a plurality of perception routers.

[0165] According to one embodiment, the plurality of perception modules are a plurality of perception sub-routers connected to a perception router.

[0166] The non-transitory computer-readable medium stores instructions for automatically selecting the plurality of perception modules from a group of perception modules.

[0167] According to one embodiment, the selection is based on domains associated with the multi-domain information.

[0168] According to one embodiment, the selection is based on a previously generated output of a perception module of the group of perception modules.

[0169] According to one embodiment, at least two different perception modules are connected to two different vehicle sensors, wherein the selection is based on an operating state of the two different vehicle sensors.

[0170] The non-transitory computer-readable medium stores instructions to maintain perception modules of the group that are different from the plurality of perception modules in a rest state.

[0171] According to one embodiment, the multi-area information is input from one of the following: road setting information relating to static objects in an environment of the vehicle; road user information relating to moving road users in the environment; traffic rule display information relating to visual traffic rule displays in the environment; regulation information relating to legal restrictions relating to the environment; environmental condition information relating to at least one of weather and light conditions of the environment; and vehicle state information relating to a movement-related state of the vehicle. forecast

[0172] The introduction of prediction in conjunction with the use of perception modules associated with a domain has been shown to offer several technical advantages, such as: • Prediction improves the management of memory units (e.g., but not limited to, cache memory units), for example by increasing the accuracy of pre-fetch operations, thereby reducing the latency associated with memory misses and reducing memory consumption - since fewer fetch operations are required. • Prediction, at least in most cases, reduces the urgency of processing and acting on the acquired information and can reduce the operating speed of the processing resources allocated to the execution of perception modules and / or AI agents. Reducing the operating speed reduces energy consumption and extends the lifetime of the processing resources. • Prediction can enable smoother driving, which is beneficial for various vehicle components (e.g., from a mechanical perspective). Furthermore, some smoother driving behaviors can be represented by fewer coefficients, resulting in a reduction in computational resources and memory requirements. • Predictions can enable better determination of the vehicle's future progress - for a longer period in advance - thereby reducing computational and / or memory resources required for more iterations of the short-term determination of the vehicle's progress.

[0173] Fig. 6A shows an example of a method 1100 for managing a group of weak artificial intelligence (AI) agents for at least partially autonomous driving.

[0174] According to one embodiment, the method 1100 comprises the step 1110 of obtaining, by a prediction circuit, a stream of metadata segments generated at multiple times and associated with a selection of one or more subsets of the group of weak AI agents.

[0175] According to one embodiment, the metadata segments are selected weak AI agent identifiers.

[0176] According to one embodiment, the metadata segments are multiple pieces of multi-domain information (MDIs). The multiple MDIs are indicative of multiple instances of multi-domain information about elements related to a vehicle with respect to the multiple points in time (PITs). An MDI generated at a particular point in time of the multiple points in time is a combination of class signatures indicating classes of pieces of multi-domain information associated with the particular point in time. The metadata segments may be generated using method 500.

[0177] According to one embodiment, step 1110 is followed by step 1120 of finding a segment of the data stream by the prediction circuit that is a predictor of receiving a next cluster identifier at a future time.

[0178] Step 1120 may include any prediction method - including, but not limited to, a machine learning prediction method, a non-machine learning method, the application of a Kalman filter, the application of any prediction filter, and the like.

[0179] According to one embodiment, step 1120 is followed by step 1130 of automatic prediction if the predictor is found to be at least one of the following: • Future metadata segments received during the future time. • A future subset of weak AI agents to be selected at the future time.

[0180] According to one embodiment, there is a known association between the metadata segments and one or more selected weak AI agents, since a given metadata segment causes a selection of one or more associated weak AI agents.

[0181] According to one embodiment, the prediction triggers a reaction to the prediction.

[0182] According to one embodiment, the method 1100 may include one or more additional steps (collectively designated 1140) consisting of: • Triggering the reaction to the prediction. • Respond to the forecast. • Triggering a response that includes predicting a next state of the vehicle at the next time. • Triggering a response that includes determining a future progress of the vehicle based on one or more next states of the vehicle at one or more next points in time. • Determining a future progress of the vehicle based on one or more next states of the vehicle at one or more next points in time. This may involve smoothing the future progress of the vehicle. The smoothing is compared to a future progress of the vehicle without the prediction. • Triggering a determination of a response to the next state of the vehicle. • Determine a response to the next state of the vehicle. • Evaluation of the accuracy of the predictor. • Prediction, based on the predictor, of selected perception modules from multiple perception modules to be used during the future time point. • Triggering the execution of a driving-related process at a time that does not exceed the future time. The driving-related process may involve changing the speed of the vehicle. Method in which the driving-related process involves changing the direction of travel of the vehicle.

[0183] Fig. Figure 6B is an example of a stream of metadata segments generated at multiple points in time and associated with a selection of one or more subsets of the group of weak AI agents. In Fig. 6B, it is assumed that the metadata segments are multiple MDIs labeled 1180(1) 1180(10) associated with different points in time (PITs) 1182(1) - 1182(10).

[0184] In Fig. 6B it is assumed that: • A first predictor containing values ​​of a substream comprising the MDIs 1182(1)-1182(5) indicates that at a future PIT (e.g., PIT 1182(11)) an MDI will be output that has a value corresponding to the MDI 1182(6). • A second predictor containing values ​​of a substream comprising MDI 1182(3), MDI 1182(6) and MDI 1182(9) indicates that at another future PIT (e.g. PIT 1182(12)) an MDI will be output whose value is equal to MDI 1182(7).

[0185] The first and second predictors can be used to at least: • Maintain the values ​​of MDI 1182(6) and MDI 1182(7) in a cache memory - at least until PIT 1182(11). • Determining the vehicle state at PITs 1182(10) and 1182(11) - and determining the desired vehicle state at the PITs preceding PITs 1182(21) and 1182(22).

[0186] In this case, the movement of the vehicle can be smoothed to avoid extensive changes in direction and / or speed in PIT 1182(9) and the like.

[0187] According to one embodiment, the selection of weak AI agents can be modified to take the predictions into account. For example, the multiple prediction modules referenced in the figures.

[0188] Fig. 6C shows an example of: • Perception modules 582. • MDI Generator 584. • Selection unit 587. • Weak AI agents 588. • Driving decision unit 590. • Forecast unit 1150. • Memory management unit 1152. • Processing Management Unit 1153. • First storage unit 1154. • Second storage unit 1155. • Processing units 1154(1) - 1154(K). • Perception modules metadata 1158. • Predictive driving decision unit 590-1. • Non-predictive driving decision unit 590-2.

[0189] The second memory unit 1155 is smaller than the first memory unit 1154—and for simplicity, it is assumed to be a cache memory. There may be more than two memory units.

[0190] The prediction may affect the driving decision issued by the driving decision unit and / or may affect a memory management unit 1152 responsible for managing one or more memory units used for the at least partially autonomous driving, and / or may affect a processing management unit 1153 configured to manage processing resources, such as processing resources used for implementing the perception modules and / or the selection unit and / or the weak AI agents and the like.

[0191] According to one embodiment, the processing units 1154(1)-1154(K) are processing circuits or parts of processing circuits.

[0192] According to one embodiment, the processing units 1154(1)-1154(K) are configured (e.g., programmed or hardware-configured) to implement (using data and / or metadata stored internally by the processing units and / or stored in one or more of the storage units) the perception modules 582, the MDI generator 584, the selection unit 587, the weak AI agents 588, the driving decision unit 590, the prediction unit 1150, the memory management unit 1152, and the processing management unit 1153.

[0193] According to one embodiment, the at least two processing units 1154(1)-1154(K) may differ from each other by at least one parameter selected from the amount of computing resources, power consumption, complexity, and the like. The selection of which processing unit to activate represents a compromise between the various parameters of the at least two different processing units.

[0194] When executing a perception module, it may be necessary to provide metadata of the perception module (such as coefficients of a neural network model) and information (such as information of a domain associated with the perception module) to a processing unit and / or to an associated storage unit (for example, an internal storage unit of the processing units or the second storage unit).

[0195] According to one embodiment, the prediction 1151 allows to control the retrieval of information and / or perception metadata in an efficient manner - for example, by at least one of the following measures: (a) predicting prefetch operations that result in cache hits, (b) determining an assignment of perception modules to processing units that can reduce the number of configurations of the processing circuits - since it is advantageous to assign the same processing unit to implement the same perception module during consecutive operating cycles.

[0196] According to one embodiment, the prediction may help reduce the power consumption of the processing units by idling unused processing units by reducing the clock frequency of a processing module implementing a unit that must output a result within a few cycles, rather than outputting the result every cycle.

[0197] According to one embodiment, the driving decision unit may include a prediction-based driving decision unit 590-1 and a non-prediction-based driving decision unit 590-2—the former associated with driving decisions in the absence of a prediction and the latter associated with driving decisions in the presence of a prediction—where the units may be selectively enabled or disabled depending on the presence or absence of a prediction. The prediction-based driving decision unit 590-1 may operate at a lower clock frequency—or be otherwise configured to operate at lower power—compared to the non-prediction-based driving decision unit 590-2. Localized driving

[0198] The environment of a vehicle at another location may be location-dependent in the sense that the statistical distribution of elements that can affect a vehicle at one location differs from the statistical distribution of elements that can affect the vehicle at another location.

[0199] Predicting the environment the vehicle will encounter on different paths along an expected vehicle path can be very effective and offer various technical advantages, such as: • Prediction improves the management of memory units (such as, but not limited to, cache memory units) - e.g., by increasing the accuracy of pre-fetch operations, thereby reducing the latency associated with memory misses and reducing memory consumption - since fewer fetch operations are required. • Prediction, at least in most cases, reduces the urgency of processing and acting on perceived information and can reduce the operating speed of the processing resources allocated to executing perception modules and / or AI agents. Reducing the operating speed reduces energy consumption and extends the lifetime of processing resources. • Prediction can enable smoother driving, which is beneficial for various vehicle components (e.g., from a mechanical perspective). Furthermore, some smoother driving behaviors may be represented by fewer coefficients, resulting in a reduction in computational and potentially memory resources. • Predictions can enable better determination of the vehicle's future progress - for a longer period in advance - thereby reducing computational and / or memory resources required for repeated short-term determination of the vehicle's progress.

[0200] Fig. 7A shows an example of an urban environment comprising multiple buildings 1202 forming a road network. An expected local path of the vehicle (expected path of the vehicle within a small environment—for example, within a distance of 10-1000 meters from the vehicle) 1205 includes traveling west, avoiding a roundabout 1201-1, continuing west, turning north, continuing north, turning west, and traveling west.

[0201] Fig. 7A also shows various locations (1202-1 through 1202-5 and 1202-11 through 1202-15) associated with MDI statistics. The MDIs of each location are indicative of elements affecting a vehicle at that location. The MDIs may be generated in any manner—including the manner illustrated in the text above—for example, by method 500.

[0202] In the example of Fig. 7A, locations 1201-1 through 1202-5 are path-related locations, identified as such by a processing circuit based on the expected local path 1205 and information about the locations. Locations 1202-11 through 1202-15 are classified as irrelevant—for example, they are far from the expected local path or belong to road segments not included in the expected local path.

[0203] Fig. 7B is an example of a first-place MDI statistic 1210-1, a second-place MDI statistic 1210-2, a third-place MDI statistic 1210-3, a fourth-place MDI statistic 1210-4, and a fifth-place MDI statistic 1210-5.

[0204] In the example of Fig. 7B, these MDI statistics include the most popular MDIs of each location—the most popular MDIs of the first location 1210-1, the most popular MDIs of the second location 1210-2, the most popular MDIs of the third location 1210-3, the most popular MDIs of the fourth location 1210-4, and the most popular MDIs of the fifth location 1210-5.

[0205] The most popular MDIs of a location can be MDIs that occur at least a certain number of times at the location, or have a probability of occurrence of at least a predefined probability (for example, over 10, 15, 20, 25, 30, 40 percent, and the like), or the Xth most popular MDIs - X can be determined in any way - for example, most 5, 10, 15, 20, 25, 30 MDIs, and the like.

[0206] According to one embodiment, there may be one or more instances of the most popular MDIs—for one or more constraints. For example, the constraints may relate to different hours, to night or day, to different dates, to different seasons, to different weather conditions, and the like. See, for example, the two instances 1211-1-1 and 1211-2.

[0207] Fig. 7C shows an example of the method 1300 for localized driving.

[0208] According to one embodiment, the method 1300 includes the step 1310 of obtaining information about locations associated with statistics from multi-domain identifications (MDIs), wherein the MDIs of each location indicate elements related to a vehicle at the location.

[0209] According to one embodiment, the method 1300 includes the step 1320 of obtaining an expected local path of a vehicle.

[0210] According to one embodiment, step 1310 and step 1320 are followed by step 1330 of identifying path-related locations by a processing circuit based on the expected local path and the information about the locations.

[0211] According to one embodiment, step 1330 is followed by step 1340, in which the processing circuitry determines expected local path MDI statistics for use in at least partially autonomously driving a vehicle through the expected local path.

[0212] According to one embodiment, step 1340 includes identifying the most popular MDIs per path-related location of the path-related locations.

[0213] According to one embodiment, determining the expected local path MDIs statistics triggers a determination of at least partially autonomous driving of the vehicle through the expected local path.

[0214] According to one embodiment, step 1340 is followed by step 1350 of responding to the expected local path MDIs statistics.

[0215] Step 1350 may include at least one of the following steps: • Triggering a determination of at least partially autonomous driving of the vehicle through the expected local path. • Triggering an execution of the at least partially autonomous driving of the vehicle through the expected local path. • Predictions, based on the expected local path MDIs statistics, of selected perception modules from multiple perception modules to be used during future time points in conjunction with the expected local path and with respect to the path-related points. • Prefetching the selected perception modules into a cache. • Determination of at least partially autonomous driving of the vehicle through the expected local path. • Carrying out at least partially autonomous driving of the vehicle through the expected local route.

[0216] According to one embodiment, the determination of the at least partially autonomous driving of the vehicle through the expected local path can fulfill at least one of the following tasks: • Modify the expected local path. • Be executed without changing the expected local path. • Modify at least one driving parameter (e.g., speed and / or acceleration and / or direction of travel) based on the most popular MDIs along the path. For example, an MDI may indicate an obstacle at a specific location, and the driving parameter can be adjusted to avoid the obstacle or travel over it at a reduced speed, etc. If the vehicle is expected to be close to the obstacle at a specific time (PIT), the driving parameters can be modified in advance, even before the specific PIT. • Selection of perception modules capable of dealing with situations indicated by the expected local path MDIs statistics. • Selection of weak Cl modules capable of handling situations indicated by the expected local path MDIs statistics.

[0217] Fig. 7D shows an example of: • Perception modules 582. • MDI Generator 584. • Selection unit 587. • Weak AI agents 588. • Driving decision unit 590. • Forecast unit 1150. • Memory management unit 1152. • Processing Management Unit 1153. • First storage unit 1154. • Second storage unit 1155. • Processing units 1154(1) - 1154(K). • Perception modules metadata 1158. • Statistics of locations and MDIs 1380. • Expected local path provider 1381. • Identification of route-related locations 1382.

[0218] The second memory unit 1155 is smaller than the first memory unit 1154, and for simplicity, it is assumed to be a cache memory. There may be more than two memory units.

[0219] The prediction (using the MDI statistics of the path-related locations) may impact the driving decision issued by the driving decision unit and / or may impact a memory management unit 1152 responsible for managing one or more memory units used for the at least partially autonomous driving, and / or may impact a processing management unit 1153 configured to manage processing resources, such as processing resources used for implementing the perception modules and / or the selection unit and / or the weak AI agents and the like.

[0220] According to one embodiment, the processing units 1154(1)-1154(K) are processing circuits or parts of processing circuits.

[0221] According to one embodiment, the processing units 1154(1)-1154(K) are configured (e.g., programmed or hardware-configured) to control (using data and / or metadata stored internally by the processing units and / or stored in one or more of the storage units) the perception modules 582, the perception modules 582, the MDI generator 584, the selection unit 587, the weak AI agents 588, the driving decision unit 590, the prediction unit 1150, the memory management unit 1152, the processing management unit 1153, the location and MDI statistics 1380, the expected local path provider 1381, and the path-related location identifier 1382.

[0222] According to one embodiment, the at least two processing units 1154(1)-1154(K) may differ from each other by at least one parameter selected from the amount of computing resources, power consumption, complexity, and the like. The selection of which processing unit to activate represents a compromise between the various parameters of the at least two different processing units.

[0223] When executing a perception module, it may be necessary to provide metadata of the perception module (e.g., coefficients of a neural network model) and information (e.g., information about a domain associated with the perception module) to a processing unit and / or to an associated storage unit (e.g., an internal storage unit of the processing units or the second storage unit).

[0224] According to one embodiment, the prediction 1151 allows to control the retrieval of information and / or perception metadata in an efficient manner - for example, by at least one of the following measures: (a) predicting prefetch operations that result in cache hits, (b) determining an assignment of perception modules to processing units that can reduce the number of configurations of the processing circuits - since it is advantageous to assign the same processing unit to implement the same perception module during consecutive operating cycles.

[0225] According to one embodiment, the prediction may help reduce the power consumption of the processing units by idling unused processing units by reducing the clock frequency of a processing module implementing a unit that must output a result within a few cycles, rather than outputting the result every cycle.

[0226] The Place and MDI Statistics 1380 stores information about (a) places associated with the MDI statistics and (b) the MDI statistics.

[0227] The expected local path provider 1381 is configured to provide the expected local path. The expected local path provider 1381 may determine the expected local path in any manner or receive the expected local path. The expected local path provider 1381 may be a navigation device, an ADAS device, an AV device, and the like.

[0228] The path-related location identifier 1382 may be configured to perform step 1330.

[0229] In the foregoing specification, the invention has been described with reference to specific embodiments of the invention. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0230] Furthermore, the terms "front," "rear," "top," "bottom," "above," "below," and the like, if any, are used in the specification and claims for descriptive purposes and not necessarily to describe permanent relative positions. It is understood that the terms so used are interchangeable under appropriate circumstances, so that, for example, the embodiments of the invention described herein are capable of operating in orientations other than those depicted or otherwise described herein.

[0231] Additionally, the terms "assert" or "assign" and "negate" (or "deactivate" or "clear") are used herein to refer to transitioning a signal, status bit, or similar device to its logically true or logically false state, respectively. If the logically true state is a logical level of one, the logically false state is a logical level of zero. And if the logically true state is a logical level of zero, the logically false state is a logical level of one.

[0232] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and that alternative embodiments may merge logic blocks or circuit elements, or impose a different decomposition of functionality on different logic blocks or circuit elements. It is therefore to be understood that the architectures presented herein are merely exemplary, and that, in fact, many other architectures may be implemented that achieve the same functionality.

[0233] Any arrangement of components intended to achieve the same functionality is effectively "connected" to achieve the desired functionality. Therefore, any two components combined here to achieve a particular functionality can be considered "connected" to achieve the desired functionality, regardless of architectures or intermedia components. Likewise, any two components connected in this manner can be considered "operably connected" or "operably coupled" to achieve the desired functionality.

[0234] Furthermore, those skilled in the art will recognize that the boundaries between the operations described above are merely illustrative. The various operations may be combined into a single operation, a single operation may be distributed among additional operations, and the operations may be performed at least partially overlapping in time. Furthermore, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments.

[0235] In one embodiment, the illustrated examples may also be implemented as circuits on a single integrated circuit or in a single device. Alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices that are suitably interconnected.

[0236] However, other modifications, variations, and alternatives are possible. Accordingly, the specifications and drawings are to be understood as illustrative rather than restrictive.

[0237] In the claims, the reference signs placed in parentheses are not to be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps other than those recited in a claim. Furthermore, the terms "a" or "an" as used herein are defined as one or more than one. Also, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be construed to mean that the introduction of a further claim element by the indefinite articles "a" or "an" limits a particular claim containing such introduced claim element to inventions containing only one such element, even if the same claim contains the introductory phrases "one or more" or "at least one" and indefinite articles such as "a" or "an". The same applies to the use of definite articles.Unless otherwise indicated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements they describe. These terms are therefore not necessarily intended to indicate a temporal or other prioritization of these elements. The mere fact that certain measures are listed in different claims does not mean that a combination of those measures cannot be advantageous.

[0238] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will occur to those skilled in the art. It is therefore intended that the appended claims are intended to cover all modifications and changes that fall within the true spirit of the invention.

[0239] It is recognized that various features of the embodiments of the disclosure that are described in conjunction with separate embodiments for clarity may also be provided in combination in a single embodiment. Conversely, various features of the embodiments of the disclosure that are described in conjunction with a single embodiment for brevity may also be provided separately or in any suitable subcombination.

[0240] Those skilled in the art will understand that the embodiments of the disclosure are not limited by what has been particularly shown and described herein. Rather, the scope of the embodiments of the disclosure is defined by the appended claims and their equivalents. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] WO 2020 / 079508

[0073] US 16 / 035,732

[0078] US 18 / 036,150

[0120]

Claims

[1] A method for operating weak artificial intelligence (AI) agents for at least partially autonomous driving, the method comprising: Receiving multi-domain information about elements relating to a vehicle by a plurality of perception modules; wherein each of the plurality of perception modules is associated with a particular domain of the multi-domain information; generating class signatures by the multiple perception modules that indicate classes of the elements of the multi-domain information; Determining a multi-domain identifier that identifies the generated class signatures of the multiple perception modules; and Identifying, based on the multi-domain identifier, one or more weak AI agents relevant to processing at least part of the multi-domain information; wherein identifying triggers execution of further processing of at least a portion of the multi-domain information by the one or more identified weak AI agents to provide one or more weak AI driving-related decisions. [2] The method of claim 1, wherein the plurality of perception modules are a plurality of perception routers. [3] The method of claim 1, wherein the plurality of perception modules are a plurality of perception subrouters connected to a perception router. [4] The method of claim 1, further comprising automatically selecting the plurality of perception modules from a group of perception modules. [5] The method of claim 4, wherein the selection is based on domains associated with the multi-domain information. [6] The method of claim 4, wherein the selection is based on a previously generated output of a perception module of the group of perception modules. [7] The method of claim 4, wherein at least two different sensing modules are connected to two different vehicle sensors, the selection being based on an operating state of the two different vehicle sensors. [8] The method of claim 4, further comprising maintaining perception modules of the group that are different from the plurality of perception modules in an idle mode. [9] The method of claim 1, wherein the multi-domain information is input from a group comprising: road situation information relating to static objects in an environment of the vehicle; road user information relating to moving road users in the environment; traffic rule display information relating to visual traffic rule displays in the environment; regulatory information relating to legal restrictions related to the environment; environmental condition information relating to at least weather and light conditions of the environment; and vehicle state information relating to a movement-related state of the vehicle. [10] The method of claim 9, further comprising generating a class signature by the road user perception module that indicates a behavior class of the moving road users. [11] The method of claim 1, further comprising triggering further processing of at least a portion of the multi-domain information by the one or more weak AI agents to make one or more narrow AI driving-related decisions. [12] The method of claim 1, further comprising connecting to the one or more identified weak AI agents. [13] A non-transitory computer-readable medium for operating weak artificial intelligence (AI) agents for at least partially autonomous driving, wherein the non-transitory computer-readable medium has stored instructions for: Receiving multi-domain information about elements relating to a vehicle by a plurality of perception modules; wherein each of the plurality of perception modules is associated with a particular domain of the multi-domain information; generating class signatures by the multiple perception modules that indicate classes of the elements of the multi-domain information; Determining a multi-domain identifier that identifies the generated class signatures of the multiple perception modules; and Identifying, based on the multi-domain identifier, one or more weak AI agents relevant to processing at least a portion of the multi-domain identifier; and wherein identifying triggers execution of further processing of at least a portion of the multi-domain identifier by the identified one or more weak AI agents to provide one or more weak AI driving-related decisions. [14] The non-transitory computer-readable medium of claim 13, wherein the plurality of perception modules are a plurality of perception routers. [15] The non-transitory computer-readable medium of claim 13, wherein the plurality of perception modules are a plurality of perception sub-routers connected to a perception router. [16] The non-transitory computer-readable medium of claim 13, further comprising automatically selecting the plurality of perception modules from a group of perception modules. [17] The non-transitory computer-readable medium of claim 16, wherein the selection is based on domains associated with the multi-domain identifier. [18] The non-transitory computer-readable medium of claim 16, wherein the selection is based on a previously generated output of a perception module of the group of perception modules. [19] The non-transitory computer-readable medium of claim 16, wherein at least two different sensing modules are connected to two different vehicle sensors, the selection being based on an operating state of the two different vehicle sensors. [20] The non-transitory computer-readable medium of claim 13, wherein the multi-domain information is input by a group comprising: Road situation information relating to static objects in a vehicle's surroundings; road user information relating to moving road users in the surroundings; Traffic rule display information relating to visual traffic rule displays in the surrounding area; rule information relating to legal restrictions in the surrounding area; Environmental condition information relating to at least one of the weather and lighting conditions of the environment; and Vehicle status information relating to a movement-related status of the vehicle.

Images