Perception- and prediction-based driving

Abstract

To provide a method and non-transitory computer-readable medium for managing a group of narrow artificial intelligence (AI) agents for autonomous driving.SOLUTION: The method includes obtaining 1110, by a prediction circuit, a stream of metadata segments generated at multiple points in time and associated with a selection of one or more sub-groups of the group of narrow AI agents; where the metadata segments are selected out of selected narrow AI agent identifiers and multiple multi-domain identifiers; finding 1120, by the prediction circuit, a segment of the stream that is a predictor of receiving a next cluster identifier at a future point in time; and automatically predicting 1130, when finding the predictor, future metadata segments to be received during the future point of time and / or a future sub-group of narrow AI agents to be selected at the future point of time.SELECTED DRAWING: Figure 6A

Description

[Technical field]

[0001] The autonomous vehicle includes a perception system that performs semantic decomposition (SD), where the AV flow consists of three modules: the perception module feeds its output to a route planning module, which feeds its output to actuation modules (brake / accelerate / steer, etc.).

[0002] Since the route planning and actuation modules do not have access to the raw sensor inputs, the role of perception in the SD architecture is to comprehensively label the raw inputs from the car's sensors, e.g., to detect cars and pedestrians, segment the free driving space, recognize speed limits via road signs, and anything else in the scene that could possibly affect driving in this scene. SD perception modules tend to be complex and computationally intensive models, since a large number of objects / entities need to be identified and precisely located in the scene. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] PCT Patent Application International Publication No. 2020 / 079508 [Patent Document 2] U.S. Patent Application Serial No. 16 / 035,732 [Patent Document 3] U.S. Patent Application Serial No. 18 / 036,150 Summary of the Invention

[0004] The embodiments of the present disclosure will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings, in which: [Brief description of the drawings]

[0005] [Figure 1] FIG. 1 illustrates an example of a system. [Diagram 2] FIG. 1 illustrates an example of a system. [Diagram 3] FIG. 1 illustrates an example of a method. [Figure 4] FIG. 4 illustrates an example of the steps of the method of FIG. 3. [Figure 5A] FIG. 1 illustrates an example of a method. [Figure 5B] FIG. 1 illustrates an example of an embodiment of a method. [Figure 5C] FIG. 1 is a diagram showing an example of an image. [Figure 5D] FIG. 1 illustrates an example of a vehicle. [Figure 5E] FIG. 1 is a diagram showing an example of an image. [Figure 5F] FIG. 1 is a diagram showing an example of an image. [Figure 5G] FIG. 1 is a diagram showing an example of an image. [Figure 6A] FIG. 1 is a diagram showing an example of an image. [Figure 6B] FIG. 1 illustrates an example of a prediction. [Figure 6C] FIG. 2 shows an example of the various units. [Figure 7A] FIG. 2 is a diagram showing an example of a unit. [Figure 7B] FIG. 1 shows an example of MDI statistics. [Figure 7C] FIG. 1 illustrates an example of a method. [Figure 7D] FIG. 2 shows an example of the various units. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0007] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. However, the invention, both as to organization and method of operation, together with its objects, features, and advantages, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

[0008] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or similar elements.

[0009] Because the illustrative embodiments of the present invention can, for the most part, be implemented using electronic components and circuits known to those skilled in the art, details will not be described to any greater extent than is deemed necessary, as set forth above, for an understanding and appreciation of the underlying concepts of the present invention and in order not to obscure or deviate from the teachings of the present invention.

[0010] Any reference herein to a method should apply mutatis mutandis to a device or system capable of performing the method and / or to a non-transitory computer readable medium storing instructions for performing the method.

[0011] Any reference herein to a system or device should apply mutatis mutandis to methods that may be performed by the system and / or may apply mutatis mutandis to a non-transitory computer-readable medium that stores instructions executable by the system.

[0012] Any reference herein to a non-transitory computer readable medium should apply mutatis mutandis to a device or system capable of executing instructions stored on the non-transitory computer readable medium and / or may apply mutatis mutandis to a method of executing the instructions.

[0013] Any combination of any modules or units listed in any of the figures, any part of this specification and / or any claim may be provided.

[0014] Any one of the perception unit, specialized AI agent, and driving decision unit may be implemented in hardware and / or with code, instructions and / or commands stored on a non-transitory computer readable medium and may be contained within the vehicle, external to the vehicle, in a mobile device, in a server, etc.

[0015] The vehicle may be any type of vehicle, such as a land vehicle, an aircraft, or a watercraft.

[0016] The specification and / or drawings may refer to images. The images are examples of media units. Any reference to images may apply mutatis mutandis to media units. The media units may be examples of sensed information. Any reference to media units may apply mutatis mutandis to any type of natural signal, such as, but not limited to, signals produced by nature, signals representing human behavior, signals representing operations related to the stock market, medical signals, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, and the like. Any reference to media units may apply mutatis mutandis to sensed information. The sensed information may be of any kind and may be sensed by any type of sensor, such as a visible light camera, an audio sensor, a sensor capable of sensing infrared, radar imagery, ultrasonic, electro-optical, radiography, LIDAR (light detection and ranging), and the like. Sensing may include generating samples (e.g., pixels, audio signals) representative of signals transmitted or otherwise reaching the sensor.

[0017] The specification and / or drawings may refer to spanning elements. The spanning elements may be implemented in software or hardware. Different spanning elements of a particular iteration are configured to apply different mathematical functions to the inputs they receive. Non-limiting examples of mathematical functions include filtering, although other functions may be applied.

[0018] The specification and / or drawings may refer to a conceptual structure. The conceptual structure may include one or more clusters. Each cluster may include a signature and associated metadata. Each reference to the one or more clusters may be applicable to the reference to the conceptual structure.

[0019] The specification and / or drawings may refer to a processor. The processor may be processing circuitry. The processing circuitry may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or the like, or a combination of such integrated circuits.

[0020] Any combination of any steps of any method illustrated in the specification and / or drawings may be provided.

[0021] Any combination of the subject matter of any of the claims may be provided.

[0022] Any combination of the systems, units, components, processors and sensors illustrated in the specification and / or drawings may be provided.

[0023] Any reference to an object can be applicable to a pattern, and therefore any reference to object detection can be applied mutatis mutandis to pattern detection.

[0024] A situation can be a unique location / property combination at a point in time. A scenario is a sequence of events that follow logically within a causal frame of reference. Any reference to a scenario should apply mutatis mutandis to a situation.

[0025] The sensed information units may be sensed by one or more sensors of one or more types, the one or more sensors may belong to the same device or system or may belong to different devices of the system.

[0026] A perception unit may be provided, which may be preceded by one or more sensors and / or one or more interfaces for receiving one or more sensory information units. The perception unit may be configured to receive the sensory information units from an I / O interface and / or from a sensor. The perception unit may be followed by multiple specialized AI agents, also referred to as an ensemble of specialized AI agents.

[0027] An artificial intelligence (AI) agent can refer to an autonomous entity that directs its activities toward achieving a goal (i.e., it is an agent) while acting on the environment using observations through sensors and resulting actuators (i.e., it is intelligent). Intelligent agents can also learn or use knowledge to achieve their goals. They can be very simple or very complex. A reflex machine, such as a thermostat, is considered an example of an intelligent agent (www.wikipedia.org).

[0028] The sensory information units may or may not be processed before reaching the perception unit. Any processing - filtering, noise reduction, etc. - may be provided.

[0029] The number of specialized AI agents can be, for example, more than 100, more than 500, more than 1000, more than 10,000, more than 100,000, etc. A large number of specialized AI agents can provide more accurate driving decisions.

[0030] AI-specialized agents are specialized in the sense that they are not trained to respond to every possible (or perhaps all or even most) scenarios to be handled by the entire ensemble. For example, each AI-specialized agent may be trained to respond to a fraction (e.g., less than one percent) of the scenarios managed by the entire ensemble. Specialized AI agents may be trained to respond to only some of the factors or elements or parameters or variables that make up the scenarios.

[0031] The specialized AI agents may be of the same complexity and / or the same parameters (depth, energy consumption, technical implementation), although at least some of the specialized AI agents may differ from each other by at least one of the complexity and / or parameters.

[0032] Specialized AI agents can be trained in a supervised and / or unsupervised manner.

[0033] The one or more specialized AI agents may be a neural network or may be distinct from a neural network.

[0034] An ensemble may include one or more sensors and any other entities that generate sensing information units and / or may receive (via an interface) one or more sensing information units from one or more sensors.

[0035] A perception unit can process one or more sensory information units and determine which specialized AI agents are relevant to the processing of the one or more sensory information units.

[0036] An autonomous vehicle system may be provided that can use a perception unit to classify an observed scene into multiple coarse-grained categories. The system may include an ensemble of specialized AI agents (EoN).

[0037] The perception unit can receive and / or generate anchors that, when detected (by the perception unit), can influence the selection of which specialized AI agent to select. The number of anchors can be very large (e.g., 100, 500, 1000, 10,000, 20,000, 50,000, over 100,000 anchors, and even more).

[0038] For a given scenario (which may be represented by one or more sensory information units, such as but not limited to one or more images), the perception unit may detect one or more anchors.

[0039] The detected anchors can provide sufficient contextual cues to enable the perception unit to determine which are the relevant specialized AI agents.

[0040] The contextual cues may be high-level sensed information unit context. Determining the contextual cues is high-level in the sense that it is less complex and / or requires less computational resources than performing object detection of small objects in the sensed information units. A small object may be the smallest size to be detected, e.g., tens of pixels in size, smaller than 0.1, 0.5, 1, 2, 3 percent of the sensed information units, etc. Determining the contextual cues may not include, for example, determining the exact location of each object in the image, including the location of objects that appear as tens of pixels in the image.

[0041] By retrieving high-level sensory information unit context, the power consumption of the perception unit can be much lower (e.g., up to two orders of magnitude lower) than the power consumption of prior art systems built to perform the entire process of object detection and to determine which driving maneuver to perform.

[0042] At least some of the power savings may be due to the fact that the high-level sensing information unit context may not include location information, there being no need to determine whether objects of different sizes are the same type of object, etc.

[0043] A specialized AI agent can receive inputs directly from sensors (e.g., as the output of a perception module) and provide as output proposed behaviors (desired vehicle control parameters - and desired vehicle behavior - also called driving decisions), e.g. steering wheel angle, acceleration / braking signals, or control of any aspect of driving.

[0044] The outputs from the various selected specialized AI agents are fed to a driving decision unit (also called a coordinator), which outputs one or more output driving decisions, such as one or more commands or requests or recommendations to various modules of the vehicle and / or to the driver.

[0045] The coordinator may apply any method of generating one or more output operational decisions, such as one or more commands and requests, based on the outputs from the various selected specialized AI agents.

[0046] These methods may include arbitration, competition, selecting a response based on the risk imposed by adopting the output of specialized AI agents, etc.

[0047] Referring again to perceptual units, some non-limiting examples of anchors are: Approaching a crosswalk. Detecting a pedestrian on the sidewalk triggers different behaviors depending on the proximity of an intersection. If a pedestrian is standing on the sidewalk adjacent to an intersection, the car should slow down (if the pedestrian is going to start crossing). The same pedestrian on the sidewalk can be safely ignored in other areas. Thus, different specialized agents may be needed to drive through a crosswalk. Approaching a pothole on the road, which can trigger the vehicle to slow down or change lanes. Limited visibility and slippery road conditions - a slower speed and / or a longer safety distance from the vehicle ahead may be indicated. Encoded navigation signals - when crossing a junction, the specialized agent should get a signal if this junction is to be crossed straight, turned left, or turned right. This anchor / signal will change the meaning of what it means to "stay in lane". · Changing lanes - this anchor triggers a specific specialized agent in charge of changing lanes to the left or right. · Approaching a road with a mixture of road signs – do I give the right-of-way or do I have the priority? Presence of unsafe drivers in the vicinity: Tailgating, reckless driving, road rage cues can induce drivers to change to the right-most lane, slow down, and let unsafe vehicles pass. · A police car follows you with its emergency lights flashing: it will switch to the right-most lane and then stop on the shoulder. · Cut-in vehicles perform emergency stops: activating specialized agents that perform either emergency braking or swerving onto the shoulder and stopping there at lower deceleration conditions.

[0048] Anchors can be selected, generated, and / or learned in various ways - manually, automatically, based on human input tagging, based on autonomous input tagging, based on manual identification, based on tagging of scenarios (e.g., approaching a roundabout, approaching a pedestrian crossing, environmental conditions, presence of road users) - where the corresponding desired behavior can be devised from the recorded behavior of human drivers in this situation, identifying unfamiliar or poorly predictive patterns of human driving. These patterns can suggest world states that are not well covered by existing specialized agents, thus providing for the allocation of new agents.

[0049] A specialized AI agent may be or may include a simple model (e.g., a neural network) that receives raw (or in some manner preprocessed) sensor data as its input, processes it internally, and outputs a proposed behavior.

[0050] Examples of specialized AI agents include: A specialized AI agent for "stay in lane" steering behavior when not at a junction. The specialized AI agent can be a convolutional neural network that maps the image of the road captured by the front camera to the vehicle's steering angle. · A specialized AI agent for "stay in lane" steering behavior at left-turn junctions. Note that "stay in lane" at a junction has different meanings if we drive straight or turn left. The specialized agent for left receives images from the front and left-facing cameras and maps them to steering angles. Depending on the specification, the information indicating that the agent will turn left instead of driving straight is either received as input from the navigation block or encoded as an "anchor". This specialized AI agent can be implemented as a CNN. A specialized AI agent that implements the behavior "change to the lane on the left." This specialized AI agent gets access to the left-facing camera and proposes the actions needed to change lanes, i.e., the decision on when to change lanes, the acceleration before and during the change, and the steering to effect the change of lanes. A specialized AI agent that steers when the vehicle is stopped at a red light. This specialized AI agent is implemented as an "if" condition and simply keeps the steering angle constant. A specialized AI agent implementing the “keep the distance” logic: This specialized AI agent maps images captured by the front and rear cameras to acceleration / braking signals. A specialized AI agent that implements "give right of way on pedestrian crossing" - this specialized AI agent will stop if a pedestrian is at the crossing, or slow down if a pedestrian is near but not at the crossing, and uses input from the front camera and maps it to braking / accelerating signals.

[0051] An example of how a coordinator can process the output from one or more specialized AI agents: Combine independently. The specialized AI agent in charge of the steering angle that allows the car to stay in a given lane is almost independent of the driving speed, so the behavior proposed by this agent can be combined independently with the acceleration / braking behavior of the "keep distance" specialized agent. Override. The vehicle should stay in its lane, but the car ahead suddenly stops in front of us and there is not enough braking distance to stop. Now, the desire to avoid a collision overrides the "stay in lane" behavior and we swerve onto the shoulder and come to a stop. Animals on the road – give up on “staying in your lane” and take evasive action. "Modulate" Slippery roads will leave the “stay in lane” and “keep distance” behavior largely unchanged, but will dictate slower speeds and longer safety distances. · Implement risk reduction optimization.

[0052] The system is expected to be superior to SD systems in all aspects, whether it be behavioral accuracy, model size and complexity, or computational intensity.

[0053] The perception unit can work in a more efficient manner than state-of-the-art perception systems, which analyze the scene completely with the goal of reconstructing it, i.e. focusing on all the details of the scene, such as a rigorous analysis of all the agents on the road (pedestrians, cars, bikes, lanes, road signs, traffic lights, trees, obstacles, etc.) and their attributes (exact location, pose, distance, angle, etc.). This is a very power- and resource-intensive process - since the aim is to generate an extensive environment model as input for the policy module.

[0054] The perception unit does not need to focus on every detail, and does not need to identify every agent - it only needs to analyze and classify the scene in order to activate the relevant agents. This is a very light processing performance scheme. Once activated, the relevant agents will analyze the input image for relevant features.

[0055] It should be noted that even when the number of specialized AI agents is large - some agents may be stored in RAM and some agents may be kept in non-volatile memory (e.g., on disk, in inexpensive non-volatile memory, etc.) - the specialized AI agents can be retrieved when needed.

[0056] The perception module can be very fast, since its task is to detect anchors and select relevant specialized AI agents based on the anchors. Each specialized AI agent can also be very fast, but we have a very large number of specialized AI agents. Fortunately, at any given time, only a small number of them actually need to be executed, i.e., at any given time, the execution time is very short.

[0057] The suggested units may be executed or hosted by a processor, which may be processing circuitry, which may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or the like, or a combination of such integrated circuits.

[0058] FIG. 1 illustrates an example of a system 10 .

[0059] The system 10 includes a perception unit such as a perception router 30 (which receives a sensed information unit, which is an image 8 having pixels), an ensemble of specialized AI agents 40 (e.g., specialized AI agents associated with a roundabout, a pedestrian walking across a zebra crossing, a particular road sign, or a traffic jam), and a driving decision unit such as a coordinator 50. The driving decision unit 50 can control and / or communicate with a response unit 60.

[0060] FIG. 2 shows an example of a system 10'.

[0061] The system 10' includes an acquisition unit 20' (for receiving one or more sensory information units 15), a perception unit 30', specialized AI agents 40(1)-40(K) (K being the number of specialized AI agents), and a driving decision unit 50'. The driving decision unit can control and / or communicate with a response unit 60', such as a brake control system, any vehicle computer, an autonomous driving module, an ADAS driving module, etc.

[0062] FIG. 3 illustrates a method 300 for operating an ensemble of specialized AI agents associated with a vehicle.

[0063] By vehicle related it is meant that the output of the method 300 can be one or more driving decisions that, when implemented, can affect the operation of the vehicle.

[0064] The method may include various steps, some of which may include providing the desired driving decisions (e.g., during training of any portion of the entity used during method 300, the entity may include a perception unit, a specialized AI agent, and a driving decision unit).

[0065] Additionally or alternatively, a driving decision associated with any of the sensing information units provided to any of the entities may be provided, and the method may include determining which driving decision was the correct driving decision. For example, this may be determined using statistics - e.g., taking the most common driving decisions for a situation and / or scenario, or any portion thereof.

[0066] The method 300 may begin with an initialization step 310 .

[0067] Step 310 may include obtaining a perception unit, a specialized AI agent, and a driving decision unit configured to perform various steps of the method 300.

[0068] Obtaining can include receiving after being trained and / or receiving at any stage of the training and / or training process, downloading instructions, or otherwise configuring a computerized system to perform any other step of method 300.

[0069] Step 310 may include at least one of (a) training at least one of the perception unit, the specialized AI agent, and the driving decision unit; (b) receiving at least one of the perception unit, the specialized AI agent, and the driving decision unit that has already been trained; and (c) otherwise configuring at least one of the perception unit, the specialized AI agent, and the driving decision unit.

[0070] Step 310 may be followed by step 320 of obtaining one or more sensed information units.

[0071] Obtaining can include sensing, receiving without sensing, preprocessing, and the like.

[0072] Step 320 may be followed by step 330 of determining, by the perception unit, based on the one or more sensory information units, one or more relevant specialized AI agents of an ensemble that may be relevant to processing the one or more sensory information units, the entire ensemble being relevant to a first plurality of scenarios.

[0073] Each associated specialized AI agent may be associated with a dedicated class. The class may be associated with an anchor. Step 330 may include retrieving the anchor.

[0074] Each class may be defined by at least a portion of one or more scenarios, where at least a portion of the one or more scenarios may be a fragment of the first plurality of scenarios.

[0075] Various specialized AI agents may be trained to respond to various scenarios which may be (or may include) T-junctions, various road elements, zebra crossings, roundabouts, obstacles, various environmental conditions, rain, fog, night, straight highways, hill climbs, traffic jams, .... Examples of various obstacles and / or various road elements are shown in PCT Patent Application WO 2020 / 079508, entitled "METHOD AND SYSTEM FOR OBSTACLE DETECTION," which is incorporated herein in its entirety.

[0076] The different scenarios may be different or distinct from the situations.

[0077] The scenario may be, for example, at least one of: (a) vehicle location; (b) one or more weather conditions; (c) one or more contextual parameters; (d) road conditions; and (e) traffic parameters.

[0078] Various examples of road conditions may include road unevenness, the level of road maintenance, the presence of potholes or other relevant road hazards, whether the road is slippery, and whether it is covered with snow or other particles.

[0079] Various examples of the traffic parameters and the one or more contextual parameters may include time (hour, day, period or year, specific time of a specific day, etc.), traffic load, distribution of vehicles on the road, behavior of one or more vehicles (aggressive, gentle, predictable, unpredictable, etc.), presence of pedestrians close to the road, presence of pedestrians close to the vehicle, presence of pedestrians away from the vehicle, behavior of pedestrians (aggressive, gentle, predictable, unpredictable, etc.), risk associated with driving in the vicinity of the vehicle, complexity associated with driving in the vicinity of the vehicle, presence of at least one of a kindergarten, a school, a crowd (close to the vehicle), etc. The contextual parameters may relate to a context of the sensed information context - a context may be dependent on or related to a situation forming a setting for an event, utterance, or thought.

[0080] Relevant specialized AI agents can be trained to respond to one or more situations from a much larger number of situations. An example of one or more situation based processes is shown in U.S. Patent Application No. 16 / 035,732, which is incorporated herein by reference.

[0081] Each class may be defined by an anchor, which may be a contextual cue. Step 320 may be performed without object detection, which may be less than a predefined number of pixels, which may not exceed a few tens of pixels.

[0082] The specialized AI agent may be an end-to-end specialized AI agent.

[0083] For at least some of the specialized AI agents, each fraction may be less than 1 percent of the first plurality of scenarios.

[0084] Step 330 may be followed by step 340 of transmitting one or more sensed information units to an associated specialized AI agent.

[0085] It should be noted that once one or more relevant specialized AI agents have been determined, they may be uploaded to a processor and / or memory unit, which reduces the RAM or other memory resources required to store and execute step 350.

[0086] Step 340 (or method 300) may include maintaining at least one unassociated specialized AI agent in a low power mode (idle, inactive, in a sleep mode, partially operational, etc.), where the power consumption of the at least one unassociated specialized AI agent may be less than the power consumption of the associated specialized AI agent.

[0087] Step 340 may be followed by step 350 of processing one or more sensed information units by one or more associated specialized AI agents to provide one or more specialized AI agent driving decisions. Each specialized AI agent may be associated with a respective piece of a first plurality of scenarios managed by the entire ensemble.

[0088] A specialized AI agent driving decision can be a command, request, or recommendation to autonomously control a vehicle.

[0089] A specialized AI agent driving decision can be an advanced driver-assistance system (ADAS) command, an ADAS request, or an ADAS recommendation.

[0090] Step 350 may be followed by step 360 of processing one or more specialized AI agent driving decisions by a driving decision unit to provide an output driving decision.

[0091] Step 360 may include averaging one or more specialized AI agent driving decisions, or applying one or more functions (e.g., predefined and / or learned and / or time-varying) and / or applying one or more policies to one or more specialized AI agent driving decisions. An example of a policy-attempting to reduce risk-for example, in the case of two specialized AI agent driving decisions with different speeds-selecting the lower speed.

[0092] Method 300 - particularly steps 320-360 - may be repeated multiple times, for example multiple times per second, once per second, once every few seconds, continuously or non-continuously.

[0093] FIG. 4 shows an example of step 310.

[0094] Step 310 may include step 312 of training the perception units to classify the sensed information units into predefined classes.

[0095] Each class may be at least a part of one or more scenarios, and the one or more scenarios may be part of a fragment of the first plurality of scenarios.

[0096] Each class may be associated with an anchor, which may be used to classify the sensed information units into classes.

[0097] Step 312 may include receiving, by the perception unit, definitions of at least some of the classes prior to training, which may include, for example, receiving labels or any other class defining information.

[0098] Step 312 may include defining definitions of at least some of the classes by the perceptual units.

[0099] The defining can include, for example, generating signatures and clustering the signatures into conceptual structures such as clusters. Clustering virtually defines classes.

[0100] The clusters may all belong to the same level or may be arranged hierarchically. The clustering may essentially respond to statistics of contextual cues - more frequently occurring contexts may be partitioned into more clusters. Larger clusters may be divided into lower level clusters in any way - for example, by cross-correlation between cluster members, finding shared and unique signature parts, etc.

[0101] Step 312 may include performing unsupervised training.

[0102] At least a portion of the one or more scenarios may be at least one of: (a) one or more factors of the scenario; (b) one or more elements of the scenario; (c) one or more parameters of the scenario; and (d) one or more variables of the scenario.

[0103] Step 312 may include providing a first data set of the sensory information unit to the perception unit.

[0104] Step 312 may be followed by step 314 of using the trained perception units to classify sensory information units of a second data set, where the sensory information units are referred to as second sensory information units.

[0105] Step 314 may also include providing one or more maneuvering decisions per class.

[0106] Each class will include a plurality of second sensing information units and one or more driving decisions.

[0107] Each specialized AI agent may be associated with a dedicated class. Step 314 may be followed by step 316 of training each specialized AI agent to output specialized AI agent driving decisions associated with the dedicated class.

[0108] Training at step 316 may include providing a specialized AI agent associated with a given class with the second sensory information and one or more movement decisions of that class.

[0109] Step 316 may be performed in a supervised or unsupervised manner. Supervised training may include providing one or more driving decisions as a desired output of a specialized AI agent.

[0110] Step 316 may be followed by step 318 of training the driving decision unit to provide an output driving decision based on one or more specialized AI agent driving decisions.

[0111] Step 318 includes providing a sensing information unit of a third dataset (hereinafter, the third sensing information unit) to a perception unit, and enabling the perception unit to determine an associated specialized AI class (based on the class of the third sensing information unit), enabling the associated specialized AI class to output a specialized AI agent driving decision, and providing the specialized AI agent driving decision and the driving decision associated with the third dataset to a driving decision unit.

[0112] Each of the first, second and third data sets may include any number of sensing information units and may be generated in any manner, they may include randomly selected sensing information units or any combination of sensing information units.

[0113] A non-transitory computer readable medium may be provided that can store instructions for operating an ensemble of specialized AI agents, where the operating can include obtaining one or more sensed information units, determining, by a perception unit, based on the one or more sensed information units, one or more associated specialized AI agents of the ensemble associated with processing the one or more sensed information units, where the ensemble is associated with a first plurality of scenarios, processing the one or more sensed information units by the one or more associated specialized AI agents to provide one or more specialized AI agent driving decisions, where each specialized AI agent is associated with a respective fragment of the first plurality of scenarios, and processing the one or more specialized AI agent driving decisions by a driving decision unit to provide an output driving decision.

[0114] A system for driving decision making, the system including an ensemble of AI specialized agents and a perception unit configured to acquire one or more sensed information units and determine, based on the one or more sensed information units, one or more associated specialized AI agents of the ensemble associated with processing the one or more sensed information units, the ensemble associated with a first plurality of scenarios, the one or more associated specialized AI agents configured to process the one or more sensed information units to provide one or more specialized AI agent driving decisions, each specialized AI agent associated with a respective fragment of the first plurality of scenarios, and a driving decision unit configured to process the one or more specialized AI agent driving decisions to provide an output driving decision.

[0115] Use of multiple perceptual modules The factors that influence the vehicle may have different domains. It has been found by the inventors that allocating a perception module per domain instead of using a single perception unit provides various technical advantages, such as: The entire perception module is smaller than a single perception unit used to manage all information domains, reducing computational and / or memory resources allocated to the selection of specialized AI agents. Reduces computational and / or memory resources allocated to training, since each perception module is trained with much more compact information and covers much fewer classification scenarios. Providing a more accurate selection of specialized AI agents - Perception modules assigned to a domain can better adapt to the domain-related choices. This does not force the perception modules to make trade-offs between different domains of information. · Facilitate adaptation to new domains of information - this requires training or adding already trained perceptual modules dedicated to the new domain of information. · When the domain of information changes, the selection of a perceptual module (of a group of perceptual modules) is changed in a dynamic manner. · Keeping the non-selected perception modules of the group in an idle state - thus reducing power consumption. Retrieving information about selected perception modules while avoiding retrieving information about non-selected perception modules - this reduces power consumption and / or memory resources allocated to the selection of specialized AI agents.

[0116] FIG. 5A illustrates one example of a method 500 for operation of specialized artificial intelligence (AI) agents for at least partially autonomous driving.

[0117] According to one embodiment, the method 500 includes automatically selecting 510 a number of perception modules from a group of perception modules.

[0118] According to one embodiment, the selecting is based on a domain associated with the multi-domain information.

[0119] According to one embodiment, the selection is based on previously generated outputs of the perception modules of the group of perception modules. For example, if there is not enough light (which can be detected by image processing, location and time information - for example, driving on an unlit road in an unlit environment during dark night), the perception module based on image processing may not be activated.

[0120] According to one embodiment, at least two different perception modules are associated with two different vehicle sensors. The selection can be based on the operational status of the two different vehicle sensors. For example, if one vehicle sensor is found to have failed or to generate uncertain sensing information, the perception router associated with that vehicle sensor can be deselected, kept in an idle mode, etc.

[0121] According to one embodiment, step 510 may include maintaining a different group of perception modules in an idle mode from the plurality of perception modules, or such maintaining may be continued after step 510.

[0122] According to one embodiment, the plurality of perceptual modules are a plurality of perceptual routers. According to one embodiment, the plurality of perceptual modules are a plurality of perceptual sub-routers associated with a perceptual router. An example of a perceptual router is shown in U.S. Patent Application No. 18 / 036,150, which is incorporated herein by reference.

[0123] According to one embodiment, step 510 is followed by step 520 of receiving multi-domain information regarding factors affecting the vehicle by a plurality of perception modules, each one of which is associated with a dedicated domain of the multi-domain information.

[0124] According to one embodiment, step 520 is followed by step 530 of generating, by a plurality of perception modules, class signatures indicative of classes of elements of the multi-domain information.

[0125] According to one embodiment, the perception module is a classifier configured to classify information of a domain associated with the perception module. The classifier may implement a machine learning process, may be a deep neural network (DNN), may be a neural network other than a DNN, may be different from a neural network, etc.

[0126] According to one embodiment, step 530 is followed by step 540 of determining a multi-domain identifier (MDI) that identifies the generated class signatures of a plurality of perception modules. The MDI can be a sequence of class signatures or can identify the generated class signatures in any other manner.

[0127] According to one embodiment, step 540 is followed by step 550 of identifying, based on the MDI, one or more specialized AI agents associated with processing at least a portion of the MDI.

[0128] According to one embodiment, one or more specialized AI agents are associated with clusters of MDIs. For example, a first specialized AI agent may be associated with a first set of one or more MDI clusters, and a second specialized AI agent may be associated with a second set of one or more MDI clusters. The number of MDIs in the first set may be the same as the number of MDIs in the second set. The number of MDIs in the first set may be different from the number of MDIs in the second set. The first and second sets may share one or more MDIs. The first and second sets may not include any shared MDIs.

[0129] A specialized AI agent or cluster of MDIs may be associated with a skill. Selection of one or more specialized AI agents may virtually involve selecting one or more skills required to address a situation or scene.

[0130] According to one embodiment, the multi-domain information is input from at least one of the following: Road configuration information related to static objects in the vehicle's environment: Examples of static objects include road elements (lanes, curves, junctions, roundabouts), non-road static elements (houses, trees), and location information (city, rural, urban environment, non-urban environment). Road user information relating to mobile road users in the environment. Examples of mobile road users include pedestrians, other vehicles. The road user information may include information about the location and / or behavior of the road users. Traffic rule markings information related to visual traffic rule indicators in the environment. Traffic rule markings are traffic signs, markings on the roadway that indicate traffic rules (e.g., lane markings, zebra crossings, arrows, text, symbols indicating permitted driving directions, etc.). Regulatory information regarding legal constraints related to the environment. Legal constraints may include maximum speeds, minimum speeds, etc. The regulatory information may be obtained from the regulatory information based location or from other sources. Ambient condition information relating to weather and / or light conditions of the environment. Vehicle status information regarding the movement related status of the vehicle.

[0131] According to one embodiment, the plurality of perception modules may include at least one of the following: Road setting perception module. The road setting perception module may be configured to generate a class signature indicative of one or more classes of one or more static objects in the vehicle's environment. The class signature may be indicative of a scenario that includes all static objects that (i) may affect the vehicle and (ii) are in the vehicle's environment. Road User Perception Module. The road user perception module may be configured to generate class signatures indicative of classes associated with mobile road users. The class signatures may indicate scenarios including all mobile road users that (i) may affect the vehicle and (ii) are within the vehicle's environment. Traffic Rule Perception Module. The traffic rule perception module may be configured to generate a class signature indicative of a class of traffic rules indicated by traffic signs or other elements captured in the image. The class signature may represent a scenario including all traffic rules that (i) may affect the vehicle and (ii) are indicated by traffic rule indicators in the vehicle's environment. A regulatory perception module may be configured to generate a class signature indicative of a class of legal constraints. The class signature may represent a scenario that includes all legal constraints that (i) may affect the vehicle and (ii) are applicable to the vehicle's environment. Ambient Condition Perception Module. The ambient condition perception module may be configured to generate a class signature indicative of a class of ambient conditions within the vehicle's environment, such as at least one of weather and light conditions of the environment. Vehicle State Perception Module. The vehicle state module may be configured to generate class signatures indicative of classes of vehicle states, particularly vehicle motion related states. The motion related states may include kinematic information, speed information, acceleration information, etc.

[0132] It should be noted that any other perceptual modules may be provided, and that at any given time, one or more of said perceptual modules may be selected. There may be any number of domains and any number of perceptual modules assigned to a domain.

[0133] According to one embodiment, the identification of step 550 triggers performance of further processing of at least a portion of the multi-domain information by the identified specialized AI agent or agents to provide one or more specialized AI driving-related decisions.

[0134] FIG. 5B shows an example of various information sources, various perception modules, an MDI generator, a selection unit specialized AI agent, and a driving decision unit.

[0135] Various sources of information include: Information about the vehicle's environment from one or more vehicle sensors 581(1). Information from one or more vehicle condition sensors 581(2). · Surrounding area information 581(3). · Legal Constraints Database 581(4).

[0136] The Perception Module includes: Road setting perception module - represented as RS perception module 582(1). Road User Perception Module--represented by the RU Perception Module 582(2). · Traffic rule perception module - represented by TR perception module 582(3). Regulatory Perception Module--represented by the REG Perception Module 582(4). Ambient Condition Perception Module - represented by AMB Perception Module 582(5). Vehicle state perception module - represented as VS perception module 582(6).

[0137] FIG. 5B also shows unselected perception module 582(6), which may be idle.

[0138] A selection unit 587 identifies one or more specialized AI agents by comparing the MDI 585 with the MDI cluster signatures 586(1)-586(J) to find one or more matching MDI clusters associated with the selected specialized AI agent or agents.

[0139] The matching MDI cluster indicates the specialized AI agent to be selected, represented at 588. Figure 5B also shows the specialized AI agents that are not selected, represented as including a dashed inner pattern.

[0140] The selected specialized AI agent 588 outputs a specialized AI agent driving decision 589 .

[0141] The driving decision unit 590 is configured to receive the specialized AI agent driving decisions 589 and is configured to generate and output output driving decisions 591 that can be executed by one or more units of the vehicle.

[0142] FIG. 5C shows an example of an image 610 captured by a vehicle sensor.

[0143] Image 610 shows two lanes of a road 611 and 612, a building on the right 613, a first vehicle 615, a second vehicle 616, a tree 618, a table 617, a building on the left 621, a child 623, a ball 624, a traffic sign 630 indicating a maximum speed limit (e.g., 50 Km / H), a driving direction arrow 634, a zebra crossing 633, and a crowd 628 that has reached the zebra crossing and is in the process of crossing it from right to left. Figure 6 also shows the advance (behavior) 641 of the child 623, the advance 642 of the crowd 628, the advance 645 of the first vehicle 615, and the advance 646 of the second vehicle 616.

[0144] The road user information may be indicative of mobile road users such as a first vehicle 615, a second vehicle 616, a child 623 and a crowd 628.

[0145] The road setting information may indicate static objects selected from lanes 611 and 612, road 610, tree 618, table 617, right building 613 and left building 621 - not all of these static objects may be taken into account as not all of these static objects have an effect on the vehicle.

[0146] The traffic regulation display information shows traffic signs 630, driving direction arrows 634, and lane markings.

[0147] The ambient condition information may indicate that the vehicle's environment is illuminated by sunlight and is not obscured by rain or other weather elements, for example.

[0148] Vehicle state information can be learned from information not contained in the images.

[0149] Regulatory information can be learned from information not contained in the image.

[0150] 5D shows an example of a vehicle 700. The vehicle 700 includes a vehicle transmission unit 710, which may include one or more sensors, such as vehicle sensors 712 and 714. The vehicle also includes one or more processing circuits, designated 720, a memory unit 730, a communication unit 740, one or more vehicle units, such as one or more vehicle computers, one or more units controlled by the vehicle units, a motor unit, a chassis, wheels, etc.

[0151] The one or more processing circuits are configured to perform the method 600.

[0152] FIG. 5E illustrates an example of the method 800.

[0153] According to one embodiment, the method 800 includes steps 510 , 520 , 530 , 540 , and 550 .

[0154] According to one embodiment, the method 800 includes step 860 of triggering further processing of at least a portion of the multi-domain information by the identified specialized AI agent or agents to provide one or more specialized AI driving-related decisions.

[0155] According to one embodiment, the method 800 includes step 870 of further processing at least a portion of the multi-domain information by the identified specialized AI agent or agents to provide one or more specialized AI driving-related decisions.

[0156] According to one embodiment, the method 800 includes processing 360 one or more specialized AI agent driving decisions by a driving decision unit to provide an output driving decision.

[0157] It should be noted that the method 800 may include only one or two of steps 860, 870, and 360.

[0158] FIG. 5F illustrates an example of the method 900.

[0159] According to one embodiment, the method 900 includes receiving 910, by one or more identified specialized AI agents, information. The information may be multidimensional information provided to the plurality of perception modules, one or more segments of multidimensional information provided to the plurality of perception modules, or any representation or metadata relating to at least a portion of the plurality of perception modules.

[0160] According to one embodiment, the one or more identified specialized AI agents are identified by performing method 500 .

[0161] According to one embodiment, step 910 is followed by step 920 of processing information by the identified specialized AI agent(s) to provide one or more specialized AI driving related decisions that are sent to the driving decision unit. The one or more specialized AI driving related decisions may trigger the generation of one or more output driving decisions from the driving decision unit. The one or more output driving decisions may be one or more commands or requests or recommendations to various modules of the vehicle and / or to the driver.

[0162] FIG. 5G illustrates an example of the method 1000.

[0163] According to one embodiment, the method 1000 includes receiving 1010, by a driving decision unit, one or more specialized AI driving related decisions from the identified one or more specialized AI agents.

[0164] According to one embodiment, step 1010 is followed by step 1020 of determining, by the driving decision unit, one or more output driving decisions. The one or more output driving decisions may be one or more commands or requests or recommendations to various modules of the vehicle and / or to the driver.

[0165] A non-transitory computer-readable medium for operation of specialized artificial intelligence (AI) agents for at least partially autonomous driving may be provided, the non-transitory computer-readable medium storing instructions for receiving, by a plurality of perception modules, multi-domain information regarding elements affecting a vehicle, where each of the plurality of perception modules is associated with a dedicated domain of the multi-domain information, generating, by the plurality of perception modules, class signatures indicative of classes of elements of the multi-domain information, determining multi-domain identifiers that identify the generated class signatures of the plurality of perception modules, and identifying, based on the multi-domain identifiers, one or more specialized AI agents associated with processing at least a portion of the multi-domain information, where the identifying triggers execution of further processing of at least a portion of the multi-domain information by the identified one or more specialized AI agents to provide one or more specialized AI driving-related decisions.

[0166] According to one embodiment, the plurality of perception modules are a plurality of perception routers.

[0167] According to one embodiment, the plurality of perception modules are a plurality of perception sub-routers associated with a perception router.

[0168] A non-transitory computer readable medium stores instructions for automatically selecting a plurality of perception modules from a group of perception modules.

[0169] According to one embodiment, the selecting is based on a domain associated with the multi-domain information.

[0170] According to one embodiment, the selecting is based on pre-generated outputs of the perceptual modules of the group of perceptual modules.

[0171] According to one embodiment, the at least two different perception modules are associated with two different vehicle sensors, and the selecting is based on an operating state of the two different vehicle sensors.

[0172] The non-transitory computer readable medium stores instructions for maintaining a group of perception modules, distinct from the plurality of perception modules, in an idle mode.

[0173] According to one embodiment, the multi-domain information is input from one of road setting information associated with static objects in the vehicle's environment, road user information associated with mobile road users in the environment, traffic rule display information associated with visual traffic rule indicators in the environment, regulatory information associated with legal constraints associated with the environment, ambient condition information associated with at least one of weather and light conditions of the environment, and vehicle state information associated with movement-related states of the vehicle.

[0174] prediction Introducing prediction in the context of using domain-specific perceptual modules has been found to provide a variety of technical advantages, including: Prediction improves the management of memory units (e.g., but not limited to, cache memory units) by reducing the latency associated with memory misses - for example by increasing the accuracy of prefetch operations - as well as reducing memory consumption - since fewer fetch operations are required. Prediction, at least in many cases, reduces the urgency of processing and responding to sensory information - and can slow down processing resources allocated to running perception modules and / or AI agents, reducing energy consumption and increasing the lifespan of processing resources. Prediction can allow for smoother driving maneuvers - which is beneficial for the various units of the vehicle (e.g. from a mechanical point of view). Furthermore, some smoother driving maneuvers can be represented by lower coefficients - thereby reducing computational resources and also memory resources. Prediction allows to better determine the future progress of the vehicle in advance, over a longer period of time, thereby reducing the computational and / or memory resources required for more short-term iterations of determining the vehicle’s progress.

[0175] FIG. 6A is an example of a method 1100 for maintaining a swarm of specialized artificial intelligence (AI) agents for at least partially autonomous driving.

[0176] According to one embodiment, the method 1100 includes a step 1110 of obtaining, by a prediction circuit, a stream of metadata segments generated at multiple times and associated with a selection of one or more subgroups of a group of specialized AI agents.

[0177] According to one embodiment, the metadata segment is a selected specialized AI agent identifier.

[0178] According to one embodiment, the metadata segment is a plurality of multi-domain identifiers (MDIs). The plurality of MDIs indicate a plurality of instances of multi-domain information regarding elements that may affect the vehicle for a plurality of points in time (PITs). The MDI generated at a given one of the plurality of points in time is a combination of class signatures that indicate classes of elements of multi-domain information associated with the given point in time. The metadata segment may be generated using method 500.

[0179] According to one embodiment, step 1110 is followed by step 1120 of finding, by the prediction circuitry, a segment of the stream that is a predictor for the receipt of the next cluster identifier at a future time.

[0180] Step 1120 may include any prediction process, including but not limited to a machine learning prediction process, a non-machine learning process, applying a Kalman filter, applying any prediction filter, etc.

[0181] According to one embodiment, after step 1120, when finding the predictor, Future metadata segments that are received during the future time point. A future subpopulation of specialized AI agents to be selected at a future time This is followed by a step 1130 of automatically predicting at least one of

[0182] According to one embodiment, a given metadata segment results in the selection of one or more associated specialized AI agents, so that a known mapping exists between the metadata segment and one or more selected specialized AI agents.

[0183] According to one embodiment, predicting elicits a response to the predicting.

[0184] According to one embodiment, the method 1100 may include one or more of the following additional steps (collectively designated 1140): Eliciting a response to the prediction. · Responding to the prediction. · Eliciting a response that includes predicting a next state of the vehicle at a next time point. Triggering a response that includes determining a future forward movement of the vehicle based on one or more next states of the vehicle at one or more next time points. Determining a future progress of the vehicle based on one or more next states of the vehicle at one or more next time points, which may include smoothing the future progress of the vehicle, the smoothing being compared to the future progress of the vehicle without prediction. Triggering a response decision to the next state of the vehicle. Determining the response of the vehicle to the next condition. Evaluate the accuracy of the predictors. · predicting a selected perceptual module from the plurality of perceptual modules to be utilized during a future time point based on the predictor. triggering performance of a driving related operation at a time not more than a future time, the driving related operation may involve changing a speed of the vehicle, the method wherein the driving related operation may involve changing a direction of the vehicle.

[0185] FIGURE 6B is an example of a stream of metadata segments generated at multiple points in time and associated with the selection of one or more subgroups of a group of specialized AI agents. In FIGURE 6B, the metadata segments are represented as multiple MDIs 1180(1)-1180(10) associated with various points in time (PITs) 1182(1)-1182(10).

[0186] In FIG. 6B, assume that: A first predictor, which includes values ​​of the substream including MDIs 1182(1) through 1182(5), indicates that in a future PIT (e.g., PIT 1182(11)), an MDI having a value equal to MDI 1182(6) will be output. A second predictor, which includes values ​​of the sub-stream including MDI 1182(3), MDI 1182(6), and MDI 1182(9), indicates that in another future PIT (e.g., PIT 1182(12)), an MDI having a value equal to MDI 1182(7) will be output.

[0187] The first predictor and the second predictor may be utilized to do at least the following: Maintain the values ​​of MDI1182(6) and MDI1182(7) in cache memory until at least PIT1182(11). Determine the state of the vehicle at PITs 1182(10) and 1182(11) - and determine the desired state of the vehicle at the PITs prior to PITs 1182(21) and 1182(22).

[0188] In this case, the vehicle's movement may be smoothed to prevent large changes in direction and / or speed, such as at PIT 1182(9).

[0189] According to one embodiment, the selection of specialized AI agents may be altered to take predictions into account, for example - multiple prediction modules as referenced in the figures.

[0190] FIG. 6C shows an example of the following: Perception module 582. · MDI generator 584. · Elective Unit 587. · 588 Specialized AI Agents. Driving Decision Unit 590. · 1150 prediction units. Memory Management Unit 1152. Processing control unit 1153. A first memory unit 1154. Second memory unit 1155. Processing units 1154(1) to 1154(K). · Perception Module Metadata 1158. Prediction-Based Driving Decision Unit 590-1. Non-Prediction Based Driving Decision Unit 590-2.

[0191] The second memory unit 1155 is shown as being smaller than the first memory unit 1154 and, for simplicity of explanation, is assumed to be a cache memory. There may be more than two memory units.

[0192] The prediction may affect the output driving decision output by the driving decision unit, and / or may affect a memory management unit 1152, involved in managing one or more memory units used for at least partially autonomous driving, and / or may affect a processing management unit 1153 configured to manage processing resources, such as processing resources used to implement a perception module and / or a selection unit and / or a specialized AI agent, etc.

[0193] According to one embodiment, the processing units 1154(1)-1154(K) are processing circuits or portions of processing circuits.

[0194] According to one embodiment, the processing units 1154(1)-1154(K) are configured (e.g., are hardware programmed or configured to) implement (using data and / or metadata stored internally by the processing units and / or stored in one or more of the memory units) the perception module 582, the MDI generator 584, the selection unit 587, the specialized AI agent 588, the driving decision unit 590, the prediction unit 1150, the memory management unit 1152, and the processing management unit 1153.

[0195] According to one embodiment, at least two of the processing units 1154(1)-1154(K) may differ from each other by at least one parameter of a certain amount of computational resources, power consumption, complexity, etc. The selection of which processing unit to activate results in a trade-off between the various parameters of the at least two different processing units.

[0196] Execution of the perception module may require providing perception module metadata (such as coefficients of a model of a neural network) and information (such as information about the domain associated with the perception module) to the processing unit and / or an associated memory unit (e.g. to an internal memory unit of the processing unit or a second memory unit).

[0197] According to one embodiment, the prediction 1151 enables to efficiently control the fetching of information and / or sensory metadata, for example by at least one of (a) predetermining a prefetch operation and thus obtaining a cache hit, and (b) determining an allocation of sensory modules to processing units, which may reduce the amount of configuration of processing circuitry, since it is beneficial to allocate the same processing unit for implementing the same sensory module during successive cycles of operation.

[0198] According to one embodiment, prediction can help reduce the power consumption of a processing unit by reducing the clock frequency of a processing module implementing any unit that may be required to output a result within a few cycles, instead of outputting a result every -cycle, thereby making the processing unit unused and idle.

[0199] According to one embodiment, the driving decision units may include a prediction-based driving decision unit 590-1 and a non-prediction-based driving decision unit 590-2, where the prediction-based driving decision unit 590-1 is associated with driving decisions in the absence of prediction and the non-prediction-based driving decision unit 590-2 is associated with driving decisions in the presence of prediction, whereas the units may be selectively activated or deactivated in the presence or absence of prediction. The prediction-based driving decision unit 590-1 may operate at a lower clock frequency or may be otherwise configured to operate at reduced power compared to the non-prediction-based driving decision unit 590-2.

[0200] Local operation The environment of a vehicle at different locations may be location-biased in the sense that the statistical distribution of factors that may affect a vehicle at one location may differ from the statistical distribution of factors that may affect a vehicle at another location.

[0201] Prediction of the environment that a vehicle will encounter at various paths along its expected path can be highly effective and can provide a variety of technical advantages, such as: Prediction improves the management of memory units (e.g., but not limited to, cache memory units) by reducing the latency associated with memory misses - for example by increasing the accuracy of prefetch operations - as well as reducing memory consumption - since fewer fetch operations are required. Prediction, at least in many cases, can reduce the urgency of processing and responding to sensory information, slowing down processing resources allocated to running perception modules and / or AI agents, thereby reducing energy consumption and increasing the lifespan of processing resources. Prediction can allow for smoother driving maneuvers, which is beneficial for the various units of the vehicle (e.g. from a mechanical point of view). Furthermore, some smoother driving maneuvers can be represented by lower coefficients, thereby reducing computational resources and also memory resources. Prediction allows to better determine the future progress of the vehicle in advance, over a longer period of time, thereby reducing the computational and / or memory resources required for more short-term iterations of determining the vehicle’s progress.

[0202] 7A shows an example of an urban environment including multiple buildings 1202 forming a grid of streets. The expected local path of the vehicle (the expected path of the vehicle within the small environment, e.g., within a distance of 10 to 1000 meters from the vehicle) 1205 includes traveling west, going around a roundabout 1201-1, continuing traveling west, turning north, continuing north, turning west, and traveling west.

[0203] 7A also shows various locations (1202-1 through 1202-5 and 1202-11 through 1202-15) associated with MDI statistics. The MDI for each location indicates factors affecting the vehicle at that location. The MDIs may be generated in any manner, including those described in the text above, such as by method 500.

[0204] 7A, places 1201-1 through 1202-5 are path-related places that are themselves identified by processing circuitry based on expected local path 1205 and information about the places. Places 1202-11 through 1202-15 are considered irrelevant, e.g., they are too far away from the expected local path or belong to road segments that are not included within the expected local path.

[0205] FIG. 7B is an example of an MDI statistic value for a first location 1210-1, an MDI statistic value for a second location 1210-2, an MDI statistic value for a third location 1210-3, an MDI statistic value for a fourth location 1210-4, and an MDI statistic value for a fifth location 1210-5.

[0206] In the example of FIG. 7B, these MDI statistics include the most common MDI for each location - the most common MDI for the first location 1210-1, the most common MDI for the second location 1210-2, the most common MDI for the third location 1210-3, the most common MDI for the fourth location 1210-4, and the most common MDI for the fifth location 1210-5.

[0207] The most common MDI at a location may be an MDI that occurs at that location for at least a particular amount of time, or may have at least a predetermined probability of occurrence (e.g., greater than 10, 15, 20, 25, 30, 40 percent, etc.), or may be the Xth most common MDI, where -X may be determined in any manner, e.g., the most 5, 10, 15, 20, 25, 30 MDIs, etc.

[0208] According to one embodiment, there may be one or more instances of the most common MDI for one or more constraints. For example, the constraints may refer to different times, night or day, different dates, different seasons, different weather conditions, etc. See, for example, the two instances 1211-1-1 and 1211-2.

[0209] FIG. 7C illustrates an example of a method 1300 for localized operation.

[0210] According to one embodiment, the method 1300 includes obtaining 1310 information regarding locations associated with multi-domain identifier (MDI) statistics, where the MDI for each location indicates factors affecting vehicles at the location.

[0211] According to one embodiment, the method 1300 includes obtaining 1320 an expected local route for the vehicle.

[0212] According to one embodiment, steps 1310 and 1320 are followed by step 1330 of identifying, by the processing circuitry, path-related locations based on the expected local path and location-related information.

[0213] According to one embodiment, step 1330 is followed by step 1340 of determining, by the processing circuitry, MDI statistics of the predicted local path for use in at least partially autonomous driving of the vehicle through the predicted local path.

[0214] According to one embodiment, step 1340 includes identifying the most common MDI for the route-related location.

[0215] According to one embodiment, determining the MDI statistics of the expected local route triggers a decision to at least partially autonomously drive the vehicle through the expected local route.

[0216] According to one embodiment, step 1340 is followed by step 1350 of responding to the MDI statistics of the expected local path.

[0217] Step 1350 may include at least one of the following: Inducing a decision to at least partially autonomously drive the vehicle through an expected local path. Inducing the execution of at least partially autonomous driving of the vehicle through an expected local path. · predicting a selected sensory module from the plurality of sensory modules to be utilized for a path-related point during a future time point associated with the predicted local path based on an MDI statistic of the predicted local path. Prefetching selected perception modules into cache memory. · Determining at least partially autonomous driving of the vehicle through an expected local path. · Implementing at least partially autonomous driving of the vehicle through an expected local path.

[0218] According to one embodiment, the decision to at least partially autonomously drive the vehicle through the expected local path may accomplish at least one of the following: · Altering the expected local route. · It is performed without altering the expected local route. Modifying at least the driving parameters (e.g., speed and / or acceleration and / or forward direction) based on the most common MDI along the route. For example, one MDI may be indicative of an obstacle at a particular location, and driving parameters may be set to help go around the obstacle, go over the obstacle at a reduced speed, etc. If the vehicle is expected to be close to an obstacle at a particular time point (PIT), the driving parameters can be pre-modified, or even pre-modified to the particular PIT. Selecting the perception module capable of managing the situation indicated by the MDI statistics of the expected local pathway. Selecting specialized AI modules capable of managing the situation indicated by the MDI statistics of the expected local path.

[0219] FIG. 7D shows an example of the following: Perception module 582. · MDI generator 584. · Elective Unit 587. · 588 Specialized AI Agents. Driving Decision Unit 590. · 1150 prediction units. Memory Management Unit 1152. Processing control unit 1153. A first memory unit 1154. Second memory unit 1155. Processing units 1154(1) to 1154(K). · Perception Module Metadata 1158. · Location and MDI statistics 1380. Expected local route provider 1381. · Route related location identifier 1382.

[0220] The second memory unit 1155 is shown as being smaller than the first memory unit 1154 and is assumed to be a cache memory for simplicity of explanation. There may be more than two memory units.

[0221] The prediction (using MDI statistics of path-related locations) may affect the output driving decision output by the driving decision unit, and / or may affect the memory management unit 1152, which is responsible for managing one or more memory units used for at least partially autonomous driving, and / or may affect the processing management unit 1153, configured to manage processing resources such as processing resources used to implement the perception modules and / or selection units and / or specialized AI agents, etc.

[0222] According to one embodiment, the processing units 1154(1)-1154(K) are processing circuits or portions of processing circuits.

[0223] According to one embodiment, the processing units 1154(1)-1154(K) are configured (e.g., are hardware programmed or configured to) implement (using data and / or metadata stored internally by the processing units and / or stored in one or more of the memory units) the perception module 582, the MDI generator 584, the selection unit 587, the specialized AI agent 588, the driving decision unit 590, the prediction unit 1150, the memory management unit 1152, the processing management unit 1153, the location and MDI statistics 1380, the expected local route provider 1381 and the route related location identifiers 1382.

[0224] According to one embodiment, at least two of the processing units 1154(1)-1154(K) may differ from each other by at least one parameter of a certain amount of computational resources, power consumption, complexity, etc. The selection of which processing unit to activate results in a trade-off between the various parameters of the at least two different processing units.

[0225] Execution of the perception module may require providing perception module metadata (such as coefficients of a model of a neural network) and information (such as information about the domain associated with the perception module) to the processing unit and / or an associated memory unit (e.g. to an internal memory unit of the processing unit or a second memory unit).

[0226] According to one embodiment, the prediction 1151 enables to efficiently control the fetching of information and / or sensory metadata, for example by at least one of (a) predetermining a prefetch operation and thus obtaining a cache hit, and (b) determining an allocation of sensory modules to processing units, which may reduce the amount of configuration of processing circuitry, since it is beneficial to allocate the same processing unit for implementing the same sensory module during successive cycles of operation.

[0227] According to one embodiment, prediction can help reduce the power consumption of a processing unit by reducing the clock frequency of a processing module implementing any unit that may be required to output a result within a few cycles, thereby making the processing unit unused and idle, instead of outputting a result every cycle.

[0228] Location and MDI statistics 1380 stores information about (a) the location associated with the MDI statistic, and (b) the MDI statistic.

[0229] The expected local route provider 1381 is configured to provide an expected local route. The expected local route provider 1381 may determine the expected local route in any manner or may receive the expected local route. The expected local route provider 1381 may be a navigation unit, an ADAS unit, an AV unit, etc.

[0230] The route-related location identifier 1382 may be configured to perform step 1330 .

[0231] In the foregoing specification, the invention has been described with reference to specific examples of embodiments thereof. It will, however, be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0232] Furthermore, where there are terms such as "front," "back," "top," "bottom," "over," "under," and the like in the description and claims, these terms are used for descriptive purposes and not necessarily to describe permanent relative locations. It is understood that the terms so used are interchangeable under appropriate circumstances, such that the embodiments of the invention described herein are capable of being practiced, for example, in orientations other than those shown or otherwise described herein.

[0233] Additionally, the terms "assert" or "set" and "negate" (or "deassert" or "clear") are used herein to refer to the rendering of a signal, status bit, or similar device to its logically true or logically false state, respectively. If the logically true state is a logic level 1, then the logically false state is a logic level 0. And, if the logically true state is a logic level 0, then the logically false state is a logic level 1.

[0234] Those skilled in the art will recognize that the boundaries between logic blocks are illustrative only, and that alternative embodiments may merge logic blocks or circuit elements, or impose alternate decompositions of functionality on various logic blocks or circuit elements. Thus, it should be understood that the architectures illustrated herein are illustrative only, and that in fact many other architectures may be implemented which achieve the same functionality.

[0235] Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved. Thus, any two components herein that combine to achieve a particular functionality may be considered to be "associated with" one another, such that the desired functionality is achieved without regard to the architecture or intermediary components. Likewise, any two components so associated may also be considered to be "operably connected" or "operably coupled" to one another to achieve the desired functionality.

[0236] Moreover, those skilled in the art will recognize that the boundaries between operations described above are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed into additional operations, and operations may be performed with at least partial overlap in time. Additionally, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be altered in various other embodiments.

[0237] Also for example, in one embodiment, the illustrated examples may be implemented as circuit portions located on a single integrated circuit or within the same device. Alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in any suitable manner.

[0238] However, other modifications, variations, and alternatives are also possible.The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

[0239] In the claims, any reference signs placed between parentheses shall not be construed as limiting the scope of the claim. The word "comprising" does not exclude the presence of elements or steps other than those recited in the claim. Furthermore, the terms "a" or "an" as used herein are defined as one or more. Similarly, the use of introductory phrases such as "at least one" and "one or more" in the claims should not be construed as suggesting that the introduction of another claim element by the indefinite article "a" or "an" limits any particular claim containing the claim element so introduced to the invention containing only one such element, even if the same claim also contains the introductory phrase "one or more" or "at least one" and an indefinite article such as "a" or "an". The same applies to the use of definite articles. Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements that such terms describe. Thus, these terms are not necessarily intended to indicate a temporal or other priority of such elements.The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

[0240] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will now occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

[0241] It will be appreciated that various features of the embodiments of the present disclosure that are, for clarity, described in the context of a separate embodiment, may also be provided in combination in a single embodiment. Conversely, various features of the embodiments of the present disclosure that are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.

[0242] It will be appreciated by those skilled in the art that the embodiments of the present disclosure are not limited by what has been particularly shown and described above. Instead, the scope of the embodiments of the present disclosure is defined by the appended claims and equivalents thereof.

Claims

1. A method for managing a group of specialized artificial intelligence (AI) agents for at least partially autonomous driving, The prediction circuit obtains a stream of metadata segments generated at multiple time points and associated with the selection of one or more subgroups of specialized AI agents, wherein the metadata segments are selected from (a) a specialized AI agent identifier and (b) a plurality of multi-domain identifiers, the plurality of multi-domain identifiers representing multiple instances of multi-domain information relating to elements affecting a vehicle for the plurality of time points, and the multi-domain identifier generated at a given time point among the plurality of time points is a combination of class signatures indicating the class of the elements of the multi-domain information associated with the given time point. The prediction circuit finds a segment of the stream that is a predictor for the reception of the next cluster identifier at a future point in time, When finding the aforementioned predictor, (a) Future metadata segments to be received during the aforementioned future time period, (b) A future subgroup of specialized AI agents selected at the aforementioned future point in time, To automatically predict at least one of the following, Methods that include...

2. The method according to claim 1, wherein the automatic prediction described above induces a prediction of the next state of the vehicle at the next time point.

3. The method according to claim 2, comprising determining the vehicle's response to the following states.

4. The method according to claim 1, comprising searching for a new predictor in at least the stream of metadata segments.

5. The method according to claim 1, wherein the automatic prediction of the future metadata segment triggers the execution of a driving-related operation at a time no later than the future time.

6. A non-temporary computer-readable medium for managing a group of specialized artificial intelligence (AI) agents for at least partially autonomous driving, wherein the non-temporary computer-readable medium is The prediction circuit obtains a stream of metadata segments generated at multiple time points and associated with the selection of one or more subgroups of specialized AI agents, wherein the metadata segments are selected from (a) a specialized AI agent identifier and (b) a plurality of multi-domain identifiers, the plurality of multi-domain identifiers representing multiple instances of multi-domain information relating to elements affecting a vehicle for the plurality of time points, and the multi-domain identifier generated at a given time point among the plurality of time points is a combination of class signatures indicating the class of the elements of the multi-domain information associated with the given time point. The prediction circuit finds a segment of the stream that is a predictor for the reception of the next cluster identifier at a future point in time, When finding the aforementioned predictor, (a) Future metadata segments to be received during the aforementioned future time period, (b) A future subgroup of specialized AI agents selected at the aforementioned future point in time, To automatically predict at least one of the following, A non-temporary, computer-readable medium for storing instructions for [a specific purpose].

7. The non-temporary computer-readable medium according to claim 6, wherein the automatic prediction triggers a prediction of the next state of the vehicle at the next time point.

8. A non-temporary computer-readable medium according to claim 7, comprising determining the response of the vehicle to the following states.

9. The non-temporary computer-readable medium according to claim 6, comprising searching for new predictors in at least the stream of metadata segments.

10. The non-temporary computer-readable medium according to claim 6, wherein the automatic prediction of the future metadata segment triggers the execution of a driving-related operation at a time not exceeding the future time.

Images