Perception and prediction based driving

By acquiring and selecting appropriate narrow AI agents in the prediction circuit, the problem of high resource consumption of autonomous vehicle perception systems is solved, and efficient driving decisions and resource optimization are achieved.

CN118982804APending Publication Date: 2024-11-19ULTRABERRY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410564533.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2023-08-31
Filing Date
2024-05-08
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

The perception systems of existing autonomous vehicles are computationally intensive and resource-consuming when processing large amounts of original sensor inputs, making it difficult to effectively manage narrow AI agents to achieve efficient driving decisions.

Method used

The prediction circuit obtains the stream of metadata fragments, selects the relevant narrow AI agent, and predicts the received metadata fragments and subgroups at future time points to automatically select the appropriate narrow AI agent, and optimizes the use and storage resources of the perception module.

Benefits of technology

Reduces the consumption of computing and storage resources, improves the accuracy and efficiency of driving decisions, reduces power consumption, and achieves a smoother driving operation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118982804A_ABST
    Figure CN118982804A_ABST
Patent Text Reader

Abstract

A method for managing a set of restricted artificial intelligence (AI) agents, the restricted artificial intelligence agents for at least partially autonomous driving, the method comprising: (i) obtaining, by prediction circuitry, a stream of metadata fragments generated at a plurality of points in time and associated with a selection of one or more subgroups of the set of restricted AI agents; wherein the metadata segment is selected from (a) a selected narrowly-defined AI proxy identifier, and (b) a plurality of multi-domain identifiers representing a plurality of instances of multi-domain information relating to the plurality of points in time regarding the vehicle element; wherein the multi-domain identifier generated at a given time point in the plurality of time points is a combination of class signatures, and the combination of class signatures represents classes of elements of the multi-domain information associated with the given time point; (ii) finding, by prediction circuitry, a segment of the stream, the segment of the stream being a predicted value for receiving a next cluster identifier at a future point in time; and (iii) when the predicted value is found, automatically predicting at least one of (c) a future metadata segment to be received during the future time point, or (d) a future subset of narrowly-defined AI agents selected at the future time point.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a method for managing a group of narrow artificial intelligence (AI) agents, which are used for at least partially autonomous driving. Background Art

[0002] The autonomous vehicle includes a perception system that performs semantic decomposition (SD). Here, the AV flow consists of three modules: a perception module, a route planning module, and an actuation module (brake / accelerate / steering, etc.). The perception module feeds its output to the route planning module, and the route planning module feeds its output to the actuation module.

[0003] Since the route planning module and the actuation module do not have access to the raw sensor inputs, the role of perception in the SD architecture is to exhaustively label the raw inputs from the car's sensors, such as detecting cars and pedestrians, demarcating free drivable spaces, identifying speed limits through road signs, and all other raw inputs in the scene that may affect driving in that scene. Since there are so many objects / entities that need to be recognized and accurately located within the scene, the SD perception module tends to be a complex, computationally intensive model. Summary of the invention

[0004] The present invention discloses a method for managing a group of narrow artificial intelligence (AI) agents for at least partially autonomous driving, the method comprising:

[0005] Obtaining, by prediction circuitry, a stream of metadata segments, the stream of metadata segments generated at a plurality of time points and associated with a selection of one or more subsets of the set of narrow AI agents; wherein the metadata segments are selected from: (a) a selected narrow AI agent identifier, and (b) a plurality of multi-domain identifiers, the plurality of multi-domain identifiers representing a plurality of instances of multi-domain information about vehicle-affecting elements associated with the plurality of time points; wherein the multi-domain identifier generated at a given time point in the plurality of time points is a combination of class signatures, the combination of class signatures representing a class of elements of the multi-domain information associated with the given time point;

[0006] Finding, by the prediction circuit, a segment of the stream, the segment of the stream being a predicted value for receiving a next cluster identifier at a future time point; and

[0007] When the predicted value is found, at least one of the following is automatically predicted:

[0008] (a) future metadata segments to be received during the future time point;

[0009] (b) A future subgroup of narrow AI agents that will be selected at said future time point. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The embodiments of the present disclosure will be more fully understood and appreciated through the following detailed description in conjunction with the accompanying drawings, in which:

[0011] Figure 1 Figure shows an example of a system;

[0012] Figure 2 Figure shows an example of a system;

[0013] Figure 3 illustrates an example of a method; and

[0014] Figure 4 The diagram shows Figure 3 Examples of steps of the method;

[0015] Figure 5A An example of the method is shown;

[0016] Figure 5B An example of an implementation of the method is shown;

[0017] Figure 5C The figure shows an example of an image;

[0018] Figure 5D An example of a vehicle is shown;

[0019] Figure 5E The figure shows an example of an image;

[0020] Fig. 5F The figure shows an example of an image;

[0021] Figure 5G The figure shows an example of an image;

[0022] Fig. 6A The figure shows an example of an image;

[0023] Figure 6B The figure shows an example of prediction;

[0024] Figure 6C The diagram shows examples of a number of different units;

[0025] Fig. 7A The figure shows an example of a unit;

[0026] Figure 7B Figure shows an example of MDI statistics;

[0027] Figure 7C illustrates an example of a method; and

[0028] Fig.7D Examples of a number of different units are shown. DETAILED DESCRIPTION

[0029] In the detailed description below, many specific details are described to provide a thorough understanding of the present invention. However, it will be appreciated by those skilled in the art that the present invention can be implemented without these specific details. In other examples, well-known methods, steps and components are not described to avoid confusing the present invention.

[0030] The subject matter of the invention is particularly pointed out and distinctly claimed in the concluding portion of the specification. However, the invention as to its organization and method of operation, together with objects, features, and advantages thereof, may best be understood by reference to the following detailed description when read in connection with the accompanying drawings.

[0031] It should be understood that for simplicity and clarity of illustration, the elements shown in the figures are not necessarily drawn to scale. For example, for clarity, the size of some of the elements may be exaggerated relative to other elements. In addition, where deemed appropriate, reference numerals may be repeated between different figures to indicate corresponding or similar elements.

[0032] Since most of the illustrated embodiments of the present invention can be implemented using electronic components and circuits known to those skilled in the art, in order to understand and appreciate the basic concepts of the present invention and in order not to confuse or deviate from the teachings of the present invention, the details will not be explained to a greater extent than is deemed necessary as described above.

[0033] Any reference to a method in the specification should also apply, with appropriate changes, to an apparatus or system capable of executing the method and / or a non-transitory computer-readable medium storing instructions for executing the method.

[0034] Any reference in the specification to a system or apparatus should also apply, mutatis mutandis, to a method executable by the system and / or, mutatis mutandis, to a non-transitory computer-readable medium storing instructions executable by the system.

[0035] Any reference to a non-transitory computer-readable medium in the specification should also apply, if appropriately modified, to an apparatus or system capable of executing instructions stored in the non-transitory computer-readable medium, and / or, if appropriately modified, to a method of executing the instructions.

[0036] Any modules or units listed in any drawings, any part of the specification and / or any claims may be combined in any way.

[0037] Any of the perception unit, narrow AI agent, and driving decision unit can be implemented as hardware and / or codes, instructions, and / or commands stored in non-temporary computer-readable media, and can also be included in a vehicle, outside a vehicle, in a mobile device, a server, etc.

[0038] The vehicle may be any type of vehicle, such as a ground transportation vehicle, an air vehicle, and a water vehicle.

[0039] The specification and / or drawings may refer to images. Images are examples of media units. Any reference to images, as appropriate, may also apply to media units. Media units may be examples of sensed information. Any reference to media units, as appropriate, may also apply to any type of natural signal, such as but not limited to naturally generated signals, signals representing human behavior, signals representing operations related to the stock market, medical signals, financial series, geodetic signals, geophysical, chemical, molecular, text and digital signals, time series, etc. Any reference to media units, as appropriate, may also apply to sensed information. The sensed information may be of any kind and may be sensed by any type of sensor, such as a visible light camera, an audio sensor, a sensor that can sense infrared, radar images, ultrasound, electro-optics, radiography, LIDAR (light detection and ranging), etc. The sensing may include generating samples (e.g., pixels, audio signals) representing the signal transmitted or otherwise reaching the sensor.

[0040] The description and / or drawings may refer to span elements. The span elements may be implemented as software or hardware. Different span elements of a certain iteration are configured to apply different mathematical functions based on the inputs they receive. Non-limiting examples of mathematical functions include filtering, although other functions may also be applied.

[0041] The specification and / or drawings may refer to a conceptual structure. A conceptual structure may include one or more clusters. Each cluster may include a signature and associated metadata. Each reference to one or more clusters may apply to a reference to a conceptual structure.

[0042] The description and / or drawings may refer to a processor. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a full custom integrated circuit, etc., or a combination of such integrated circuits.

[0043] Any steps of any methods shown in the specification and / or drawings may be combined arbitrarily.

[0044] Any subject matter of any claims may be combined in any manner.

[0045] The systems, units, components, processors, and sensors shown in the specification and / or drawings may be arbitrarily combined.

[0046] Any reference to objects can also be applied to patterns. Therefore - any reference to object detection can also be applied to pattern detection with appropriate changes.

[0047] A situation can be a single location / attribute combination at a point in time. A scenario is a series of events that occur logically within a causal frame of reference. Any reference to a scenario should also apply to a situation, mutatis mutandis.

[0048] The sensed information unit may be sensed by one or more sensors of one or more types. The one or more sensors may belong to the same device or system, or may belong to different devices or systems.

[0049] A perception unit may be provided, and the one or more sensors and / or one or more interfaces may be present before the perception unit to receive one or more sensed information units. The perception unit may be configured to receive sensed information units from an I / O interface and / or from a sensor. The perception unit may be followed by a plurality of narrow AI agents, also referred to as a collection of narrow AI agents.

[0050] An artificial intelligence (AI) agent may refer to an autonomous entity that observes its environment (i.e., it is intelligent) through sensors and corresponding actuators to direct its activities toward achieving a goal (i.e., it is an agent). An intelligent agent may also learn or use knowledge to achieve its goals. An intelligent agent may be very simple or very complex. A reflex machine, such as a thermostat, is considered an example of an intelligent agent (www.wikipedia.org).

[0051] The sensed information unit may or may not be processed before reaching the sensing unit. Any processing - filtering, noise reduction, etc. - may be performed.

[0052] The number of narrow AI agents may be, for example, more than 100, more than 500, more than 1000, more than 10,000, more than 100,000, etc. A larger number of narrow AI agents may provide more accurate driving decisions.

[0053] AI narrow agents are narrow in the sense that they are not trained to respond to all feasible (or all possible, or most) scenarios handled by the entire collection. For example, each AI narrow agent can be trained to respond to a portion of the scenarios managed by the entire collection (e.g., less than 1 percent). Narrow AI agents can be trained to respond to only some factors or elements or parameters or variables that form a scenario.

[0054] Narrow AI agents may have the same complexity and / or the same parameters (depth, energy consumption, technical implementation) - but at least some narrow AI agents may differ from each other in at least one of complexity and / or parameters.

[0055] Narrow AI agents can be trained in a supervised and / or unsupervised manner.

[0056] One or more narrow AI agents may be a neural network or may be other than a neural network.

[0057] The set may include one or more sensors and any other entity for generating a sensed information unit, and / or the set may receive (through an interface) one or more sensed information units from one or more sensors.

[0058] The perception unit may process the one or more sensed information units and determine which narrow AI agents are relevant to the processing of the one or more sensed information units.

[0059] An autonomous vehicle system may be provided that may classify observed scenarios into a plurality of coarse-grained categories using a perception unit. The system may include an ensemble of narrow AI agents (EoN).

[0060] The perception unit can receive and / or generate anchor nodes, which once detected by the perception unit can influence the selection of which narrow AI agents are selected. The number of anchor nodes may be large (e.g., -100, 500, 1000, 10,000, 20,000, 50,000, 100,000 or more anchor nodes or even more).

[0061] For a given scene (which may be represented by one or more sensed information units such as, but not limited to, one or more images), the perception unit may detect one or more anchor nodes.

[0062] The detected anchor nodes can provide enough context clues to allow the perception unit to decide which are the relevant narrow AI agents.

[0063] Background cues may be high-level context of the sensed information unit. Background cues are high-level in the sense that determination of background cues is less complex and / or requires less computational resources than performing object detection of small objects in the sensed information unit. Small objects may have a minimum size to be detected, may, for example, have a size of tens of pixels, may have a size that is less than 0.1, 0.5, 1, 2, 3 percent of the sensed information unit, etc. Determination of background cues, for example, may not include determining the exact location of each object in the image - including the location of objects that appear as tens of pixels in the image.

[0064] By searching a high level context of sensed information units, the power consumption of the perception unit can be much lower (e.g., even two orders of magnitude lower) than that of prior art systems that are built to perform the entire process of object detection and determining what driving maneuver to perform).

[0065] At least some of the power savings may be attributable to the fact that a high level of sensed information element context may not include location information, no need to determine whether objects of different sizes are the same type of object, and the like.

[0066] Narrow AI agents may receive input directly from sensors (e.g. - as output of a perception module) and provide proposed behaviors (desired vehicle control parameters - and desired vehicle behavior - also known as driving decisions) in the form of outputs, such as steering wheel angle, acceleration / brake signals, or control of any other aspect of driving, etc.

[0067] The outputs from the different selected narrow AI agents are fed to a driving decision unit (also called a coordinator), which outputs one or more output driving decisions, such as one or more commands or requests or suggestions, to various modules of the vehicle and / or the driver.

[0068] The coordinator may apply any method for generating one or more output driving decisions, such as one or more commands and requests, based on outputs from different selected narrow AI agents.

[0069] These methods may include arbitration, competition, selecting responses based on risk imposed by employing the output of a narrow AI agent, and the like.

[0070] Returning to the perception unit, non-limiting examples of anchor nodes are listed below:

[0071] Approaching a crosswalk. Detecting a pedestrian on a sidewalk triggers different behaviors, depending on how close or far away the intersection is. If a pedestrian is standing on the sidewalk next to the intersection, the car should slow down (in case the pedestrian will start to cross the street). The same pedestrian on a sidewalk in other areas can be safely ignored. Therefore, a different narrow agent may be needed to drive through a crosswalk.

[0072] Approaching a pothole in the road. This could trigger the car to slow down or change lanes.

[0073] Limited visibility and slippery road conditions - a slower speed and / or a greater safe distance from the vehicle ahead may be prescribed.

[0074] • Encoded navigation signals - When crossing an intersection, the narrow agent should get a signal whether we are going straight, turning left or right through the intersection. This anchor node / signal will change the original meaning of "stay in lane".

[0075] • Change Lane - This anchor node triggers a specific narrow agent responsible for changing lanes to the left or right.

[0076] Approaching a merging road with different traffic signs - do I give the right of way, or do I have priority?

[0077] Unsafe drivers nearby: Tailgating, erratic driving, and signs of road rage may trigger switching to the rightmost lane, slowing down, and letting the dangerous car pass.

[0078] A police car is following you with its emergency lights flashing: switch to the far right lane and stop on the shoulder.

[0079] Car in from performs emergency braking: a narrow agent is activated, which performs emergency braking or turns to the shoulder of the road and stops there with a lower deceleration.

[0080] Anchor nodes may be selected, generated and / or learned in a variety of ways - manually, automatically, human labeling based on input, autonomous labeling based on input, based on manual recognition, scenario-based labeling (e.g., approaching a roundabout, approaching a crosswalk, environmental conditions, presence of road users) - in which case corresponding expected behaviors may be designed based on recorded human driver behavior in such situations, identifying unfamiliar or poorly predicted human driving patterns. These patterns may suggest that a world state is not well covered by existing narrow agents, and therefore new agents are assigned.

[0081] A narrow AI agent may be or may include a simple model (e.g. - a neural network) that receives raw (or pre-processed in any way) sensor data as its input, processes it internally, and outputs a proposed behavior.

[0082] Examples of narrow AI agents are listed below:

[0083] A narrow AI agent that implements the steering behavior of “staying in the lane” when not in an intersection. The narrow AI agent can be a convolutional neural network that maps the road image taken by the front camera into the steering angle of the vehicle.

[0084] A narrow AI agent that implements the steering behavior of "staying in lane" at a left-turn intersection. Note that at an intersection, "staying in lane" has different meanings if we are driving straight or turning left. The narrow agent for left turns receives images from the front and left-view cameras and maps them into steering angles. By definition, the information instructing the agent to turn left instead of driving straight is either received as input from the navigation block or encoded as an "anchor node". This narrow AI agent can be implemented as a CNN.

[0085] A narrow AI agent that implements the “change to left lane” behavior. This narrow AI agent accesses the left-facing camera and proposes the actions required to change lanes, i.e., deciding when to change lanes, the acceleration before and during the change, and the steering to achieve the lane change.

[0086] A narrow AI agent that turns the vehicle when it stops at a red traffic light. This narrow AI agent is implemented as an "if" conditional and simply keeps the steering angle constant.

[0087] Narrow AI agent implementing “keep distance” logic: This narrow AI agent maps images from the front-facing and rear-facing cameras into acceleration / braking signals.

[0088] Narrow AI agent that implements “yield at crosswalk” - This narrow AI agent stops when there is a pedestrian in a crosswalk, or slows down when a pedestrian is nearby but not in a crosswalk. Uses input from the front-facing camera and maps it into brake / accelerate signals.

[0089] Examples of how a coordinator might process output from one or more narrow AI agents:

[0090] ·Combined independently.

[0091] The narrow AI agent responsible for steering angles to keep the car in its lane is largely independent of driving speed. Therefore, the behaviors proposed by this agent can be independently combined with the acceleration / braking behaviors of the "keep distance" narrow agent.

[0092] Override

[0093] The vehicle should stay in the lane, but the car in front of us stops suddenly and there is not enough braking distance to stop. Here, the desire to avoid a crash takes precedence over the behavior of "staying in the lane" and we turn to the shoulder and stop there.

[0094] Animals in the Road - abandon "Stay in Lane" and perform an evasive maneuver.

[0095] ·"modulation".

[0096] Slippery roads will cause "keep in lane" and "keep distance" behaviors to remain almost unchanged, but will dictate slower speeds and greater safety distances.

[0097] · Execute risk reduction optimization.

[0098] The system is expected to outperform the SD system in all aspects, whether in terms of behavioral accuracy, model size and complexity, and computational intensity.

[0099] The perception unit can work in a more efficient way than the perception systems of the prior art. The perception systems of the prior art analyze the scene comprehensively, and their goal is to reconstruct the scene, that is, to pay attention to all the details of the scene, such as - accurately analyzing all the subjects on the road (pedestrians, cars, bicycles, lanes, traffic signs, traffic lights, trees, obstacles, etc.) and their attributes (accurate position, posture, distance, angle, etc.). This is a very power- and resource-intensive process - because its purpose is to generate an extensive environmental model as an input to the policy module.

[0100] The perception unit does not need to focus on all the details and identify each agent - it only needs to analyze and classify the situation in order to activate the relevant agent. This is a very simple process in terms of performance. Once the relevant agent is activated, it will analyze the relevant features of the input image.

[0101] It should be noted that even when the number of narrow AI agents is large, some can be stored in RAM, while others can be saved in non-volatile storage (e.g., on disk, in cheap non-volatile memory, etc.) - and the narrow AI agents can be retrieved when needed.

[0102] The perception module can be very fast, since its job is to detect an anchor node and select the relevant narrow AI agent based on that anchor node. Each narrow AI agent can also be very fast, however our number of narrow AI agents is very large. Fortunately, at any given point in time, only a few of them need to actually run, i.e., at any given point in time, the running time is very short.

[0103] The proposed unit may be executed or hosted by a processor. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a full custom integrated circuit, etc., or a combination of such integrated circuits.

[0104] Figure 1 An example of a system 10 is shown.

[0105] The system 10 includes a perception unit, such as a perception router 30 (which receives a sensed information unit, which is an image 8 having pixels), a set 40 of narrow AI agents (e.g., narrow AI agents associated with a roundabout, a pedestrian walking across a zebra crossing, a specific traffic sign, or a traffic jam), and a driving decision unit, such as a coordinator 50. The driving decision unit 50 can control and / or communicate with a response unit 60.

[0106] Figure 2 An example of a system 10 ′ is shown.

[0107] The system 10' includes an acquisition unit 20' (for receiving one or more sensed information units 15), a perception unit 30', narrow AI agents 40(1)-40(K), K being the number of narrow AI agents, and a driving decision unit 50'. The driving decision unit can control and / or communicate with a response unit 60', such as a braking control system, any vehicle computer, an autonomous driving module, an ADAS driving module, etc.

[0108] Figure 3 A method 300 for operating a collection of narrow AI agents associated with a vehicle is illustrated.

[0109] By vehicle related it is meant that the output of method 300 may be one or more driving decisions which, when implemented, may affect the driving of the vehicle.

[0110] The method may include various steps, some of which may include providing desired driving decisions (e.g., during training of any portion of the entities used in the process of method 300, which entities may include a perception unit, a narrow AI agent, and a driving decision unit).

[0111] Additionally or alternatively, driving decisions associated with any sensed information unit fed to any entity may be provided, and the method may include determining which driving decision is correct. For example, this may be determined using statistical information, such as taking the most common driving decision for each situation and / or scenario, or any parameters therein.

[0112] Method 300 may begin at initialization step 310 .

[0113] Step 310 may include acquiring a perception unit, a narrow AI agent, and a driving decision unit configured to perform the various steps of method 300 .

[0114] The acquiring step may include receiving after being trained, and / or training and / or receiving at any stage in the training process, downloading instructions or otherwise configuring the computerized system to perform any other steps of method 300 .

[0115] Step 310 may include at least one of the following: (a) training at least one of the perception unit, the narrow AI agent, and the driving decision unit, (b) receiving at least one of the perception unit, the narrow AI agent, and the driving decision unit that has been trained, and (c) configuring at least one of the perception unit, the narrow AI agent, and the driving decision unit in other ways.

[0116] Step 310 may be followed by step 320 of acquiring one or more sensed information units.

[0117] The acquisition step may include sensing, sensorless reception, pre-processing, etc.

[0118] Step 320 may be followed by step 330 of determining, by the perception unit and based on the one or more sensed information units, a set of one or more related narrow AI agents that may be related to processing of the one or more sensed information units. The entire set may be related to the first plurality of scenarios.

[0119] Each related narrow AI agent can be associated with a dedicated class. The class can be associated with an anchor node. Step 330 can include searching for an anchor node.

[0120] Each class may be defined by at least a portion of one or more scenes, wherein the at least a portion of the one or more scenes may be a small portion of the first plurality of scenes.

[0121] Different narrow AI agents can be trained to respond to different scenarios, which may be (or may include) T-junctions, different road elements, zebra crossings, roundabouts, obstacles, different environmental conditions, rain, fog, night, straight on the highway, going up a hill, traffic jams, ...). Examples of different obstacles and / or different road elements are described in PCT patent application WO2020 / 079508 entitled "Obstacle Detection Method and System", which is incorporated herein in its entirety.

[0122] Different scenarios may be different situations or may differ depending on the situation.

[0123] The scenario may be, for example, at least one of: (a) the location of the vehicle, (b) one or more weather conditions, (c) one or more background parameters, (d) road conditions, (e) traffic parameters.

[0124] Various examples of road conditions may include the roughness of the road, the level of maintenance of the road, the presence of potholes or other relevant road obstacles, whether the road is wet, covered with snow or other particles.

[0125] Various examples of traffic parameters and one or more context parameters may include time (hour, day, period or year, certain hours of certain days, etc.), traffic load, distribution of vehicles on the road, behavior of one or more vehicles (aggressive, calm, predictable, unpredictable, etc.), presence of pedestrians near the road, presence of pedestrians near the vehicle, presence of pedestrians away from the vehicle, behavior of pedestrians (aggressive, calm, predictable, unpredictable, etc.), risk associated with driving near the vehicle, complexity associated with driving near the vehicle, presence of one of kindergartens, schools, crowds, etc. (near the vehicle). Context parameters may relate to the context of the sensed information - the context may depend on or relate to the conditions of the setting in which the event, statement or idea was formed.

[0126] The relevant narrow AI agent can be trained to respond to one or more of a large number of situations. Examples of situations and situation-based processing are described in U.S. Patent Application 16 / 035,732, which is incorporated herein by reference.

[0127] Each class may be defined by anchor nodes which may be background cues. Step 320 may be performed if no object is detected which may be below a predetermined number of pixels. The predetermined number of pixels may not exceed several tens of pixels.

[0128] The narrow AI agent can be an end-to-end narrow AI agent.

[0129] For at least some narrow AI agents, the corresponding portion may be less than one percent of the first plurality of scenes.

[0130] Step 330 may be followed by step 340 of sending one or more sensed information units to the associated narrow AI agent.

[0131] It should be noted that once the one or more relevant narrow AI agents are determined, they can be uploaded to a processor and / or storage unit. This reduces the RAM or other storage resources required to store and execute step 350.

[0132] Step 340 (or method 300) may include maintaining at least one unrelated narrow AI agent in a low power mode (idle, not activated, in sleep mode, partially operational, etc.), wherein the power consumption of the at least one unrelated narrow AI agent may be lower than the power consumption of the related narrow AI agent.

[0133] Step 340 may be followed by step 350 of processing the one or more sensed information units by one or more associated narrow AI agents to provide one or more narrow AI agent driving decisions. Each narrow AI agent may be associated with a respective portion of the first plurality of scenarios managed by the entire collection.

[0134] Narrow AI agent driving decisions can be commands, requests, or suggestions for autonomously controlled vehicles.

[0135] Narrow AI agent driving decisions can be advanced driver assistance system (ADAS) commands, ADAS requests, or ADAS recommendations.

[0136] Step 350 may be followed by step 360 , which is processing one or more narrow AI agent driving decisions by a driving decision unit to provide an output driving decision.

[0137] Step 360 may include averaging one or more narrow AI agent driving decisions, or applying one or more functions (e.g. - predefined and / or learned and / or time varying), and / or applying one or more strategies to one or more narrow AI agent driving decisions. Example of strategy - attempting to reduce risk - e.g. where two narrow AI agent driving decisions have different speeds - selecting the lower speed.

[0138] Method 300—particularly steps 320 - 360 , may be repeated multiple times—eg, multiple times per second, once per second, once every few seconds, continuously or discontinuously.

[0139] Figure 4 An example of step 310 is shown.

[0140] Step 310 may include step 312 of training the perception unit to classify the sensed information unit.

[0141] Each class may be at least a portion of one or more scenes, which may be a small portion of the first plurality of scenes.

[0142] Each class may be associated with an anchor node. The anchor node may be used to classify the sensed information unit into the class.

[0143] Step 312 may include receiving, by the perception unit, definitions of at least some of the classes prior to training. This may include, for example, receiving labels or any other class-defining information.

[0144] Step 312 may include defining, by the perception unit, at least some of the classes.

[0145] This definition may include, for example, generating signatures and clustering the signatures into conceptual structures, such as clusters. Clustering actually defines classes.

[0146] These clusters can all be of the same level, or can be arranged in a hierarchical fashion. Clustering can essentially respond to the statistics of contextual cues - more frequently occurring contexts can be split into more clusters. Larger clusters can be split into lower level clusters - in any arbitrary way - for example by cross correlations between cluster members, by finding shared signature parts and unique signature parts, etc.

[0147] Step 312 may include performing unsupervised training.

[0148] At least a portion of one or more scenarios may be at least one of: (a) one or more factors of the scenario, (b) one or more elements of the scenario, (c) one or more parameters of the scenario, and (d) one or more variables of the scenario.

[0149] Step 312 may include feeding a first data set of sensed information units to a perception unit.

[0150] Step 312 may be followed by step 314 of classifying the sensed information units of the second data set using the trained perceptual unit. These sensed information units are referred to as second sensed information units.

[0151] Step 314 may also include feeding one or more driving decisions for each class.

[0152] Each class will include a plurality of second sensed information units and one or more driving decisions.

[0153] Each narrow AI agent may be associated with a dedicated class. Step 314 may be followed by step 316, which is to train each narrow AI agent to output a narrow AI agent driving decision associated with the dedicated class.

[0154] The training of step 316 may include feeding the narrow AI agent associated with a given class the second sensed information and one or more driving decisions of that class.

[0155] Step 316 may be performed in a supervised or unsupervised manner. Supervised training may include providing one or more driving decisions as a requested output of the narrow AI agent.

[0156] Step 316 may be followed by step 318 of training the driving decision unit to provide an output driving decision based on one or more narrow AI agent driving decisions.

[0157] Step 318 may include feeding the sensed information unit of the third data set (hereinafter referred to as the third sensed information unit) to the perception unit, thereby allowing the perception unit to determine the relevant narrow AI class (based on the class of the third sensed information unit), allowing the relevant narrow AI class to output the narrow AI agent driving decision, and feeding the narrow AI agent driving decision and the driving decision associated with the third data set to the driving decision unit.

[0158] The first data set, the second data set and the third data set or each of them may include any number of sensed information units and may be generated in any manner. The first data set, the second data set and the third data set may include randomly selected sensed information units or any combination of sensed information units.

[0159] A non-transitory computer-readable medium storing instructions for operating a set of narrow AI agents may be provided, the operations may include: obtaining one or more sensed information units; determining, by a perception unit and based on the one or more sensed information units, one or more related narrow AI agents in the set that are related to processing of the one or more sensed information units; wherein the set is related to a first plurality of scenarios; processing the one or more sensed information units by the one or more related narrow AI agents to provide one or more narrow AI agent driving decisions; wherein each narrow AI agent is related to a corresponding portion of the first plurality of scenarios; and processing the one or more narrow AI agent driving decisions by a driving decision unit to provide an output driving decision.

[0160] A system for driving decision determination, the system may include a set of AI narrow agents, a perception unit, and a driving decision unit, wherein the perception unit is configured to: obtain one or more sensed information units; and based on the one or more sensed information units, determine one or more related narrow AI agents in the set that are related to the processing of the one or more sensed information units; wherein the set is related to a first plurality of scenarios; wherein one or more related narrow AI agents are configured to process the one or more sensed information units to provide one or more narrow AI agent driving decisions; wherein each narrow AI agent is related to a corresponding portion of the first plurality of scenarios; and the driving decision unit is configured to process the one or more narrow AI agent driving decisions to provide an output driving decision.

[0161] Using multiple perception modules

[0162] Elements that affect the vehicle may belong to different domains. The inventors have found that assigning a perception module to each domain instead of using a single perception unit provides various technical benefits, such as:

[0163] Reduced computational and / or storage resources allocated to narrow AI agent selection - since all perception modules are smaller than a single perception unit that manages all information domains.

[0164] Reduced computational and / or storage resources allocated to training – because each perception module is trained using more compact information and covers fewer classification scenarios.

[0165] Providing more accurate narrow AI agent choices - Perception modules assigned to a domain can better adapt to choices relevant to that domain. This does not force the perception module to make trade-offs between different information domains.

[0166] Simplify adaptation to new information domains - this requires training or adding already trained perception modules that are specialized for the new information domain.

[0167] • Changing the selection of a perception module (among a set of perception modules) in a dynamic manner when the information domain changes.

[0168] • Keeping the unselected sensing modules in the group in an idle state - thus reducing power consumption.

[0169] Retrieving information about selected perception modules - while avoiding retrieving information about unselected perception modules, which reduces power consumption and / or reduces storage resources allocated to narrow AI agent selection.

[0170] Figure 5A An example of a method 500 of operation of a narrow artificial intelligence (AI) agent for at least partially autonomous driving is illustrated.

[0171] According to an embodiment, method 500 comprises a step 510 of automatically selecting a plurality of perception modules from a set of perception modules.

[0172] According to an embodiment, the selection is performed based on a domain associated with the multi-domain information.

[0173] According to an embodiment, the selection is made based on the output of a previously generated perception module in the set of perception modules. For example, assuming that there is not enough light (which can be sensed by image processing, position and time information, such as driving on an unlit road in an unlit environment at a moonless night), the image processing-based perception module may not be started.

[0174] According to an embodiment, at least two different perception modules are associated with two different vehicle sensors. The selection may be made based on the operating states of the two different vehicle sensors. For example, assuming that one vehicle sensor fails or is found to generate unreliable sensed information, the perception route associated with the vehicle sensor may not be selected, leaving it in an idle mode, etc.

[0175] According to an embodiment, step 510 may include or may be followed by the following step: maintaining the sensing modules in the group other than the plurality of sensing modules in an idle mode.

[0176] According to an embodiment, the plurality of perception modules are a plurality of perception routers. According to an embodiment, the plurality of perception modules are a plurality of perception sub-routers associated with a perception router. Examples of perception routers are described in U.S. Patent Application 18 / 036,150, which is incorporated herein by reference.

[0177] According to an embodiment, step 510 is followed by step 520 of receiving multi-domain information about elements affecting the vehicle through a plurality of perception modules. Each perception module in the plurality of perception modules is associated with a dedicated domain of the multi-domain information.

[0178] According to an embodiment, step 520 is followed by step 530 of generating a class signature of a class representing an element of multi-domain information by a plurality of perception modules.

[0179] According to an embodiment, the perception module is a classifier configured to classify information of a domain associated with the perception module. The classifier may implement a machine learning process, may be a deep neural network (DNN), may be a neural network different from a DNN, may be different from a neural network, etc.

[0180] According to an embodiment, step 530 is followed by step 540 of determining a multi-domain identifier (MDI) that identifies the generated class signatures of multiple perception modules. The MDI may be a concatenation of class signatures, or may identify the generated class signatures in any other manner.

[0181] According to an embodiment, step 540 is followed by step 550 of identifying, based on the MDI, one or more narrow AI agents associated with processing of at least a portion of the MDI.

[0182] According to an embodiment, one or more narrow AI agents are associated with a cluster of MDIs. For example, a first narrow AI agent may be associated with a first group of one or more MDI clusters, and a second narrow AI agent may be associated with a second group of one or more MDI clusters. The number of MDIs in the first group may be the same as the number of MDIs in the second group. The number of MDIs in the first group may be different from the number of MDIs in the second group. The first group and the second group may share one or more MDIs. The first group and the second group may not include any shared MDIs.

[0183] A cluster of narrow AI agents or MDIs may be associated with a skill. The selection of one or more narrow AI agents may actually include selecting one or more skills required to deal with a situation or scenario.

[0184] According to an embodiment, the multi-domain information is input from at least one of the following:

[0185] Road setting information related to static objects within the vehicle's environment. Examples of static objects include road elements (lanes, curves, intersections, roundabouts), static elements other than road elements (houses, trees), location information (city, country, urban environment, non-urban environment).

[0186] • Road user information relating to movable road users within the environment. Examples of movable road users include pedestrians, other vehicles. The road user information may include information about the location and / or behavior of road users.

[0187] Traffic rule indication information related to visual traffic rule indicators within the environment. Traffic rule indications may be traffic signs, content written on the road and indicating traffic rules (e.g. lane boundaries, zebra crossings, arrows, text, symbols indicating permitted directions of travel, etc.).

[0188] Regulatory information related to legal constraints regarding the environment. Legal constraints may include maximum speed, minimum speed, etc. Regulatory information may be obtained from location-based regulatory information or from other sources.

[0189] • Environmental condition information relating to at least one of weather and light conditions of the environment.

[0190] Vehicle state information related to the motion-related state of the vehicle.

[0191] According to an embodiment, the plurality of perception modules may include at least one of the following:

[0192] Road setting perception module. The road setting perception module may be configured to generate a class signature indicating one or more classes of one or more static objects in the vehicle's environment. The class signature may indicate a scene including all static objects that (i) may affect the vehicle and (ii) are within the vehicle's environment.

[0193] Road User Awareness Module. The road user awareness module may be configured to generate a class signature indicating classes associated with movable road users. The class signature may indicate a scene including all movable road users that (i) may affect the vehicle and (ii) are within the environment of the vehicle.

[0194] Traffic rule perception module. The traffic rule perception module may be configured to generate a class signature indicating a class of traffic rules indicated by traffic signs or other elements captured in the image. The class signature may indicate a scene including all traffic rules that (i) may affect the vehicle and (ii) are indicated by traffic rule indicators within the vehicle's environment.

[0195] · Regulatory awareness module. The regulatory awareness module may be configured to generate a class signature indicating a class of legal constraints. The class signature may indicate a scenario including all legal constraints that (i) may affect the vehicle and (ii) are applicable to the environment of the vehicle.

[0196] • Environmental condition perception module. The environmental condition perception module may be configured to generate a class signature indicative of a class of environmental conditions within the vehicle's environment, such as at least one of weather and light conditions of the environment.

[0197] Vehicle state perception module. The vehicle state module may be configured to generate a class signature indicating a vehicle state, in particular, a class indicating a motion-related state of the vehicle. The motion-related state may include kinematic information, velocity information, acceleration information, etc.

[0198] It should be noted that any other perception modules may be provided and at any given point in time one or more of the perception modules may be selected.There may be any number of domains and any number of perception modules assigned to the domains.

[0199] According to an embodiment, the identification of step 550 triggers execution of at least a portion of further processing of the multi-domain information by the identified one or more narrow AI agents to provide one or more narrow AI driving-related decisions.

[0200] Figure 5B The diagram shows examples of various information sources, various perception modules, MDI generators, selection unit narrow AI agents, and driving decision units.

[0201] Various information sources include:

[0202] • Information about the vehicle's environment from one or more vehicle sensors 581 ( 1 ).

[0203] • Information from one or more vehicle status sensors 581 (2).

[0204] Environmental Information 581(3).

[0205] Legal Constraints Database 581(4).

[0206] The perception module includes:

[0207] Road Setting Perception Module—denoted as RS Perception Module 582 (1).

[0208] • Road User Awareness Module—denoted as RU Awareness Module 582 (2).

[0209] Traffic Regulation Perception Module—denoted as TR Perception Module 582 (3).

[0210] • Regulatory Awareness Module—denoted as REG Awareness Module 582 (4).

[0211] • Environmental Condition Sensing Module—denoted as AMB Sensing Module 582 (5).

[0212] Vehicle state perception module—denoted as VS perception module 582 (6).

[0213] Figure 5B Also illustrated are unselected perception modules 582(6) which may be idle.

[0214] Selection unit 587 identifies one or more narrow AI agents by comparing MDI 585 to MDI cluster signatures 586 ( 1 )- 586 (J) to find one or more matching MDI clusters associated with one or more narrow AI agents to be selected.

[0215] The matching MDI cluster indicates the narrow AI agent (denoted as 588) that should be selected. Figure 5B Also illustrated are narrow AI agents that were not selected (represented as an inner pattern including dashed lines).

[0216] The selected narrow AI agent 588 outputs a narrow AI agent driving decision 589 .

[0217] The driving decision unit 590 is configured to receive the narrow AI agent driving decision 589 and generate and output an output driving decision 591 that can be executed by one or more units of the vehicle.

[0218] Figure 5C An example of an image 610 acquired by a vehicle sensor is shown.

[0219] Image 610 shows two lanes 611 and 612 of a road, a right building 613, a first vehicle 615, a second vehicle 616, a tree 618, a table 617, a left building 621, a child 623, a ball 624, a traffic sign 630 indicating a maximum speed limit (e.g., -50 km / H), a travel direction arrow 634, a zebra crossing 633, and a crowd 628 that is reaching the zebra crossing and crossing the zebra crossing from right to left. FIG6 also illustrates a progress (behavior) 641 of child 623, a progress 642 of crowd 628, a progress 645 of first vehicle 615, and a progress 646 of second vehicle 616.

[0220] The road user information may indicate movable road users, such as a first vehicle 615 , a second vehicle 616 , a child 623 , and a crowd 628 .

[0221] The road setting information may indicate static objects selected from lanes 611 and 612, road 610, tree 618, table 617, right building 613 and left building 621 - not all of these static objects may be taken into account - because not all of these static objects have an impact on the vehicle.

[0222] The traffic regulation indication information indicates a traffic sign 630 , a driving direction arrow 634 , and a lane boundary.

[0223] The environmental condition information may indicate that the vehicle's environment is illuminated by solar radiation and is not obscured by fog, rain or other weather elements.

[0224] Vehicle status information may be learned from information not included in the image.

[0225] Regulatory information may be learned from information not included in the image.

[0226] Figure 5D The figure shows an example of a vehicle 700. The vehicle 700 includes a vehicle transmission unit 710, which may include one or more sensors, such as vehicle sensors 712 and 714. The vehicle also includes one or more processing circuits, represented as 720, a storage unit 730, a communication unit 740, and one or more vehicle units, such as one or more vehicle computers, units controlled by one or more vehicle units, motor units, chassis, wheels, etc.

[0227] One or more processing circuits are configured to perform method 600 .

[0228] Figure 5E An example of a method 800 is shown.

[0229] According to an embodiment, method 800 includes steps 510 , 520 , 530 , 540 , and 550 .

[0230] According to an embodiment, method 800 includes step 860 of triggering further processing of at least a portion of the multi-domain information by the identified one or more narrow AI agents to provide one or more narrow AI driving-related decisions.

[0231] According to an embodiment, method 800 includes step 870 of further processing at least a portion of the multi-domain information by the identified one or more narrow AI agents to provide one or more narrow AI driving-related decisions.

[0232] According to an embodiment, the method 800 includes a step 360 of processing, by a driving decision unit, one or more narrow AI agent driving decisions to provide an output driving decision.

[0233] It should be noted that method 800 may include only one or two of steps 860 , 870 , and 360 .

[0234] Fig. 5F An example of a method 900 is shown.

[0235] According to an embodiment, method 900 includes a step 910 of receiving information by one or more identified narrow AI agents. The information may be multi-dimensional information fed to a plurality of perception modules, one or more segments of multi-dimensional information fed to a plurality of perception modules, or any representation or metadata about at least a portion of a plurality of perception modules.

[0236] According to an embodiment, one or more identified narrow AI agents are identified by executing method 500 .

[0237] According to an embodiment, step 910 is followed by step 920, which is to process the information through the identified one or more narrow AI agents to provide one or more narrow AI driving-related decisions to be sent to the driving decision unit. The one or more narrow AI driving-related decisions may trigger the generation of one or more output driving decisions by the driving decision unit. The one or more output driving decisions may be one or more commands or requests or suggestions sent to various modules of the vehicle and / or the driver.

[0238] Figure 5G An example of a method 1000 is shown.

[0239] According to an embodiment, method 1000 includes step 1010 , in which a driving decision unit receives one or more narrow AI driving-related decisions from one or more identified narrow AI agents.

[0240] According to an embodiment, step 1010 is followed by step 1020 of determining one or more output driving decisions by the driving decision unit. The one or more output driving decisions may be one or more commands or requests or suggestions sent to various modules of the vehicle and / or the driver.

[0241] There may be a non-transitory computer-readable medium for operation of a narrow artificial intelligence (AI) agent for at least partially autonomous driving, the non-transitory computer-readable medium storing instructions for: receiving multi-domain information about an element affecting a vehicle through a plurality of perception modules; wherein each of the plurality of perception modules is associated with a dedicated domain of the multi-domain information; generating a class signature indicating a class of an element of the multi-domain information by the plurality of perception modules; determining a multi-domain identifier that identifies the generated class signatures of the plurality of perception modules; and identifying one or more narrow AI agents associated with processing at least a portion of the multi-domain information based on the multi-domain identifier. The identification triggers execution of further processing of at least a portion of the multi-domain information by the identified one or more narrow AI agents to provide one or more narrow AI driving-related decisions.

[0242] According to an embodiment, the plurality of awareness modules are a plurality of awareness routers.

[0243] According to an embodiment, the plurality of awareness modules are a plurality of awareness sub-routers associated with an awareness router.

[0244] The non-transitory computer-readable medium stores instructions for automatically selecting a plurality of perception modules from a set of perception modules.

[0245] According to an embodiment, the selection is performed based on a domain associated with the multi-domain information.

[0246] According to an embodiment, the selection is performed based on an output of a previously generated perception module in the set of perception modules.

[0247] According to an embodiment, at least two different perception modules are associated with two different vehicle sensors, wherein the selection is performed based on an operational state of the two different vehicle sensors.

[0248] The non-transitory computer readable medium stores instructions for maintaining a sensing module in the group other than the plurality of sensing modules in an idle mode.

[0249] According to an embodiment, the multi-domain information is input from one of: road setting information associated with static objects within the vehicle's environment; road user information associated with movable road users within the environment;

[0250] Traffic rule indication information associated with visual traffic rule indicators within the environment; regulatory information associated with legal constraints regarding the environment;

[0251] Environmental condition information related to at least one of weather and light conditions of the environment; and vehicle state information related to a motion-related state of the vehicle.

[0252] predict

[0253] It has been found that introducing prediction in the context of using per-domain perception modules provides various technical benefits, such as:

[0254] Predictions improve management of memory cells (such as, but not limited to, cache memory cells) - for example, by improving the accuracy of pre-fetch operations, thereby reducing latency associated with memory misses, and reducing memory consumption - because fewer fetch operations are required.

[0255] Prediction reduces the urgency of processing and responding to sensed information, at least in most cases - and can reduce the speed at which processing resources allocated to execute perception modules and / or AI agents operate. This reduction in speed reduces energy consumption and extends the life of processing resources.

[0256] Prediction can allow for smoother driving maneuvers - this is beneficial for all units of the vehicle (e.g. from a mechanical point of view). In addition - some smoother driving maneuvers can be represented with fewer coefficients - thus reducing computational resources and also memory resources.

[0257] • Prediction may allow for better determination of the vehicle's future progress - for longer periods of time in advance - thereby reducing the computational and / or storage resources required for more repetitive, short-term vehicle progress determinations.

[0258] Fig. 6A is an example of a method 1100 for managing a group of narrow artificial intelligence (A1) agents for at least partially autonomous driving.

[0259] According to an embodiment, method 1100 includes step 1110 of acquiring, by a prediction circuit, a stream of metadata segments generated at a plurality of time points and associated with selections of one or more subsets of a set of narrow AI agents.

[0260] According to an embodiment, the metadata segment is a selected narrow AI agent identifier.

[0261] According to an embodiment, the metadata segment is a plurality of multi-domain identifiers (MDIs). The plurality of MDIs represent a plurality of instances of multi-domain information about vehicle elements associated with a plurality of time points (PITs). The MDI generated at a given time point in the plurality of time points is a combination of class signatures indicating classes of elements of the multi-domain information associated with the given time point. The method 500 may be used to generate the metadata segment.

[0262] According to an embodiment, step 1110 is followed by step 1120 of finding, by a prediction circuit, a segment of the stream that is a predicted value for receiving a next cluster identifier at a future time point.

[0263] Step 1120 may include any prediction process - including but not limited to a machine learning prediction process, a non-machine learning process, applying a Kalman filter, applying any prediction filter, etc.

[0264] According to an embodiment, step 1120 is followed by step 1130, which is to automatically predict at least one of the following when a predicted value is found:

[0265] The future metadata fragment to be received at this future point in time.

[0266] A future subgroup of narrow AI agents to be selected at that future point in time.

[0267] According to an embodiment, there is a known mapping between metadata segments and one or more selected narrow AI agents - in that a given metadata segment results in the selection of one or more associated narrow AI agents.

[0268] According to an embodiment, the prediction triggers a response to the prediction.

[0269] According to an embodiment, the method 1100 may include one or more additional steps (collectively referred to as 1140) of the following steps:

[0270] Trigger responses to predictions.

[0271] Respond to forecasts.

[0272] Triggering a response that includes predicting the next state of the vehicle at the next point in time.

[0273] • Triggering a response that includes determining a future progress of the vehicle based on one or more next states of the vehicle at one or more next points in time.

[0274] Determining a future schedule of the vehicle based on one or more next states of the vehicle at one or more next points in time. This may include smoothing the future schedule of the vehicle. Comparing the smoothed schedule to the future schedule of the vehicle without prediction.

[0275] Triggering the determination of a response to the vehicle's next state.

[0276] Determine the response to the vehicle's next state.

[0277] Evaluate the accuracy of the predicted values.

[0278] Based on the predicted value, predicting a selected perception module from among the plurality of perception modules for use at a future time point.

[0279] Triggering execution of a driving related operation at a time point not exceeding a future time point. The driving related operation may include changing the speed of the vehicle. In the method, the driving related operation includes changing the direction of the vehicle.

[0280] Figure 6B is an example of a stream of metadata segments generated at multiple points in time and associated with a selection of one or more subgroups of a set of narrow AI agents. Figure 6B , assume that the metadata segments are multiple MDIs represented as 1180(1)-1180(10), which are associated with different time points (PITs) 1182(1)-1182(10).

[0281] exist Figure 6B In, assume that:

[0282] • A first predicted value, which includes the value of the substream containing MDIs 1182(1)-1182(5) - indicates that in the future the PIT (eg, PIT 1182(11)) will output an MDI having a value equal to MDI 1182(6).

[0283] • A second predicted value, which includes the value of the substream including MDI 1182(3), MDI 1182(6), and MDI 1182(9) - indicating that another future PIT (eg, PIT 1182(12)) will output an MDI having a value equal to MDI 1182(7).

[0284] The first prediction value and the second prediction value may be used to perform at least the following operations:

[0285] - Maintain the values ​​of MDI 1182(6) and MDI 1182(7) in cache memory - at least until PIT 1182(11).

[0286] • Determine the state of the vehicle at PITs 1182 ( 10 ) and 1182 ( 11 ) - and determine the desired state of the vehicle at PITs prior to PIT 1182 ( 21 ) and PIT 1182 ( 22 ).

[0287] In this case, the vehicle's motion may be smoothed to prevent large changes in direction and / or speed from occurring at the PIT 1182(9) or the like.

[0288] According to an embodiment, the selection of narrow AI agents can be modified to take predictions into account. For example, the multiple prediction modules mentioned in the figure.

[0289] Figure 6C The figure shows examples of:

[0290] • Perception module 582.

[0291] MDI generator 584.

[0292] ·Selection unit 587.

[0293] Narrow AI Agent 588.

[0294] Driving decision unit 590.

[0295] • Prediction unit 1150.

[0296] Storage management unit 1152.

[0297] Processing management unit 1153.

[0298] A first storage unit 1154.

[0299] A second storage unit 1155.

[0300] Processing units 1154(1)-1154(K).

[0301] • Perception module metadata 1158.

[0302] Prediction-based driving decision unit 590 - 1 .

[0303] Non-prediction based driving decision unit 590 - 2 .

[0304] The second storage unit 1155 is illustrated as being smaller than the first storage unit 1154 - and for simplicity of explanation, it is assumed that the second storage unit 1155 is a cache memory. There may be more than two storage units.

[0305] The prediction may affect the output driving decision output by the driving decision unit, and / or may affect the storage management unit 1152 responsible for managing one or more storage units used for at least partially autonomous driving, and / or may affect the processing management unit 1153 configured to manage processing resources, such as processing resources for implementing a perception module and / or a selection unit and / or a narrow AI agent, etc.

[0306] Depending on the embodiment, processing units 1154(1)-1154(K) are processing circuits or parts of processing circuits.

[0307] According to an embodiment, the processing units 1154(1)-1154(K) are configured to (e.g., programmed or hardware constructed to) implement (using data and / or metadata stored internally by the processing unit and / or stored in one or more storage units) the perception module 582, the MDI generator 584, the selection unit 587, the narrow AI agent 588, the driving decision unit 590, the prediction unit 1150, the storage management unit 1152 and the processing management unit 1153.

[0308] According to an embodiment, at least two of the processing units 1154(1)-1154(K) may differ from each other in at least one parameter of the amount of computing resources, power consumption, complexity, etc. The selection of which processing unit to activate seeks to balance between the different parameters of the at least two different processing units.

[0309] Execution of the perception module may require providing perception module metadata (such as coefficients of the neural network model) and information (such as information of the domain associated with the perception module) to the processing unit and / or an associated storage unit (e.g., an internal storage unit of the processing unit or a second storage unit).

[0310] According to an embodiment, prediction 1151 allows the extraction of information and / or perception metadata to be controlled in an efficient manner - for example by at least one of: (a) predetermining prefetch operations that will result in cache hits, (b) determining an allocation scheme for assigning perception modules to processing units that can reduce the amount of configuration of the processing circuitry - because it is beneficial to assign the same processing unit to implement the same perception module during consecutive operating cycles.

[0311] According to an embodiment, prediction can help reduce power consumption of processing units - by placing unused processing units in an idle state, and by reducing the clock frequency of the processing module, any unit required by the processing module can be implemented to output results in a few cycles - rather than every cycle.

[0312] According to an embodiment, the driving decision unit may include a prediction-based driving decision unit 590-1 and a non-prediction-based driving decision unit 590-2 - the former is associated with a driving decision when there is no prediction - and the latter is associated with a driving decision when there is a prediction - however, these units can be selectively activated or deactivated when there is or does not exist a prediction. Relative to the non-prediction-based driving decision unit 590-2, the prediction-based driving decision unit 590-1 can operate at a lower clock frequency - or can be configured to operate at a reduced power.

[0313] Localized driving

[0314] The environment of the vehicle at different locations may be location-biased, meaning that the statistical distribution of elements that may affect the vehicle at one location is different than the statistical distribution of elements that may affect the vehicle at another location.

[0315] Predicting the environment faced by a vehicle at different paths along the vehicle's intended path can be very effective and provide various technical benefits, such as:

[0316] Predictions improve management of memory cells (such as, but not limited to, cache memory cells) - for example, by improving the accuracy of pre-fetch operations, thereby reducing delays associated with memory misses and reducing memory consumption - because fewer fetch operations are required.

[0317] Prediction reduces the urgency of processing and responding to sensed information, at least in most cases - and can reduce the operational speed of processing resources allocated to execute perception modules and / or AI agents. The reduction in operational speed reduces energy consumption and extends the life of processing resources.

[0318] Prediction can allow for smoother driving maneuvers - this is beneficial for all units of the vehicle (e.g. from a mechanical point of view). In addition - some smoother driving maneuvers can be represented with fewer coefficients - thus reducing computational resources and also memory resources.

[0319] • Prediction may allow for better determination of the vehicle's future progress - for longer periods of time in advance - thereby reducing the computational and / or storage resources required for more repetitive, short-term vehicle progress determinations.

[0320] Fig. 7A The diagram shows an example of an urban environment including a plurality of buildings forming a street grid 1202. The vehicle's expected local path (the vehicle's expected path within a smaller environment - e.g., within a distance of 10-1000 meters from the vehicle) 1205 includes moving west, going around roundabout 1201-1, continuing west, turning north, proceeding north, turning west, and moving west.

[0321] Fig. 7A Various locations (1202-1 to 1202-5 and 1202-11 to 1202-15) associated with MDI statistics are also illustrated. The MDI for each location represents the elements that affect the vehicle at that location. The MDI can be generated in any manner - including the manner shown above - such as by method 500.

[0322] exist Fig. 7AIn the example of , positions 1201-1 to 1202-5 are path-related positions, which are identified by the processing circuit based on the expected local path 1205 and the information about the positions. Positions 1202-11 to 1202-15 are considered irrelevant - for example, they are tools far from the expected local path or belong to a road segment not included in the expected local path.

[0323] Figure 7B It is an example of MDI statistical information of the first position 1210-1, MDI statistical information of the second position 1210-2, MDI statistical information of the third position 1210-3, MDI statistical information of the fourth position 1210-4, and MDI statistical information of the fifth position 1210-5.

[0324] exist Figure 7B In the example of , these MDI statistics include the most common MDI for each position - the most common MDI for the first position 1210-1, the most common MDI for the second position 1210-2, the most common MDI for the third position 1210-3, the most common MDI for the fourth position 1210-4, and the most common MDI for the fifth position 1210-5.

[0325] The most common MDI at a position may be the MDI that occurs at least a certain number of times at that position, or the MDI that occurs with at least a predetermined probability (e.g., greater than 10, 15, 20, 25, 30, 40, etc. percent), or the Xth most common MDI - X may be determined in any manner - for example, it may be the maximum 5, 10, 15, 20, 25, 30 MDIs, etc.

[0326] According to the embodiment, there may be one or more instances with the most common MDI for each or more constraints. For example, a constraint may refer to different hours, nights or days, different dates, different seasons, different weather conditions, etc. For example, see two instances 1211-1-1 and 1211-2.

[0327] Figure 7C An example of a method 1300 for localized driving is shown.

[0328] According to an embodiment, the method 1300 includes a step 1310 of obtaining information about locations associated with multi-domain identifier (MDI) statistics, the MDI for each location representing elements affecting a vehicle at the location.

[0329] According to an embodiment, method 1300 includes a step 1320 of obtaining an expected local path of the vehicle.

[0330] According to an embodiment, step 1310 and step 1320 are followed by step 1330 of identifying, by a processing circuit, a path-related location based on the expected partial path and the information about the location.

[0331] According to an embodiment, step 1330 is followed by step 1340 of determining, by processing circuitry, expected local path MDI statistics for use in at least partially autonomous driving of the vehicle while traversing the expected local path.

[0332] According to an embodiment, step 1340 includes identifying the most prevalent MDI for each of the path-related locations.

[0333] According to an embodiment, determination of the expected local path MDI statistics triggers a determination of at least partially autonomous driving of the vehicle as it traverses the expected local path.

[0334] According to an embodiment, step 1340 is followed by step 1350 of responding to the expected local path MDI statistics.

[0335] Step 1350 may include at least one of the following:

[0336] • A determination that triggers at least partially autonomous driving of the vehicle through the intended local path.

[0337] Triggering execution of at least partially autonomous driving of the vehicle through the intended local path.

[0338] Based on the expected local path MDI statistics, predicting a selected perception module from the plurality of perception modules for use at a future time point associated with the expected local path and associated with the path-related point.

[0339] • Pre-fetch selected perception modules into cache memory.

[0340] Determining at least partially autonomous driving of the vehicle through the intended local path.

[0341] Performing at least partially autonomous driving of the vehicle through the intended local path.

[0342] According to an embodiment, determining at least partially autonomous driving of the vehicle through the expected local path may achieve at least one of the following:

[0343] Modifications to the expected local path.

[0344] · Execute without modifying the intended local path.

[0345] Based on the most common MDI along the path, at least the driving parameters (such as speed and / or acceleration and / or heading) are modified. For example, one MDI may indicate an obstacle at a certain location, and the driving parameters may be set to assist in going around the obstacle, passing the obstacle at a reduced speed, etc. If the vehicle is expected to approach an obstacle at a certain point in time (PIT), the driving parameters may be modified in advance, even up to a certain PIT.

[0346] Select a perception module that is able to manage the situation indicated by the expected local path MDI statistics.

[0347] Select narrow AI modules that are able to manage the situations indicated by the expected local path MDI statistics.

[0348] Fig.7D The figure shows examples of:

[0349] • Perception module 582.

[0350] MDI generator 584.

[0351] ·Selection unit 587.

[0352] Narrow AI Agent 588.

[0353] Driving decision unit 590.

[0354] • Prediction unit 1150.

[0355] Storage management unit 1152.

[0356] Processing management unit 1153.

[0357] A first storage unit 1154.

[0358] A second storage unit 1155.

[0359] Processing units 1154(1)-1154(K).

[0360] • Perception module metadata 1158.

[0361] Position and MDI statistics 1380.

[0362] · Expected local path provider 1381.

[0363] · Path-related location identifier 1382.

[0364] The second storage unit 1155 is illustrated as being smaller than the first storage unit 1154 - and for simplicity of explanation, it is assumed that the second storage unit 1155 is a cache memory. There may be more than two storage units.

[0365] This prediction (using MDI statistics for path-related positions) may affect an output driving decision output by the driving decision unit, and / or may affect a storage management unit 1152 responsible for managing one or more storage units for at least partially autonomous driving, and / or may affect a processing management unit 1153 configured to manage processing resources, such as processing resources for implementing a perception module and / or a selection unit and / or a narrow AI agent, etc.

[0366] Depending on the embodiment, processing units 1154(1)-1154(K) are processing circuits or parts of processing circuits.

[0367] According to an embodiment, the processing units 1154(1)-1154(K) are configured to (e.g., programmed or hardware constructed to) implement (using data and / or metadata stored internally by the processing unit and / or stored in one or more storage units) the perception module 582, the MDI generator 584, the selection unit 587, the narrow AI agent 588, the driving decision unit 590, the prediction unit 1150, the storage management unit 1152, the processing management unit 1153, the location and MDI statistics 1380, the expected local path provider 1381, and the path-related location identifier 1382.

[0368] According to an embodiment, at least two of the processing units 1154(1)-1154(K) may differ from each other in at least one parameter of the amount of computing resources, power consumption, complexity, etc. The selection of which processing unit to activate seeks to balance between the different parameters of the at least two different processing units.

[0369] Execution of the perception module may require providing perception module metadata (such as coefficients of the neural network model) and information (such as information of the domain associated with the perception module) to the processing unit and / or an associated storage unit (e.g., an internal storage unit of the processing unit or a second storage unit).

[0370] According to an embodiment, prediction 1151 allows the extraction of information and / or perception metadata to be controlled in an efficient manner - for example by at least one of: (a) predetermining prefetch operations that will result in cache hits, (b) determining an allocation scheme for assigning perception modules to processing units that can reduce the amount of configuration of the processing circuitry - because it is beneficial to assign the same processing unit to implement the same perception module during consecutive operating cycles.

[0371] According to an embodiment, prediction can help reduce power consumption of processing units - by placing unused processing units in an idle state, and by reducing the clock frequency of the processing module, any unit required by the processing module can be implemented to output results in a few cycles - rather than every cycle.

[0372] Position and MDI statistics 1380 stores information about (a) positions associated with MDI statistics, and (b) MDI statistics.

[0373] The expected local path provider 1381 is configured to provide an expected local path. The expected local path provider 1381 may determine the expected local path in any manner, or may receive the expected local path. The expected local path provider 1381 may be a navigation unit, an ADAS unit, an AV unit, etc.

[0374] Path-relative location identifier 1382 may be configured to perform step 1330 .

[0375] In the foregoing specification, the invention has been described with reference to specific examples of embodiments of the invention. It will, however, be evident that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims.

[0376] Furthermore, the terms "front", "rear", "top", "bottom", "upper", "lower", etc., in the specification and claims, if present, are used for descriptive purposes and not necessarily for describing permanent relative positions. It is to be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are, for example, capable of operation in other orientations than those illustrated or otherwise described herein.

[0377] Furthermore, the terms "assert" or "set" and "negate" (or "deassert" or "clear") as used herein refer to assuming a signal, status bit, or the like to its logically true or logically false state, respectively. If the logically true state is a logical level one, then the logically false state is a logical level zero. Also, if the logically true state is a logical level zero, then the logically false state is a logical level one.

[0378] Those skilled in the art will recognize that the boundaries between logic blocks are merely illustrative, and that alternative embodiments may merge logic blocks or circuit elements, or impose alternative functional decompositions on various logic blocks or circuit elements. Therefore, it should be understood that the architectures described herein are merely exemplary, and that in fact many other architectures may be implemented that achieve the same functionality.

[0379] Any arrangement of components that perform the same function is effectively "associated" to achieve the desired function. Thus, any two components combined to achieve a particular function herein may be considered "associated" with each other to achieve the desired function, regardless of architecture or intermediate components. Likewise, any two components so associated may also be considered "operably connected" or "operably coupled" to each other to achieve the desired function.

[0380] In addition, those skilled in the art will recognize that the boundaries between the above operations are merely illustrative. Multiple operations may be combined into a single operation, a single operation may be distributed in additional operations, and operations may be performed at least partially overlapping in time. In addition, alternative embodiments may include multiple instances of a particular operation, and the order of the operations may change in various other embodiments.

[0381] For another example, in one embodiment, the examples shown in the figure may be implemented as a circuit located on a single integrated circuit or in the same device. Alternatively, these examples may be implemented as any number of independent integrated circuits or independent devices interconnected with each other in a suitable manner.

[0382] However, other modifications, changes, and alternatives are possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense.

[0383] In the claims, any reference signs in brackets shall not be construed as limiting the claims. The word "comprising" does not exclude the presence of other elements or steps than those listed in the claim. Unless otherwise indicated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements described by these terms. Therefore, these terms are not necessarily intended to indicate a temporal or other priority order of these elements. The fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage.

[0384] While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes, and equivalents will now occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes that fall within the true spirit of the invention.

[0385] It should be understood that, for the sake of clarity, various features of embodiments of the present disclosure described in the context of separate embodiments may also be provided in combination in a single embodiment. Conversely, for the sake of brevity, various features of embodiments of the present disclosure described in the context of a single embodiment may also be provided separately or in any suitable sub-combination.

[0386] Those skilled in the art will appreciate that the embodiments of the present disclosure are not limited by what has been specifically shown and described above. Rather, the scope of the embodiments of the present disclosure is defined by the appended claims and their equivalents.

Claims

1. A method for managing a group of narrow artificial intelligence (AI) agents, the narrow artificial intelligence agents being used for at least partially autonomous driving, the method include: Obtaining, by prediction circuitry, a stream of metadata segments, the stream of metadata segments generated at a plurality of time points and associated with a selection of one or more subsets of the set of narrow AI agents; wherein the metadata segments are selected from: (a) a selected narrow AI agent identifier, and (b) a plurality of multi-domain identifiers, the plurality of multi-domain identifiers representing a plurality of instances of multi-domain information about vehicle-affecting elements associated with the plurality of time points; wherein the multi-domain identifier generated at a given time point in the plurality of time points is a combination of class signatures, the combination of class signatures representing a class of elements of the multi-domain information associated with the given time point; Finding, by the prediction circuit, a segment of the stream, the segment of the stream being a predicted value for receiving a next cluster identifier at a future time point; and When the predicted value is found, at least one of the following is automatically predicted: (a) future metadata segments to be received during the future time point; (b) A future subset of narrow AI agents that will be selected at said future time point.

2. The method according to claim 1, in, The future subgroups of narrow AI agents are associated with future subgroups of skills.

3. The method according to claim 1, in, The automatic prediction triggers a prediction of a next state of the vehicle at a next point in time. The method of claim 3 , comprising triggering determination of a response to the next state of the vehicle. The method of claim 3 , comprising determining a response to the next state of the vehicle. The method of claim 1 , comprising evaluating the accuracy of the predicted value.

7. The method of claim 1, comprising searching at least the stream of metadata segments for new prediction values.

8. The method according to claim 1, comprising predicting a selected perception module based on the predicted value, the selected perception module being selected from a plurality of perception modules and used at the future time point.

9. The method according to claim 1, in, The automatic prediction of the future metadata segment triggers performance of a driving related maneuver at a point in time not exceeding the future point in time.

10. The method according to claim 9, in, The driving-related operation includes changing the speed of the vehicle.

11. The method according to claim 9, in, The driving-related operation includes changing the direction of the vehicle.

12. A non-transitory computer-readable medium for managing a group of narrow artificial intelligence (AI) agents for at least partially autonomous driving, the non-transitory computer-readable medium storing instructions for: acquiring, by prediction circuitry, a stream of metadata segments, the stream of metadata segments generated at a plurality of points in time and associated with selections of one or more subsets of the set of narrow AI agents; in, The metadata segment is selected from: (a) a selected narrow AI agent identifier, and (b) a plurality of multi-domain identifiers, the plurality of multi-domain identifiers representing a plurality of instances of multi-domain information about vehicle elements related to the plurality of time points; wherein the multi-domain identifier generated at a given time point in the plurality of time points is a combination of class signatures, the combination of class signatures representing a class of elements of the multi-domain information associated with the given time point; Finding, by the prediction circuit, a segment of the stream, the segment of the stream being a predicted value for receiving a next cluster identifier at a future time point; and When the predicted value is found, at least one of the following is automatically predicted: (a) future metadata segments to be received during the future time point; (b) A future subset of narrow AI agents that will be selected at said future time point.

13. The non-transitory computer readable medium of claim 12, in, The future subgroups of narrow AI agents are associated with future subgroups of skills.

14. The non-transitory computer readable medium of claim 12, in, The automatic prediction triggers a prediction of a next state of the vehicle at a next point in time.

15. The non-transitory computer readable medium of claim 14, comprising triggering a determination of a response to the next state of the vehicle.

16. The non-transitory computer readable medium of claim 14, comprising determining a response to the next state of the vehicle.

17. The non-transitory computer readable medium of claim 12, comprising evaluating the accuracy of the predicted value.

18. The non-transitory computer-readable medium of claim 12, comprising searching at least the stream of metadata segments for new prediction values.

19. The non-transitory computer readable medium of claim 12, comprising predicting a selected perception module based on the predicted value, the selected perception module being selected from a plurality of perception modules and used at the future time point.

20. The non-transitory computer readable medium of claim 12, in, The automatic prediction of the future metadata segment triggers performance of a driving related maneuver at a point in time not exceeding the future point in time.