Driving related augmented virtual field

Abstract

To provide a method for enhancing driving related augmenting virtual fields.SOLUTION: A method includes: obtaining object information regarding one or more objects located within an environment of a vehicle; and determining, by a processing circuit, and based on the object information, a desired virtual acceleration of the vehicle, in which the determining of the desired virtual acceleration of the vehicle is based on a virtual physical model that represents an impact of the one or more objects on a behavior of the vehicle, in which the virtual physical model is built based on one or more physical laws and at least one additional driving related parameter.SELECTED DRAWING: Figure 17

Description

[Technical field]

[0001] This application is a continuation-in-part of U.S. Patent Application No. 17 / 823,069, filed August 29, 2022, which claims priority from U.S. Provisional Patent Application No. 63 / 260,839, which is incorporated herein by reference. This application claims priority from U.S. Provisional Patent Application No. 63 / 368,874, filed July 19, 2022, which is incorporated herein in its entirety. This application claims priority from U.S. Provisional Patent Application No. 63 / 373,454, filed August 24, 2022, which is incorporated herein in its entirety. [Background technology]

[0002] Autonomous vehicles (AVs) can help significantly reduce the number of road accidents and CO2 emissions, as well as contribute to a more efficient transportation system. However, today's candidate AV technologies are not scalable in three ways:

[0003] Limited field of view, lighting and weather challenges, and occlusions all lead to detection errors and noisy localization / kinematics. To address such poor real-world perception output, one approach for AV technologies is to invest in expensive sensors and / or integrate specialized infrastructure into the road network. However, such attempts are very costly and, in the case of infrastructure, geographically limited, and therefore may not lead to generally accessible AV technologies.

[0004] AV technologies that are not based on expensive hardware and infrastructure rely entirely on machine learning and, therefore, data to handle real-world situations. To address detection errors and learn driving policies sufficient for complex driving tasks, huge amounts of data and computational resources are required, and there are still edge cases that are not handled properly. A common feature in these edge cases is that machine learning models do not generalize well to unseen or confusing situations, and the erroneous behavior is difficult to analyze due to the black-box nature of deep neural networks.

[0005] Current road-enabled autonomous driving is implemented in the form of separate ADAS features such as ACC, AEB, and LCA. Reaching full autonomous driving requires seamlessly combining existing ADAS features together, as well as covering any gaps that are not currently automated by adding more such features (e.g., lane changing, intersection handling, etc.). In short, current autonomous driving is not based on a holistic approach that can be easily extended to bring about full autonomous driving.

[0006] The embodiments of the present disclosure will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings, in which: [Prior art documents] [Patent documents]

[0007] [Patent Document 1] U.S. Patent Application Serial No. 16 / 729,589 [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a method. [Diagram 2] FIG. 1 illustrates an example of a method. [Diagram 3] FIG. 1 illustrates an example of a method. [Figure 4]FIG. 1 illustrates an example of a method. [Diagram 5] FIG. 1 is a diagram showing an example of a vehicle. [Figure 6] FIG. 1 illustrates an example of a situation and a perceptual field. [Figure 7] FIG. 1 illustrates an example of a situation and a perceptual field. [Figure 8] FIG. 1 illustrates an example of a situation and a perceptual field. [Figure 9] FIG. 1 illustrates an example of a situation and a perceptual field. [Figure 10] FIG. 1 illustrates an example of a method. [Figure 11] FIG. 2 shows an example of a scene. [Figure 12] FIG. 1 illustrates an example of a method. [Figure 13] FIG. 13 is a diagram showing an example of an image. [Figure 14] FIG. 13 is a diagram showing an example of an image. [Figure 15] FIG. 13 is a diagram showing an example of an image. [Figure 16] FIG. 13 is a diagram showing an example of an image. [Figure 17] FIG. 1 illustrates an example of a method. [Figure 18] FIG. 1 illustrates an example of a method. [Figure 19] FIG. 1 illustrates an example of a method. [Figure 20] FIG. 1 illustrates an example of a method. [Figure 21] FIG. 1 illustrates an example of a method. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0010] The subject matter which is regarded as the invention is particularly pointed out and distinctly claimed in the concluding portion of this specification. However, the invention, both as to the organization and method of operation, together with its objects, features, and advantages, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

[0011] It will be appreciated that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, dimensions of some of the elements may be exaggerated relative to other elements for clarity. Further, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

[0012] Because the illustrative embodiments of the present invention can be implemented for the most part using electronic components and circuits known to those skilled in the art, details will not be described to any greater extent than is deemed necessary, as set forth above, for the understanding and appreciation of the basic concepts of the present invention and in order not to obscure or deviate from the teachings of the present invention.

[0013] Any reference in this specification to a method is intended to apply mutatis mutandis to a device or system capable of performing the method, and / or to a non-transitory computer readable medium storing instructions for performing the method.

[0014] Any reference in this specification to a system or device shall apply mutatis mutandis to methods that may be performed by the system and / or may apply mutatis mutandis to a non-transitory computer readable medium storing instructions executable by the system.

[0015] Any reference in this specification to a non-transitory computer readable medium shall apply mutatis mutandis to a device or system capable of executing instructions stored on the non-transitory computer readable medium and / or may apply mutatis mutandis to a method of executing instructions.

[0016] Any combination of any modules or units recited in any of the figures, in any part of the specification and / or in any claim may be provided.

[0017] Any one of the units and / or modules shown in this application may be implemented in hardware and / or code, instructions and / or commands stored on a non-transitory computer readable medium and may be included in the vehicle, outside the vehicle, in a mobile device, in a server, etc.

[0018] The vehicle may be any type of vehicle, such as, for example, a land transport vehicle, an air transport vehicle, or a watercraft.

[0019] The present specification and / or drawings may refer to images. An image is an example of a media unit. Any reference to an image may apply mutatis mutandis to a media unit. A media unit may be an example of a sensed information unit (SIU). Any reference to a media unit may apply mutatis mutandis to any type of natural signal such as, but not limited to, signals generated by nature, signals representing human actions, signals representing operations related to the stock market, medical signals, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, etc. Any reference to a media unit may apply mutatis mutandis to a sensed information unit (SIU). The SIU may be of any type and may be sensed by any type of sensor such as visible light cameras, audio sensors, infrared, radar imaging, ultrasonic, electro-optical, radiographic, LIDAR (light detection and ranging) sensors, thermal sensors, passive sensors, active sensors, etc. Sensing may include generating samples (e.g., pixels, audio signals) that represent the signal transmitted or otherwise arriving at the sensor. The SIU may be one or more images, one or more video clips, textual information about one or more images, text describing kinematic information about the object, etc.

[0020] The object information may include information related to any type of object, such as, but not limited to, the position of the object, the behavior of the object, the velocity of the object, the acceleration of the object, the direction of propagation of the object, the type of object, one or more dimensions of the object, etc. The object information may be raw SIUs, processed SIUs, text information, information derived from SIUs, etc.

[0021] Obtaining the object information may include receiving the object information, generating the object information, participating in processing the object information, processing only a portion of the object information, and / or receiving only another portion of the object information.

[0022] Obtaining the object information may include object detection or may be performed without object detection.

[0023] The processing of the object information may include at least one of object detection, noise reduction, improving the signal-to-noise ratio, defining a bounding box, and the like.

[0024] The object information may be received from one or more sources, such as one or more sensors, one or more communication units, one or more memory units, one or more image processors, etc.

[0025] The object information may be provided in one or more ways, for example in an absolute manner (e.g., providing the coordinates of the object's position) or in a relative manner, for example with respect to the vehicle (e.g., the object is located at a certain distance and a certain angle with respect to the vehicle).

[0026] The vehicle is also referred to as the ego-vehicle.

[0027] The specification and / or drawings may refer to a processor or processing circuit. A processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or as one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), fully custom integrated circuits, or the like, or a combination of such integrated circuits.

[0028] Any combination of any of the steps of the methods illustrated in this specification and / or in the drawings may be provided.

[0029] Any combination of the subject matter of any of the claims may be provided.

[0030] Any combination of the systems, units, components, processors, sensors illustrated in this specification and / or in the drawings may be provided.

[0031] Any reference to an object may be applicable to a pattern, and therefore any reference to object detection may be applicable mutatis mutandis to pattern detection.

[0032] Successful driving relies on going around surrounding road objects based on their position and motion, but humans are notoriously poor at evaluating kinematics. Humans appear to use an internal representation of surrounding objects in the form of a virtual force field that immediately suggests actions, thus obviating the need for kinematic evaluation. Consider a scenario in which an ego-vehicle is traveling in one lane and a vehicle diagonally ahead in the adjacent lane begins to veer into the ego-vehicle's lane. The human response to braking or derailing is immediate and instinctive, and can be experienced as a virtual force that pushes the ego-vehicle away from the veered vehicle. This virtual force representation is learned and associated with a particular road object.

[0033] Inspired by the above considerations, we propose a new concept of a perception field: a learned road object representation in the form of a virtual force field that is "sensed" through the ego-vehicle's control system in the form of ADAS and / or AV software. A field is defined here as a mathematical function that depends on spatial position (or similar quantities).

[0034] One example of an inference method 100 is shown in FIG.

[0035] The method 100 may be performed for one or more frames of the vehicle's environment.

[0036] Step 110 of method 100 may include detecting and / or tracking one or more objects (e.g., including one or more road users). The detecting and / or tracking may be performed in any manner. The one or more objects may be any object that may affect the behavior of the vehicle. For example, a road user (pedestrian, another vehicle), a road and / or path along which the vehicle is traveling (e.g., road or path conditions, road geometry, e.g., curves, straight road segments), traffic signs, traffic lights, road intersections, schools, kindergartens, etc. Step 110 may include obtaining further information such as kinematic and contextual variables associated with the one or more objects. Obtaining may include receiving or generating. Obtaining may include processing one or more frames to generate the kinematic and contextual variables.

[0037] It should be noted that step 110 may involve acquiring kinematic variables (without acquiring one or more frames).

[0038] Method 100 may also include a step 120 of obtaining respective perceptual fields associated with one or more objects. Step 120 may include determining which mapping between the objects should be retrieved and / or used, etc.

[0039] Step 110 (and even step 120) may be followed by step 130 of determining one or more virtual forces associated with one or more objects by passing relevant input variables, such as kinematic and contextual variables, to the perception field (and one or more virtual physics model functions).

[0040] Step 130 may be followed by step 140 of determining a total virtual force applied to the vehicle based on the one or more virtual forces associated with the one or more objects. For example, step 140 may include performing a vector weighted sum (or other function) on the one or more virtual forces associated with the one or more objects.

[0041] Step 140 may be followed by step 150 of determining a desired (or target) virtual acceleration based on the total virtual force, for example based on an equivalent of Newton's second law. The desired acceleration may be a vector, or may otherwise have a direction.

[0042] Step 150 may be followed by step 160 of converting the desired virtual acceleration into one or more vehicle maneuvers that the vehicle will propagate according to the desired virtual acceleration.

[0043] For example, step 160 may involve translating a desired acceleration into an acceleration or deceleration or changing the direction of travel of the vehicle using gas pedal movement, brake pedal movement, and / or steering wheel angle. The translation may be based on a dynamics model of the vehicle with a particular control scheme.

[0044] Advantages of perceptual fields include, for example, explainability, generalizability, and robustness to noisy inputs.

[0045] Explainability. Representing the ego-vehicle's motion as a composition of individual perceptual fields suggests a decomposition of actions into more elementary components, which is itself an important step towards explainability. The possibility to visualize these fields and apply intuition from physics to predict the ego-vehicle's motion represents an additional level of explainability compared to typical end-to-end black-box deep learning approaches. This increased transparency also allows passengers and drivers to have more trust in the AV or ADAS technology.

[0046] Generalizability. Representing the ego-vehicle's response to unknown road objects as a repelling virtual force field constitutes an induced bias in unseen situations. This representation has the potential advantage that the representation can handle edge cases in a safe manner with less training. Furthermore, the perception field model is holistic in the sense that the same approach can be used for all aspects of the driving policy. The perception field model can also be divided into narrow driving features used in ADAS such as ACC, AEB, LCA, etc. Finally, the composite nature of the perception field allows the model to be trained on atomic scenarios and still be able to adequately handle more complex scenarios.

[0047] Robustness to noisy inputs: Physical constraints on the time evolution of the perceptual field combined with implicit filtering of the input can lead to a better handling of noise in the input data compared to pure filtering of the localization and kinematic data.

[0048] The physical or virtual forces allow for mathematical formulation, for example in terms of second-order ordinary differential equations involving so-called dynamical systems. The advantage of expressing the control policy as such is that it is amenable to intuition from the theory of dynamical systems and it is straightforward to incorporate external modules such as input / output prediction, navigation and filtering.

[0049] Further advantages of the perceptual field approach are that it does not depend on any specific hardware and its computational cost is no higher than existing methods.

[0050] Training process

[0051] The process of learning the perceptual field can be one of two types, namely behavioral cloning (BC) and reinforcement learning (RL), or a combination of them. BC approximates a control policy by fitting a neural network to observed human state-action pairs, whereas RL imposes learning by trial and error in a simulated environment without reference to expert demonstrations.

[0052] These two classes of learning algorithms can be combined by first learning a policy with BC and then fine-tuning it using RL as an initial policy. Another way to combine the two approaches is to first learn a so-called reward function (used in RL) by behavior cloning to infer what constitutes a desirable human behavior, and then train by trial and error using regular RL. This latter approach has been given the name inverse RL (IRL).

[0053] FIG. 2 is an example of a training method 200 used for learning with a BC.

[0054] Method 200 may begin by step 210 of collecting human data, which may be a demonstration by an expert of how to handle a scenario.

[0055] Step 210 may be followed by step 220 of constructing a loss function that accounts for differences between the kinematic variables resulting from the perceptual field model and the corresponding kinematic variables of the human demonstration.

[0056] Step 220 may be followed by step 230 of updating parameters of the perceptual field and an auxiliary function (which may be a virtual physical model function different from the perceptual field) to minimize a loss function by some optimization algorithm such as gradient descent.

[0057] FIG. 3 is an example of a training method 250 used in reinforcement learning.

[0058] The method 250 may begin by the step 260 of creating a realistic simulation environment.

[0059] Step 260 may be followed by step 270 of constructing a reward function, either by learning the reward function from expert demonstrations or by manual design.

[0060] Step 270 may be followed by step 280 of running the episode in a simulated environment and continuously updating the parameters of the perceptual field and auxiliary function by some algorithm, such as proximal policy optimization, to maximize the expected cumulative reward.

[0061] FIG. 4 illustrates an example of a method 400 .

[0062] The method 400 may be for a perceptual field driving related operation.

[0063] The method 400 may begin with an initialization step 410 .

[0064] The initialization step 410 may include receiving a population of NNs to be trained to perform step 440 of the method 400 .

[0065] Alternatively, step 410 may include training a NN that performs step 440 of method 400 .

[0066] Various examples of training NNs are presented below. a.NNs can be trained to map object information to one or more virtual forces using behavioral cloning. b. NNs can be trained to map object information to one or more virtual forces using reinforcement learning. c. NNs can be trained to map object information to one or more virtual forces using a combination of reinforcement learning and behavioral cloning. d. NNs can be trained to map object information to one or more virtual forces using reinforcement learning with a reward function defined using behavioral cloning. e.NNs can be trained to map object information to one or more virtual forces using reinforcement learning with an initial policy defined using behavioral cloning. f. NNs can be trained to map object information to one or more virtual physical model functions that are distinct from one or more virtual forces and perception fields. g. The NN group may include a first NN and a second NN, where the first NN is trained to map object information to one or more perceptual fields and the second NN is trained to map object information to one or more virtual physical model functions.

[0067] The initialization step 410 may be followed by a step 420 of obtaining object information for one or more objects located in the vehicle's environment. Step 410 may be repeated multiple times, and the following steps may also be repeated multiple times. The object information may include video, images, audio, or any other sensed information.

[0068] Step 420 may be followed by step 440 of determining one or more virtual forces to be applied to the vehicle using one or more neural networks (NNs).

[0069] The one or more NNs may be the entire population of NNs (from initialization step 410) or may be only a portion of the population of NNs, leaving one or more unselected NNs in the population.

[0070] The one or more virtual forces represent one or more influences of one or more objects on the behavior of the vehicle. The influences can be future or current influences. The influences can cause the vehicle to change its course.

[0071] The one or more virtual forces belong to a virtual physics model, which is a virtual model that can substantially apply physical laws (e.g., mechanical laws, electromagnetic laws, optical laws) to the vehicle and / or object.

[0072] Step 440 includes the following steps: a. calculating a total virtual force applied to the vehicle based on one or more virtual forces applied to the vehicle; and b. determining a desired virtual acceleration of the vehicle based on a total virtual acceleration applied to the vehicle by the total virtual forces. The desired virtual acceleration may be equal to or different from the total virtual acceleration.

[0073] The method 400 may also include at least one of steps 431, 432, 433, 434, 435, and 436.

[0074] Step 431 may include determining a status of the vehicle based on the object information.

[0075] Step 431 may be followed by step 432 of selecting one or more NNs based on the situation.

[0076] Additionally or alternatively, step 431 may be followed by step 433 of providing the situational metadata to one or more NNs.

[0077] Step 434 may include detecting a class of each of the one or more objects based on the object information.

[0078] Step 434 may be followed by step 435 of selecting one or more NNs based on a class of at least one of the one or more objects.

[0079] Additionally or alternatively, step 434 may be followed by step 436 of providing class metadata to the one or more NNs indicating a class of at least one of the one or more objects.

[0080] Step 440 may be followed by step 450 of performing one or more driving-related maneuvers of the vehicle based on the one or more virtual forces.

[0081] Step 450 may be performed without human driver intervention and may include varying the speed and / or acceleration and / or direction of the vehicle, which may include performing automated driving or advanced driver assistance system (ADAS) maneuvers that may include temporarily assuming control of the vehicle and / or one or more driving related units of the vehicle, which may include setting the vehicle acceleration to a desired virtual acceleration with or without the involvement of a human driver.

[0082] Step 440 may include suggesting to the driver to set the vehicle acceleration to a desired virtual acceleration.

[0083] 5 is an example of a vehicle. The vehicle may include one or more sensing units 501, one or more driving related units 510 (such as autonomous driving units, ADAS units, etc.), a processor 560 configured to execute any of the methods, a memory unit 508 for storing instructions and / or results of the methods, functions, etc., and a communication unit 504.

[0084] 6 shows an example method 600 for lane centering RL using lane sample points as input. The lane sample points are located within the vehicle's environment.

[0085] RL assumes a simulation environment in which an agent (ego-vehicle) generates input data on which it can enforce its learned policy (perception field).

[0086] The method 600 calculates road sample points (X L,i ,Y L,i ) and (X R,i ,Y R,i The process may begin with step 610 of detecting the nearest lane or side of the vehicle (previously referred to as the vehicle). ego As shown in the figure.

[0087] After step 610, the left lane input vector (X L,i ,Y L,i ) and V ego X L and the right lane input vector (X R,i ,Y R,i ) and V ego X R This may be followed by step 620 of coupling to

[0088] After step 620, the lane perception field f θ (X L ) and f θ (X R This may be followed by step 630 of calculating .times. ...

[0089] After step 630, a differential equation describing the host vehicle acceleration applied to the host vehicle, i.e., a=f θ (X L )+f θ (X R ) may be followed by a step 640 of constructing

[0090] This may be the output of the inference process. Step 640 may be followed by step 450 (not shown).

[0091] The method may include updating one or more NNs, where RL may assume a reward function that is either learned or handcrafted based on expert demonstrations; in the example of FIG. 6, the reward function may increase for every timestamp that the ego vehicle stays in its lane.

[0092] Updating may include step 670, which performs in a simulation environment, recording what happens in the next step, including the RL learning algorithm obtaining the reward.

[0093] Step 670 may include using a particular RL algorithm (e.g., PPO, SAC, TTD3) to iteratively update the network parameters θ to maximize the average reward.

[0094] FIG. 7 shows a method 700 for multi-object RL with visual input.

[0095] Step 710 of method 700 involves computing a sequence of panoramic segmented images (images acquired by the ego vehicle) over a short time window from the ego vehicle's viewpoint, the relative distances to individual objects, X rel,i The method may include receiving a signal from the

[0096] After step 710, the high-level spatio-temporal features X i This may be followed by step 720 of applying a spatio-temporal CNN to the individual instances (objects) to capture

[0097] After step 720, the individual perceptual fields f θ (X i ,i) and the sum Σf θ (X rel,I ,X i , i) may be followed by step 730 of calculating

[0098] After step 730, a differential equation describing the host vehicle acceleration applied to the host vehicle, i.e., a=Σf θ (Xrel,I ,X i , i) may be constructed at step 740 .

[0099] This may be the output of the inference process. Step 740 may be followed by step 450 (not shown).

[0100] The method may include updating one or more network parameters θ using any RL process.

[0101] The method may include a step 760 of performing in a simulated environment, where the RL learning algorithm records what happens in the next step, including the reward obtained.

[0102] RL may envision reward functions that are either learned or handcrafted based on expert demonstrations.

[0103] Step 760 may be followed by step 770, which uses a particular RL algorithm, such as PPO, SAC, TTD3, ​​etc., to iteratively update the network parameters θ to maximize the average reward.

[0104] FIG. 8 shows a method 800 for multi-object BC with kinematic input.

[0105] Step 810 of the method 800 comprises determining the detected object relative kinematics (X rel,i ,V rel,i ), wherein rel,i is the relative position of the detected object i with respect to the ego-vehicle, and V rel,i is the relative speed of the detected object i with respect to the host vehicle. Also, the host vehicle speed V ego Receive.

[0106] After step 810, for each object a perceptual field f θ (X rel,i ,V rel,i ,Vego, A step 820 of calculating i) may follow.

[0107] Step 820 may be followed by step 830 of summing the contributions from the individual perceptual fields. Step 830 may also include normalizing the magnitude of the resulting 2d vector so that it is equal to the maximum magnitude of the individual terms, i.e., N * Σf θ (X rel,i ,V rel,i ,V ego, i).

[0108] After step 830, a differential equation describing the ego vehicle acceleration applied to the ego vehicle, i.e., a=N * Σf θ (X rel,i ,V rel,i ,V ego, A step 840 of constructing i) may follow.

[0109] This may be the output of the inference process. Step 840 may be followed by step 450 (not shown).

[0110] The method may include updating one or more network parameters.

[0111] The method is as follows:

number

[0112] After step 860, the loss function =

number

[0113] FIG. 9 illustrates a method 900 of inference plus a loss function for an adaptive cruise control model implemented using kinematic variables as inputs.

[0114] Step 910 of the method 900 is to determine the position X ego , the vehicle speed V ego , the position of the closest vehicle in front of the vehicle, X CIPV , and the speed V of the nearest vehicle in front of the vehicle. CIPV The method may include receiving a signal from the

[0115] After step 910, the relative position X rel =X ego -X CIPV , and the relative velocity V rel =V ego -V CIPV This may be followed by step 920 of calculating

[0116] Step 920 may be followed by step 930 below. a. The first NN generates a perceptual field function g θ (X rel ,V CIPV ) is calculated. b. The second NN generates the auxiliary function h Ψ (V rel ) is calculated. c. Adjust g to provide the target acceleration (equal to the target force) θ (X rel ,V CIPV ) to h Ψ (V rel ) to multiply it.

[0117] This may be the output of the inference process. Step 930 may be followed by step 450 (not shown).

[0118] The method may include updating one or more NN parameters.

[0119] The method is as follows:

number

[0120] After step 960, the loss function =

number

[0121] Visualization

[0122] Perception fields are a novel computational framework for generating driving policies for autonomous ego-vehicles in different traffic environments (e.g., highway, urban, rural) and for different driving tasks (e.g., collision avoidance, lane keeping, ACC, overtaking, etc.). Perception fields are attributes of road objects and encode the force fields emerging from each road object i (e.g., other vehicles, pedestrians, traffic signs, road boundaries, etc.) of category c that act on the ego-vehicle and trigger driving behaviors. The key to obtaining desired driving behaviors from a perceptual field representation of the ego-vehicle environment is to model the force field such that it is general enough to enable a wide variety of driving behaviors, but specific enough to enable efficient learning using human driving data. The application of perception fields has several advantages over existing methods (e.g., end-to-end approaches), such as task decomposition and improved explainability and generalization performance, resulting in a wide variety of driving behaviors.

[0123] FIG. 10 shows an example of a method 3000 for visualization.

[0124] According to one embodiment, the method 3000 begins by obtaining 3010 object information regarding one or more objects located within the vehicle's environment.

[0125] According to one embodiment, step 3010 also includes analyzing the object information. The analysis may include determining position information and / or movement information of the one or more objects. The position and movement information may include a relative position (with respect to the vehicle) and / or a relative movement (with respect to the vehicle) of the one or more objects.

[0126] According to one embodiment, step 3010 is followed by step 3020 of determining, by the processing circuitry and based on the object information, one or more virtual fields of one or more objects, the one or more virtual fields representing the possible influence of the one or more objects on the behavior of the vehicle.

[0127] Step 3020 may be driven from a virtual physical model. For example, if the virtual physical model represents an object as an electromagnetic charge, the one or more virtual fields are virtual electromagnetic fields, and the virtual forces represent electromagnetic forces caused by the virtual charges. For example, if the virtual physical model is a mechanical model, then the virtual force fields are driven from the acceleration of the object. It is noted that the processing circuitry may be trained using at least any of the training methods shown in some applications, for example, by applying any one of the methods 200, 300, and 400, mutatis mutandis. The training may be based, for example, on behavioral cloning (BC) and / or reinforcement learning (RL).

[0128] According to one embodiment, step 3020 is followed by a step 3030 of generating, based on the one or more fields, visualization information used to visualize one or more virtual fields to the driver.

[0129] According to one embodiment, the visualization information represents multiple field lines for each virtual field.

[0130] According to one embodiment, the multiple field lines per virtual field form multiple ellipses per object for one or more objects.

[0131] The visualization information may be displayed as part of a graphical interface that includes a graphical element representing the virtual field. The method may include providing a visual representation of the virtual field to a user (e.g., an operator of the vehicle).

[0132] The visualization information and / or the graphical user interface may be displayed on a vehicle display, on a user device display (e.g., on a mobile phone), etc.

[0133] FIG. 11 shows an example of an image 3091 of the vehicle's environment as seen from the vehicle sensor, with multiple field lines 3092 for each virtual field of an object that is another vehicle.

[0134] FIG. 12 shows an example of a method 3001 for visualization.

[0135] According to one embodiment, the method 3001 begins with step 3010 .

[0136] According to one embodiment, step 3010 is followed by step 3020 .

[0137] According to one embodiment, step 3020 is followed by step 3040 of determining one or more virtual forces substantially exerted on the vehicle by one or more objects based on the one or more virtual fields.

[0138] The one or more virtual forces are associated with the physics model and represent the influence of the one or more objects on the behavior of the vehicle.

[0139] According to one embodiment, the virtual force is a force field.

[0140] According to one embodiment, the virtual force is an electric potential field.

[0141] According to one embodiment, the one or more virtual forces are represented by a virtual curve indicating a magnitude of the virtual force.

[0142] The strength of the virtual force may be represented by one or more of the strength of the virtual curve, the shape of the virtual curve, or the size (eg, width, length, etc.) of the virtual curve.

[0143] Step 3040 may include determining a total virtual force substantially applied to the vehicle. The total virtual force may be an aggregate of one or more virtual forces.

[0144] According to one embodiment, step 3040 is followed by step 3050 of calculating a desired virtual acceleration of the vehicle based on the virtual forces.

[0145] Step 3050 may be performed based on assumptions regarding a relationship between the virtual forces and a desired virtual acceleration of the vehicle. For example, the virtual forces may have a virtual acceleration (effectively applied to the vehicle) and the desired virtual acceleration of the vehicle may offset the virtual acceleration effectively applied to the vehicle.

[0146] According to one embodiment, the desired virtual acceleration may have substantially the same magnitude as the applied acceleration, but directed in the opposite direction.

[0147] According to one embodiment, the desired virtual acceleration has a magnitude different from the magnitude of the actual applied acceleration.

[0148] According to one embodiment, the desired virtual acceleration has a direction that is substantially non-opposite to the direction of the applied acceleration.

[0149] According to one embodiment, step 3050 is followed by step 3060 of generating visualization information for use in visualizing one or more virtual fields and force information based on the one or more fields.

[0150] The force information may represent one or more virtual forces and / or a desired virtual acceleration.

[0151] According to one embodiment, the visualization information represents multiple field lines for each virtual field.

[0152] According to one embodiment, the multiple field lines per virtual field form multiple ellipses per object for one or more objects.

[0153] According to one embodiment, step 3060 is followed by a step 3070 of responding to the visualization information.

[0154] Step 3060 may include transmitting the visualization information, and / or storing the visualization information, and / or displaying the content represented by the visualization information.

[0155] Step 3060 may include displaying the visualization information as part of a graphical interface that includes graphical elements representing the virtual field and / or the desired acceleration, etc. The graphical user interface provides a user (e.g., a driver of a vehicle) with a visual representation of the virtual field and / or the desired acceleration.

[0156] According to one embodiment, step 3050 is followed by step 3080 which is further responsive to a desired virtual acceleration of the vehicle.

[0157] According to one embodiment, step 3080 includes at least one of the following: a. Triggering driving-related maneuver decisions based on one or more virtual fields. b. Triggering the performance of a driving-related maneuver based on one or more virtual fields. c. Request or command the performance of driving-related operations. d. Triggering a calculation of a driving related maneuver based on the desired virtual acceleration. e. Requesting or commanding a calculation of a driving related maneuver based on the desired virtual acceleration. f. Sending information regarding the desired virtual acceleration to a control unit of the vehicle. g. Control the vehicle - Transfer control from the driver to the autonomous driving unit.

[0158] 13 shows an example of a vehicle's environment as seen from an aerial image with multiple field lines for each virtual force applied to the vehicle of an object that is another vehicle. In FIG. 13, any one (or a combination of two or more) of the color, direction, and magnitude of the points shown in FIG. 13 may indicate that one or more virtual forces are being applied to those points.

[0159] Figure 14 shows an example of an image 3093 of the vehicle's environment as seen from the vehicle sensors, with a number of field lines 3092 for each virtual field of an object, which is another vehicle, and with a representation 3094 of the virtual forces exerted by the object. In Figure 14, the representation 3094 is a portion of an ellipse that also includes a number of field lines 3092.

[0160] FIG. 15 shows an image 1033 of an example scene.

[0161] FIG. 15 shows an example of a vehicle 1031 located within a first road segment.

[0162] A pedestrian 1022 begins to traverse the segment in front of the vehicle 1301. The pedestrian is represented by a pedestrian virtual field (shown by virtual equipotential field lines 1022' and force indicators 1025) and Figure 15 also shows a directional vector 1041 (which may or may not be displayed) repelling the vehicle 1031.

[0163] Another vehicle 1039 is traveling in the oncoming lane and has another vehicle virtual field 1049, another force indicator 1049, and exerts another virtual force 1049 (which may or may not be displayed) on the vehicle 1031.

[0164] Virtual forces applied to the vehicle 1031 (due to pedestrians and other vehicles) are indicated at 1071 (which may or may not be displayed). Figure 15 also shows the desired acceleration of the vehicle 1042. The desired acceleration may or may not be displayed.

[0165] Figure 16 is an image 3095 showing the vehicle's environment and another example of a visualization using a scalar field. The visualization was generated by sampling several points and determining where in the environment the force from the virtual field equals zero. Figure 16 includes concentric ellipses 3096 that decrease in intensity until the location where the virtual force is zero indicates where the object's virtual field "ends." This type of visualization shows how much force would be exerted on the vehicle if it were to be located at a location in the virtual field.

[0166] Driving-related improved fields

[0167] There is a problem with current ADAS and AV technologies (especially vision-based) in that the control policies result in jerky and non-human driving behavior (e.g. late braking). Perceptual field models can be further improved to mitigate the problem and result in a comfortable driving experience.

[0168] To the extent that the perceptual field is trained by imitation learning, it naturally produces a behavior of the vehicle similar to human driving. To the extent that the average human driving is perceived as comfortable by the driver and passengers, the driving is guided by the perceptual field. However, there is a possibility to promote a sense of comfort by modifying the training algorithm of the perceptual field.

[0169] Research on the psychology of the driving experience and physiological responses to the driving experience suggests that minimizing jerk plays a crucial role in the experience of comfort. The further requirement of minimizing jerk can be easily incorporated into the perceptual field framework by including jerk in the loss / reward function during training to generate perceptual fields with improved comfort.

[0170] The comfort-enhanced perceptual field can also take into account other factors such as acceleration and speed, i.e. any factor known to affect the sense of comfort can be included in the loss or reward function, thus improving the basic perceptual field model.

[0171] It has been found that enhancing the virtual field based on comfort or other factors can further improve driving.

[0172] FIG. 17 illustrates an example of a method 4000 for enhancing a driving-related virtual field.

[0173] According to one embodiment, the method 4000 begins by obtaining 4010 object information for one or more objects located within the vehicle's environment.

[0174] According to one embodiment, step 4010 also includes analyzing the object information. The analysis may include determining position information and / or movement information of the one or more objects. The position and movement information may include a relative position (with respect to the vehicle) of the one or more objects and / or a relative movement (with respect to the vehicle) of the one or more objects.

[0175] According to one embodiment, step 4010 is followed by step 4020 of determining, by the processing circuit and based on the object information, one or more virtual fields of the one or more objects, the determination of the one or more virtual fields being based on a virtual physics model, the one or more virtual fields representing possible influences of the one or more objects on the behavior of the vehicle, the virtual physics model being constructed based on one or more physical laws and at least one further driving-related parameter.

[0176] According to one embodiment, the processing circuitry has been trained based on a baseline driving pattern, and the at least one further driving related parameter comprises a driver parameter related to one or more differences between one or more types of driving patterns of the driver and the one or more types of baseline driving patterns.

[0177] According to one embodiment, the at least one further driving related parameter comprises a driver driving pattern related parameter related to one or more driving patterns of the driver.

[0178] According to one embodiment, the at least one further driving related parameter comprises a fuel consumption parameter.

[0179] According to one embodiment, the at least one further driving related parameter comprises a safety parameter.

[0180] According to one embodiment, the at least one further driving related parameter comprises a comfort parameter.

[0181] The comfort parameter may relate to the comfort of the driver of the vehicle while driving.

[0182] The comfort parameter may relate to the comfort of one or more road users located outside the vehicle, for example the comfort of other drivers and / or pedestrians located in the vicinity of the vehicle (for example within 0.1 to 20 m of the vehicle).

[0183] The comfort parameter may relate to the comfort of one or more passengers other than the driver during operation.

[0184] The comfort parameter may relate to the comfort of the driver and one or more other passengers in the vehicle while it is moving.

[0185] The comfort of the driver may be prioritized over the comfort of one or more passengers for safety reasons.

[0186] The driver, passengers or any other authorized entity may define the manner in which the comfort of either should be taken into account.

[0187] The comfort level may be set by the driver, passengers, or any other authorized entity.

[0188] For example, if the virtual physics model represents objects as electromagnetic charges, then the one or more virtual fields are virtual electromagnetic fields and the virtual forces represent electromagnetic forces arising due to the virtual charges.

[0189] For example, the virtual physical model may be a mechanical model, in which case the virtual force field is driven from the acceleration of the object. Note that the processing circuitry may be trained using at least any of the training methods shown in some applications, for example, by applying any one of the methods 200, 300 and 400, mutatis mutandis. The training may be based on, for example, behavioral cloning (BC) and / or reinforcement learning (RL). The training may also take into account one or more further driving-related parameters. For example, the loss function may be given a desired value of the further driving-related parameter and an estimated current driving-related parameter, and aims to reduce the gap between the desired further driving-related parameter and the estimated current further driving-related parameter.

[0190] Given that a further driving-related parameter is comfort, comfort can be assessed based on explicit feedback from the driver, from monitoring physiological parameters (heart rate, sweating, blood pressure, skin color changes, etc.), detecting loud voices or other audio and / or visual information indicative of stress.

[0191] According to one embodiment, step 4020 is followed by step 4030 of determining total virtual forces applied to the vehicle according to the virtual physics model by one or more objects.

[0192] According to one embodiment, step 4030 is followed by step 4040 of determining a desired virtual acceleration of the vehicle based on one or more virtual fields.

[0193] Step 4040 may be performed based on assumptions regarding a relationship between the virtual forces and a desired virtual acceleration of the vehicle. For example, the virtual forces may have a virtual acceleration (effectively applied to the vehicle) and the desired virtual acceleration of the vehicle may offset the virtual acceleration effectively applied to the vehicle.

[0194] According to one embodiment, the desired virtual acceleration may have substantially the same magnitude as the applied acceleration, but directed in the opposite direction.

[0195] According to one embodiment, the desired virtual acceleration has a magnitude different from the magnitude of the actual applied acceleration.

[0196] According to one embodiment, the desired virtual acceleration has a direction that is substantially non-opposite to the direction of the applied acceleration.

[0197] According to one embodiment, step 4040 is performed without regard to the current comfort parameters of the driver of the vehicle or the current comfort parameters of any other passengers of the vehicle.

[0198] According to one embodiment, the method 4000 includes obtaining a current comfort parameter of a driver of the vehicle, and step 4040 is performed based on the current comfort parameter of the driver and / or based on the current comfort parameter of any other passengers of the vehicle.

[0199] According to one embodiment, the method 4000 includes the step 4060 of triggering a decision to perform a driving related action and / or triggering execution of the decision to perform a driving related action.

[0200] According to one embodiment, step 4060 is preceded by step 4020, in which any of the triggers are based on one or more virtual fields.

[0201] According to one embodiment, step 4060 is preceded by step 4030, where any of the triggers are based on the total virtual force applied to the vehicle.

[0202] According to one embodiment, step 4060 is preceded by step 4040, where any of the triggers are based on a desired virtual acceleration of the vehicle.

[0203] FIG. 18 shows an example of a method 4100 for training.

[0204] According to one embodiment, the method 4100 includes a step 4110 of obtaining information required to train the neural network. The information may include desired driver patterns and / or desired values ​​of further driving related parameters and / or a virtual physical model.

[0205] According to one embodiment, step 4110 is followed by step 4120 of training the neural network to determine one or more virtual fields of one or more objects based on the object information, wherein the determination of the one or more virtual fields is based on a virtual physics model, wherein the one or more virtual fields represent possible influences of the one or more objects on the vehicle behavior, and wherein the virtual physics model is constructed based on one or more physical laws and at least one further driving related parameter.

[0206] According to one embodiment, step 4120 may be followed by training a neural network (or another computational entity) to determine a total virtual force applied to the vehicle according to the virtual physics model by one or more objects.

[0207] According to one embodiment, step 4120 may be followed by training a neural network (or another computational entity) to determine a desired virtual acceleration of the vehicle based on the one or more virtual fields.

[0208] Virtual Field Personalization

[0209] A baseline perception field (PF) is trained using human and simulation data, but the possibility exists for an end user to "fine-tune" the training so that the behavior evoked by the perception field more closely matches the behavior of the end user for a given vehicle given the PF technology.

[0210] According to one embodiment, this is as follows: a. By allowing the vehicle's PF software to "relax" the weights and biases of the last layer of the neural network that contains the PF, b. For each maneuver in the family of maneuvers: i. having the end user perform one of a set of preselected driving maneuvers that are recorded by the software along with associated data regarding the environment; ii. The difference between this driving maneuver and the driving maneuver prescribed by the default PF is considered as the loss function, so that backpropagation updates the weights and biases of the last layer of the neural network, including the PF; This can be done by repeating the following:

[0211] FIG. 19 illustrates one example of a method 4200 for updating a neural network.

[0212] According to one embodiment, the method 4200 begins by step 4210 of obtaining a neural network that is trained to map object information regarding one or more objects located in the vehicle's environment to one or more virtual fields of the one or more objects, the determination of the one or more virtual fields being based on a virtual physics model, the one or more virtual fields representing possible influences of the one or more objects on the vehicle's behavior, the virtual physics model being constructed based on one or more physical laws and at least one further driving related parameter.

[0213] Examples of such neural networks and / or examples of training such neural networks are provided in the preceding text and / or figures.

[0214] According to one embodiment, the neural network is implemented by a processing circuit, such as a processing circuit performing method 4000.

[0215] Obtaining may include receiving a neural network (e.g., without training the neural network) or generating a neural network (e.g., generating may include training).

[0216] According to one embodiment, step 4210 is followed by step 4220 of fine-tuning at least a portion of the neural object based on one or more fine-tuning parameters.

[0217] According to one embodiment, the fine-tuning is performed with the same loss function used during training. Alternatively, the fine-tuning is performed with a new loss function used during training. In any case, the loss function can be determined in any manner and can be pre-defined and / or applied to the fine-tuning stage.

[0218] According to one embodiment, step 4220 does not involve training or retraining the entire neural network.

[0219] Step 4220 specifically involves limiting the resources allocated to fine-tuning to the resources needed to fully train the neural network.

[0220] The resource limitations may include at least one of the following: Fine-tuning a portion, but not all, of the neural network. This portion may be a single layer, or it may be two or more layers, up to 1, 5, 10, 15, 25, 30, 35, 40, 45, 50, 55 percent of the entire neural network. b. Limiting the size of the dataset, e.g., limiting fine-tuning to images acquired by the vehicle during a limited time period, e.g., less than 1, 5, 10, 15, 30 minutes, less than 1, 2, 3, 5 hours, less than 1, 2 days, etc. Another example of size limitation is limiting the dataset to less than 100, 200, 500, 1000, 2000, 5000 images and / or less than 0.001%, 0.01%, 0.1%, 1%, 5% of the size of the dataset used to train the neural network. c. Limit the learning rate. d. Limiting the neural network parameters affected by fine-tuning.

[0221] When fine-tuning is applied in a resource-constrained manner, various computer science benefits can be obtained: much less computational resources and / or much less memory resources may be required compared to fully training or retraining the entire neural network.

[0222] The above-mentioned fine-tuning can be performed by a more compact neural network and loss function, adapting the neural network according to one or more fine-tuning parameters.

[0223] According to one embodiment, fine-tuning may be performed by the vehicle and may not require highly complex neural networks and / or infrastructure and / or large data sets.

[0224] The reduction in resources may be at least 5, 10, 20, 50, 100, 200, 500, 1000 times and even more.

[0225] According to one embodiment, the one or more fine-tuning parameters include fuel consumption and / or vehicle wear and / or vehicle passenger comfort and / or safety parameters and / or any other driving related parameters.

[0226] According to one embodiment, the one or more fine-tuning parameters are a driving pattern undertaken by the vehicle under the control of a driver of the vehicle, the driving pattern may or may not be predetermined.

[0227] According to one embodiment, at least a portion may include (or may be limited to) one layer, or two or more layers, or all layers of a neural network, and may be modified during retraining.

[0228] For example, step 4220 may include updating one or more neural network parameters, such as at least one of the weights and biases of a last layer of the neural network, while leaving the weights and biases of other layers of the neural network unchanged.

[0229] For yet another example, only the weights and biases of one of the last layers are modified during step 4220 while leaving the other layers unchanged.

[0230] Any other layers of the neural network may be modified during retraining.

[0231] The neural network was trained to mimic a baseline driving pattern. This could be done using RL and / or BC.

[0232] According to one embodiment, step 4220 includes calculating a difference between a driving pattern performed by the driver and a baseline driving pattern and using the difference to modify the neural network. For example, the difference may be fed into a loss function.

[0233] FIG. 20 illustrates an example of a method 4300 for fine-tuning a neural network.

[0234] According to one embodiment, the method 4300 includes identifying 4310 desired driving patterns for various situations. The identification may be part of behavioral cloning or part of reinforcement learning.

[0235] The situation may be at least one of (a) the location of the vehicle, (b) one or more weather conditions, (c) one or more context parameters, (d) road conditions, and (e) traffic parameters. The road conditions may include roughness of the road, the level of maintenance of the road, the presence of potholes or other relevant road hazards, whether the road is slippery or covered with snow or other particles. The traffic parameters and the one or more context parameters may include time (hour, day, period or year, specific time on a specific day, etc.), traffic load, distribution of vehicles on the road, behavior of one or more vehicles (aggressive, calm, predictable, unpredictable, etc.), presence of pedestrians close to the road, presence of pedestrians close to the vehicle, presence of pedestrians away from the vehicle, pedestrian behavior (aggressive, calm, predictable, unpredictable, etc.), danger associated with driving within the vicinity of the vehicle, complexity associated with driving the vehicle, presence (close to the vehicle) of at least one of kindergartens, schools, crowds, etc. An illustrative example of a situation is provided in U.S. patent application Ser. No. 16 / 729,589, entitled "SITUATION BASED PROCESSING," which is incorporated herein by reference.

[0236] The proposed driving patterns are generated by using a neural network (NN) and represent virtual forces applied to the vehicle by one or more objects used to apply driving-related operations of the vehicle, the virtual forces being associated with a virtual physical model representing the influence of the one or more objects on the vehicle's behavior. The proposed driving patterns are generated by the NN when the NN is provided with sensed information units capturing various situations.

[0237] According to one embodiment, step 4310 is followed by step 4315 of obtaining suggested driving patterns for various situations. The suggested driving patterns are generated by using a NN and represent virtual forces applied to the vehicle by one or more objects used in applying the driving-related operations of the vehicle. The virtual forces are associated with a virtual physical model that represents the influence of the one or more objects on the vehicle's behavior. Examples of the virtual physical model and / or the virtual forces are shown in the previous part of this specification and / or in the figures before FIG. 20.

[0238] According to one embodiment, step 4315 is followed by step 4320 of fine-tuning at least a portion of the neural network.

[0239] According to one embodiment, the fine-tuning is performed based on one or more fine-tuning parameters, at least some examples of which are given above.

[0240] According to one embodiment, the fine-tuning may be performed by the same entity as the training of the neural network, for example, a manufacturing entity and / or a programming entity and / or a neural network training and fine-tuning entity.

[0241] According to one embodiment, the fine tuning is performed after a software update.

[0242] According to one embodiment, the fine tuning is performed before the vehicle is delivered to a user.

[0243] According to one embodiment, the fine tuning is initiated by the user of the vehicle.

[0244] According to one embodiment, the fine-tuning is based on sensed data units acquired during one or more driving sessions of the vehicle.

[0245] According to one embodiment, the fine-tuning is based on driving parameters and a relationship between a desired driving pattern and a suggested driving pattern.

[0246] According to one embodiment, the fine-tuning is based on a relationship between a desired driving pattern and a suggested driving pattern.

[0247] According to one embodiment, the fine-tuning includes reducing the difference between the desired driving pattern and the proposed driving pattern (e.g., using a loss function). The reduction may also be responsive to other fine-tuning parameters.

[0248] According to one embodiment, step 4320 includes limiting resources allocated to fine-tuning, particularly with respect to resources needed to fully train the neural network.

[0249] The resource limitations may include at least one of the following: Fine-tuning a portion, but not all, of the neural network. This portion may be a single layer, or it may be two or more layers, up to 1, 5, 10, 15, 25, 30, 35, 40, 45, 50, 55 percent of the entire neural network. b. Limiting the size of the dataset, e.g., limiting fine-tuning to images acquired by the vehicle during a limited time period, e.g., less than 1, 5, 10, 15, 30 minutes, less than 1, 2, 3, 5 hours, less than 1, 2 days, etc. Another example of size limitation is limiting the dataset to less than 100, 200, 500, 1000, 2000, 5000 images and / or less than 0.001%, 0.01%, 0.1%, 1%, 5% of the size of the dataset used to train the neural network. c. Limit the learning rate. d. Limit some neural network parameters that are affected by fine-tuning.

[0250] When fine-tuning is applied in a resource-constrained manner, various computer science benefits can be obtained: much less computational resources and / or much less memory resources may be required compared to fully training or retraining the entire neural network.

[0251] The above-mentioned fine-tuning can be performed by a more compact neural network and loss function, adapting the neural network according to one or more fine-tuning parameters.

[0252] According to one embodiment, fine-tuning may be performed by the vehicle and may not require highly complex neural networks and / or infrastructure and / or large data sets.

[0253] The reduction in resources may be at least 5, 10, 20, 50, 100, 200, 500, 1000 times and even more.

[0254] FIG. 21 shows an example of the method 4350.

[0255] According to one embodiment, the method 4350 begins by step 4360 of obtaining a neural network (NN) representing virtual forces applied to the vehicle by one or more objects for use in generating suggested driving patterns and applying driving-related operations of the vehicle, the virtual forces being associated with a virtual physics model representing the influence of the one or more objects on the behavior of the vehicle.

[0256] According to one embodiment, step 4350 is followed by step 4370 of fine-tuning at least a portion of the neural network based on one or more fine-tuning parameters.

[0257] According to one embodiment, step 4370 includes adjusting only selected portions of the NN.

[0258] According to one embodiment, step 4370 includes fine-tuning only selected layers of the NN.

[0259] According to one embodiment, step 4370 involves fine-tuning only the last layer of the NN.

[0260] According to one embodiment, step 4370 is triggered by the driver of the vehicle, for example by using a mobile device in communication with the vehicle, interacting with a man-machine interface (voice commands and / or a touch screen and / or a knob or button interface).

[0261] According to one embodiment, step 4370 is triggered by a driving action associated with the driver of the vehicle, for example performing a particular driving maneuver.

[0262] According to one embodiment, step 4370 includes that the fine-tuning is triggered by a software update. The software update may allow the driver to select whether or not to perform the fine-tuning. Alternatively, the fine-tuning may be automatically triggered by a software update.

[0263] According to one embodiment, step 4370 includes limiting the size of the dataset used during fine-tuning to be 1 percent less than the dataset used to train the NN.

[0264] The method may include obtaining a desired driving pattern (see, e.g., FIG. 20), where step 4370 includes reducing the difference between the desired driving pattern and the proposed driving pattern. Thus, the fine-tuning may provide a driving pattern that better mimics the desired driving pattern.

[0265] In the foregoing specification, the invention has been described with reference to specific examples of embodiments thereof. It will be apparent, however, that various modifications and changes may be made therein without departing from the broader spirit and scope of the invention as set forth in the appended claims. Moreover, the terms "front," "back," "top," "bottom," "over," "under" and the like in the description and claims, where applicable, are used for descriptive purposes and not necessarily to describe permanent relative positions.

[0266] It will be understood that the terms so used are interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operating in other orientations than those illustrated or otherwise described herein, for example. Additionally, the terms "assert" or "set" and "negate" (or "deassert" or "clear") are used herein to refer to the causing of a signal, status bit, or similar device to its logically true or logically false state, respectively. If the logically true state is a logic level 1, then the logically false state is a logic level 0. Also, if the logically true state is a logic level 0, then the logically false state is a logic level 1.

[0267] Those skilled in the art will recognize that the boundaries between logical blocks are merely illustrative, and that alternative embodiments may merge logical blocks or circuit elements, or impose alternative decompositions of functionality among various logical blocks or circuit elements. Accordingly, it should be understood that the architectures illustrated herein are merely illustrative, and that in fact many other architectures may be implemented which achieve the same functionality. Any arrangement of components to achieve the same functionality is effectively "associated" such that the desired functionality is achieved.

[0268] Thus, any two components herein that are combined to achieve a particular function may be considered to be "associated" with one another such that the desired functionality is achieved, regardless of architecture or intermediate components. Similarly, any two components so associated may also be considered to be "operably connected" or "operably coupled" with one another such that the desired functionality is achieved. Moreover, those skilled in the art will recognize that the boundaries between operations described above are merely exemplary. Operations may be combined into a single operation, a single operation may be distributed among additional operations, and operations may be performed with at least partial overlap in time.

[0269] Moreover, alternative embodiments may include multiple instances of a particular operation, and the order of operations may be changed in various other embodiments. Also, for example, in one embodiment, the illustrated examples may be implemented on a single integrated circuit or as circuits located within the same device. Alternatively, the examples may be implemented as any number of separate integrated circuits or separate devices interconnected with each other in an appropriate manner.

[0270] However, other modifications, variations and alternatives are possible. Accordingly, the specification and drawings are to be regarded in an illustrative rather than a restrictive sense. In the claims, any reference signs placed between parentheses shall not be construed as limiting the scope of the claims. The word "comprising" does not exclude the presence of elements or steps other than those listed in a claim.

[0271] Moreover, the terms "a" or "an" as used herein are defined as one or more than one. Also, the use of preface phrases such as "at least one" and "one or more" in a claim should not be construed as suggesting that the preface of another claim element with the indefinite article "a" or "an" limits any particular claim containing such prefaced claim element to an invention containing only one such element, even if the same claim contains the preface phrase "one or more" or "at least one" and an indefinite article such as "a" or "an". The same is true for the use of definite articles. Unless otherwise stated, terms such as "first" and "second" are used to arbitrarily distinguish between the elements that such terms describe. Thus, these terms are not necessarily intended to indicate a temporal or other prioritization of such elements.

[0272] The mere fact that certain measures are recited in mutually different claims does not indicate that a combination of these measures cannot be used to advantage. While certain features of the invention have been illustrated and described herein, many modifications, substitutions, changes and equivalents will immediately occur to those skilled in the art. It is therefore to be understood that the appended claims are intended to cover all such modifications and changes as fall within the true spirit of the invention.

[0273] It will be understood that various features of the embodiments of the present disclosure that are, for clarity, described in the context of a separate embodiment, may also be provided in combination in a single embodiment. Conversely, various features of the embodiments of the present disclosure that are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination. It will be appreciated by those skilled in the art that the embodiments of the present disclosure are not limited by what has been particularly shown and described above. Instead, the scope of the embodiments of the present disclosure is defined by the appended claims and their equivalents.

Claims

1. A method for improving the virtual field related to driving, To obtain object information about one or more objects located within the vehicle's environment, The processing circuit determines, based on the object information, one or more virtual fields of the one or more objects, wherein the determination of the one or more virtual fields is based on a virtual physical model, the one or more virtual fields represent the potential influence of the one or more objects on the behavior of the vehicle, and the virtual physical model is constructed based on one or more physical laws and at least one further driving-related parameter. Methods that include...

2. The method according to claim 1, wherein the at least one further driving-related parameter is a comfort parameter.

3. The method according to claim 1, comprising determining a desired virtual acceleration of the vehicle based on the one or more virtual fields.

4. The method according to claim 3, wherein the method comprises obtaining the current comfort parameters of the driver of the vehicle, and the determination of the desired virtual acceleration is also performed based on the current comfort parameters of the driver.

5. The method according to claim 3, wherein the processing circuit is trained based on a reference driving pattern, and the at least one further driving-related parameter includes a driver parameter relating to one or more differences between one or more types of driving patterns of the driver and one or more types of the reference driving pattern.

6. The method according to claim 1, wherein the at least one further driving-related parameter includes a fuel consumption parameter.

7. The method according to claim 1, comprising triggering a decision on an operation-related operation based on one or more virtual fields.

8. The method according to claim 1, comprising triggering the execution of an operation-related operation based on one or more virtual fields.

9. A non-temporary computer-readable medium for improving the virtual field related to driving, A command to obtain object information about one or more objects located within the vehicle's environment, An instruction to determine one or more virtual fields of one or more objects by a processing circuit and based on the object information, wherein the determination of the one or more virtual fields is based on a virtual physical model, the one or more virtual fields represent the potential influence of the one or more objects on the behavior of the vehicle, and the virtual physical model is constructed based on one or more physical laws and at least one further driving-related parameter, A non-temporary, computer-readable medium for storing data.

Images