Machine learning model for vehicle operation

Abstract

The invention provides a machine learning model for vehicle operation. A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: in response to a driving context of a vehicle being a first driving context, generate a first driving context; executing the machine learning model on the vehicle with the first portion of the machine learning model enabled and the second portion of the machine learning model disabled; and in response to the driving context being a second driving context, executing the machine learning model with the first portion disabled and the second portion enabled.

Description

Technical Field

[0001] This disclosure describes a machine learning model used in vehicles. Background Technology

[0002] Modern vehicles typically include control modules. These control modules are various computing devices. They can be programmed to perform different vehicle functions. Typical control modules in a vehicle include engine control modules, body control modules, accessory control modules, power steering control modules, anti-lock braking control modules, etc. A vehicle may contain fifty to one hundred control modules. Summary of the Invention

[0003] The machine learning model comprises at least a first part and a second part (e.g., as different heads within a deep neural network). A computer on the vehicle is programmed to execute the machine learning model, wherein a portion of the model is selectively enabled based on the vehicle's driving context. A driving context is data or a dataset that influences vehicle driving. For example, a driving context can be the vehicle's operating mode (e.g., whether adaptive cruise control is engaged) or environmental conditions experienced by the vehicle (e.g., day versus night, rain versus sunshine). The computer is programmed to enable the first part and disable the second part in response to the driving context being a first driving context; and to enable the second part and disable the first part in response to the driving context being a second driving context. Therefore, the computer stores a single machine learning model that can be used in a dedicated manner across multiple driving contexts, rather than multiple machine learning models for multiple driving contexts. This conserves memory and improves processing speed on a vehicle with a limited total capacity. The computing hardware for the vehicle can be selected accordingly (e.g., a single control module instead of two control modules). Furthermore, if the driving context depends on the vehicle's trim package or customer selection, the same choice of computing hardware can be used across different vehicles, rather than using different types of computing hardware across different vehicles, thus simplifying the manufacturing process.

[0004] A computer includes a processor and a memory, and the memory stores instructions executable by the processor to: execute a machine learning model on the vehicle with a first part of the machine learning model enabled and a second part of the machine learning model disabled, in response to a first driving situation of the vehicle; and execute the machine learning model with the first part disabled and the second part enabled, in response to a second driving situation.

[0005] In one example, the instructions may also include instructions for actuating vehicle components based on the output of a machine learning model. In another example, the output of the machine learning model may include the detection of objects in the environment surrounding the vehicle.

[0006] In one example, the first part may include at least one first head, and the second part may include at least one second head. In another example, the machine learning model may include a common part, and the at least one first head and the at least one second head may be arranged in the machine learning model to receive input from the common part. In yet another example, the common part may be trained to perform feature extraction on sensor data. In still yet another example, the at least one first head and the at least one second head may be trained to perform object detection based on features from the feature extraction.

[0007] In one example, the first part can be trained to perform object detection, and the second part can be trained to perform object detection.

[0008] In one example, the machine learning model could be a deep neural network.

[0009] In one example, the driving scenario could be the vehicle's operating mode. In another example, the operating mode could indicate whether the vehicle's components are controlled by a computer or by the vehicle's operator.

[0010] In one example, the driving scenario can be the environmental conditions experienced by the vehicle. In another example, the first driving scenario can be daytime, and the second driving scenario can be nighttime.

[0011] In one example, the driving scenario could be weather conditions.

[0012] In one example, the driving context could be the vehicle's location.

[0013] One method includes: executing a machine learning model on the vehicle while enabling a first part of the machine learning model and disabling a second part of the machine learning model, in response to a driving situation of the vehicle being a first driving situation; and executing a machine learning model while disabling the first part and enabling the second part, in response to a driving situation of a second driving situation.

[0014] In one example, the method may also include actuating vehicle components based on the output of a machine learning model.

[0015] In one example, the first part may include at least one first head, the second part may include at least one second head, the machine learning model may include a common part, and the at least one first head and the at least one second head may be arranged in the machine learning model to receive input from the common part. In another example, the common part may be trained to perform feature extraction on sensor data, and the at least one first head and the at least one second head may be trained to perform object detection based on features from the feature extraction.

[0016] In one example, the driving situation can be either the vehicle's operating mode or the environmental conditions experienced by the vehicle. Attached Figure Description

[0017] Figure 1 This is a block diagram of an example vehicle.

[0018] Figure 2 This is a diagram of an example machine learning model being executed on a vehicle.

[0019] Figure 3 This is a flowchart of an example process for executing a machine learning model based on the vehicle's driving context. Detailed Implementation

[0020] Referring to the accompanying drawings, wherein in all the plurality of views the same reference numerals indicate the same parts, computer 105 includes a processor and a memory, and the memory stores instructions that can be executed by the processor to: execute machine learning model 200 on vehicle 100 with a first part 205 of machine learning model 200 enabled and a second part 210 of machine learning model 200 disabled, in response to a driving situation of vehicle 100 being a first driving situation; and execute machine learning model 200 with the first part 205 disabled and the second part 210 enabled, in response to a driving situation being a second driving situation.

[0021] refer to Figure 1 Vehicle 100 can be any passenger or commercial vehicle, such as a sedan, truck, SUV, crossover, van, minivan, taxi, bus, etc. Vehicle 100 includes a computer 105, a communication network 110, sensors 115, a propulsion system 120, a braking system 125, and a steering system 130.

[0022] Computer 105 is a microprocessor-based computing device, such as a general-purpose computing device (including a processor and memory, electronic controller, etc.), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or a combination thereof. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components inside an FPGA can be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuitry). Therefore, computer 105 may include a processor, memory, etc. The memory of computer 105 may include media for storing instructions executable by the processor and for electronically storing data and / or databases, and / or computer 105 may include structures such as those providing programming. Computer 105 may be multiple computers coupled together.

[0023] Computer 105 can transmit and receive data via communication network 110. Communication network 110 can be a controller area network (CAN) bus, Ethernet, WiFi, local area network (LIN), on-board diagnostic connector (OBD-II), and / or any other wired or wireless communication network. Computer 105 can be communicatively coupled to sensor 115, propulsion system 120, braking system 125, steering system 130, and other components via communication network 110.

[0024] Sensor 115 can provide data about the operation of vehicle 100, such as wheel speed, wheel orientation, and engine and transmission data (e.g., temperature, fuel consumption, etc.). Sensor 115 can detect the position and / or orientation of vehicle 100. For example, sensor 115 may include a Global Positioning System (GPS) sensor; an accelerometer, such as a piezoelectric system or a microelectromechanical system (MEMS); a gyroscope, such as a rate gyroscope, a ring laser gyroscope, or a fiber optic gyroscope; an inertial measurement unit (IMU); and a magnetometer. Sensor 115 can detect the external world, including objects and / or characteristics of the environment surrounding vehicle 100, such as other vehicles, road lane markings, traffic lights and / or signs, road users, etc. For example, sensor 115 may include a radar sensor, an ultrasonic sensor, a scanning laser rangefinder, a light detection and ranging (LiDAR) device, and an image processing sensor (such as a camera).

[0025] Computer 105 can be programmed to receive sensor data 225 from sensor 115. For example, computer 105 can receive image data from one or more cameras, distance data from one or more radars or lidars, etc.

[0026] Image data is a series of image frames that form the field of view of the corresponding sensor 115 of the camera. Each image frame is a two-dimensional pixel matrix. The brightness or color of each pixel is represented as one or more numerical values, such as a scalar, unitless value of luminous intensity between 0 (black) and 1 (white), or a value for each of red, green, and blue (e.g., each using an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale). Pixels can be a mixture of representations (e.g., a repeating pattern of scalar values ​​of intensity for three pixels and a fourth pixel with three numerical color values, or some other pattern). The position within an image frame (i.e., its position in the sensor's field of view at the time the image frame is recorded) can be specified in pixel size or coordinates (e.g., a pair of ordered pixel distances), such as multiple pixels from the top edge of the image frame and multiple pixels from the left edge of the image frame.

[0027] The range data can be, for example, a point cloud. Points in the point cloud specify the corresponding locations in the environment relative to the position of a ranging sensor (e.g., radar or lidar) in sensor 115. For example, the range data can be in spherical coordinates, where the ranging sensor is located at the origin of the spherical coordinate system. Spherical coordinates can include: radial distance (i.e., the measured depth from the ranging sensor to the point measured by the ranging sensor); polar angle (i.e., the angle from the vertical axis passing through the ranging sensor to the point measured by the ranging sensor); and azimuth angle (i.e., the angle in the horizontal plane from the horizontal axis passing through the ranging sensor to the point measured by the ranging sensor). The horizontal axis can, for example, be along the vehicle's direction of travel. Alternatively, the ranging sensor can return the points as Cartesian coordinates at the origin of the ranging sensor or as coordinates in any other suitable coordinate system, or the computer 105 can convert the spherical coordinates to Cartesian coordinates or another coordinate system after receiving the range data.

[0028] The propulsion system 120 of vehicle 100 generates energy and converts that energy into motion of vehicle 100. The propulsion system 120 may be a conventional vehicle propulsion subsystem, such as a conventional powertrain including an internal combustion engine coupled to a transmission that transmits rotational motion to the wheels; an electric powertrain including a battery, an electric motor, and a transmission that transmits rotational motion to the wheels; a hybrid powertrain including elements of both a conventional powertrain and an electric powertrain; or any other type of propulsion device. The propulsion system 120 may include an electronic control unit (ECU) that communicates with and receives input from a computer 105 and / or a human operator. The human operator may control the propulsion system 120 via, for example, pedals and / or a gearshift lever.

[0029] Braking system 125 is typically a conventional vehicle braking subsystem and prevents the movement of vehicle 100, thereby slowing and / or stopping vehicle 100. Braking system 125 may include friction brakes, such as disc brakes, drum brakes, band brakes, etc.; regenerative brakes; any other suitable type of brake; or combinations thereof. Braking system 125 may include an electronic control unit (ECU), etc., that communicates with and receives input from computer 105 and / or a human operator. The human operator may control braking system 125 via, for example, the brake pedal.

[0030] Steering system 130 is typically a conventional vehicle steering subsystem and controls the turning of the wheels. Steering system 130 can be a rack and pinion system with electric power steering, a steer-by-wire system (both of which are known), or any other suitable system. Steering system 130 may include an electronic control unit (ECU) that communicates with and receives input from a computer 105 and / or a human operator. The human operator can control steering system 130 via, for example, a steering wheel.

[0031] When vehicle 100 is operated, vehicle 100 is in a driving situation. For the purposes of this disclosure, a "driving situation" is defined as data or datasets that affect the driving of vehicle 100. For example, a driving situation may be the operating mode of vehicle 100 (e.g., whether adaptive cruise control is engaged), the environmental conditions experienced by vehicle 100 (e.g., daytime versus nighttime, rainy versus sunny), or the location of vehicle 100 (e.g., restricted access highway versus surface street), as will be described below in sequence. Computer 105 may be programmed to determine the driving situation, as will be described below regarding the types of driving situations.

[0032] A driving scenario can be an operating mode of vehicle 100. For the purposes of this disclosure, an "operating mode" is defined as data or datasets indicating how one or more components of vehicle 100 (e.g., propulsion system 120, braking system 125, and / or steering system 130) operate. For example, an operating mode can indicate how to control components, such as whether the component is controlled by computer 105 or by the operator of vehicle 100. As an example, an operating mode can indicate whether adaptive cruise control is engaged, which indicates whether propulsion system 120 is controlled by computer 105 (when engaged) or by the operator (when not engaged). Adaptive cruise control is an example of an Advanced Driver Assistance System (ADAS). ADAS is an electronic technology that assists a driver in performing driving and parking functions. Examples of ADAS include forward proximity detection, lane departure detection, blind spot detection, brake actuation, adaptive cruise control, and lane keeping assist systems. An operating mode can indicate whether one or a combination of ADAS features or more advanced autonomous features is engaged. An operating mode can be selected from multiple pre-stored operating modes (e.g., a first operating mode, a second operating mode, etc.). The computer 105 can determine the operating mode by consulting the flags in its memory. The computer 105 sets the flags when the vehicle 100 is placed in a specific operating mode.

[0033] Alternatively or additionally, the driving scenario can be the environmental conditions experienced by vehicle 100. For example, the driving scenario can be ambient light level, weather conditions, etc. The ambient light level can be daytime, nighttime, etc. Computer 105 can determine the ambient light level based on data from sensor 115. Sensor 115 can include an ambient light sensor, which is a photodetector that detects the amount of ambient light present (i.e., the total light level from sources in the environment). The ambient light sensor can be of any suitable type, such as a phototransistor, photodiode, photonic integrated circuit, etc. The computer can determine that the driving scenario is daytime in response to an ambient light level exceeding a threshold and that the driving scenario is nighttime in response to an ambient light level falling below a threshold.

[0034] Weather conditions can be defined by ambient temperature (e.g., above or below freezing), precipitation classification (e.g., rain, snow, clear), wind speed, visibility measurement (e.g., foggy or clear), etc. Computer 105 can determine weather conditions by receiving weather forecasts from external sources or based on data from sensor 115. Sensor 115 may include an external ambient temperature sensor (OATS) that measures ambient temperature. Computer 105 can apply object recognition to image data from a camera received from sensor 115 to detect, for example, precipitation classification.

[0035] Alternatively or additionally, the driving situation can be the location of vehicle 100. For example, the location could be on a specific type of road, within a specific geographic area, etc. The road type could be an expressway (i.e., a restricted-access road, such as a highway or toll road), an unseparated road, a city street, a parking lot, etc. The geographic area could be a city, state or province, country, etc. Computer 105 can store multiple preset location types and can determine the driving situation by selecting from preset locations. Computer 105 can select a preset location type by comparing the location of vehicle 100 returned by sensor 115 (e.g., by a GNSS sensor) with map data that defines the preset location type (e.g., the boundaries defining different types of roads and geographic areas).

[0036] refer to Figure 2 Computer 105 stores machine learning model 200 in memory. Machine learning model 200 may include a common part 215, a first part 205, a second part 210, and possibly other parts, which will be referred to as a third part 220. Computer 105 is programmed to execute machine learning model 200. In general, common part 215 may receive sensor data 225 as input and generate output (e.g., feature map 245). Computer 105 enables one of the first part 205 or the second part 210 (or possibly the third part 220) based on the driving situation determined above, and correspondingly disables the other of the first part 205 or the second part 210. Computer 105 enables a first part 205 and disables a second part 210 (and possibly a third part 220) in response to a first driving scenario, enables a second part 210 and disables a first part 205 (and possibly a third part 220) in response to a second driving scenario, and may enable a third part 220 and disable a first part 205 and a second part 210 in response to a third driving scenario. Regardless of which parts 205, 210, or 220 are enabled, they receive the output from a common part 215 as input and generate an output (e.g., detection 250), which is the overall output of the machine learning model 200. Regardless of which parts 205, 210, or 220 are disabled, these parts will not be executed when computer 105 executes the machine learning model 200. Computer 105 may actuate components based on the output of the machine learning model 200.

[0037] The machine learning model 200 can be a deep neural network. The neural network comprises a series of layers, each of which uses one or more other layers as input. Each layer contains multiple neurons that receive data generated by a subset of neurons in other layers as input and generate outputs that are transmitted to neurons in other layers. The neural network includes multiple weights. Each weight can be applied to a connection between two neurons. Thus, for a given neuron, the outputs from neurons fed into that neuron are weighted by the corresponding weights of the connections from those neurons to the given neuron. The output of each neuron can be a function of the weighted inputs from the input neurons (i.e., n). i =f(w ji *n j ,w ki *n k (,…), where i, j, and k are the indices of neurons, n i It is the output of the i-th neuron, and w ji (This refers to the weights of the connections from neuron j to neuron i). The output of each neuron can also be a function of the bias of the input neuron.

[0038] Deep neural networks can be of any suitable type, such as convolutional neural networks, recurrent neural networks, etc. For example, in a convolutional neural network, each layer uses the layer immediately preceding it as input. Layer types include: convolutional layers, which compute the dot product of weights and small regions of input data; pooling layers, which perform downsampling operations along the spatial dimension; and fully connected layers, which are generated based on the outputs of all neurons in the previous layer.

[0039] Machine learning model 200 may include a common part 215 and multiple headers 230, 235, and 240. Headers 230, 235, and 240 are arranged in machine learning model 200 to receive input from the common part 215; in other words, they receive the output of the common part 215 as input. The common part 215 may be, for example, an encoder. Headers 230, 235, and 240 are included in a first part 205, a second part 210, and a possible third part 220. The first part 205 includes at least one first header 230, the second part 210 includes at least one second header 235, and the third part 220 may include at least one third header 240. Headers 230, 235, and 240 are different subroutines of machine learning model 200, which can be removed without affecting the operation of the rest of machine learning model 200.

[0040] The output of common part 215 includes features. For example, the output of common part 215 may include feature map 245. Feature map 245 includes multiple features. For the purposes of this disclosure, the term "feature" in its computer vision sense is used as a piece of information about the content of an image or point cloud, specifically about whether a certain region of the image or point cloud has certain attributes. The type of feature may include edges, corners, blobs, etc. For an image, feature map 245 provides the location of the feature in the image frame (e.g., represented in pixel coordinates). Compared to the image frame, feature map 245 has a reduced dimension. For example, the output may be a feature pyramid, which includes multiple feature maps 245 of different dimensions. For example, each feature map 245 of a given feature pyramid may be scaled down by a different factor than the image frame, such as by a factor of 2 to different numbers of times (e.g., the range of five feature maps 245 from scaled down by a factor of 2 to three times to scaled down by a factor of 2 to seven times). For another example, the output may embed features in different ways, such as latent vectors or another type of intermediate machine learning output. The features may or may not be easily readable by humans.

[0041] The common part 215 can be trained to perform feature extraction on sensor data 225 to detect features (e.g., generate feature map 245). For example, the common part 215 can be trained to generate feature map 245 from image data (e.g., from image frames of image data). The common part 215 can be a feature extractor. The feature extractor can include one or more suitable techniques for feature extraction, such as low-level techniques such as edge detection, corner detection, blob detection, ridge detection, scale-invariant feature transform (SIFT), etc.; shape-based techniques such as thresholding, blob extraction, template matching, Hough transform, generalized Hough transform, etc.; flexible methods such as deformable parameterized shapes, active contours, etc.; and so on. The common part 215 includes machine learning operations. For example, the first feature extractor 215 can include a residual network (ResNet) layer followed by a convolutional neural network. For another example, the common part 215 can be the encoder part of an encoder-decoder network, and the common part 215 can be trained as part of training the encoder-decoder network. In another example, public part 215 can be trained as part of training machine learning model 200, as described below.

[0042] The output of machine learning model 200 may include object detection 250 in the environment surrounding vehicle 100. First part 205, second part 210, and third part 220 can be trained to perform object detection. Different heads 230, 235, and 240 can be trained to perform object detection of different corresponding types of objects. For example, one or more heads 230, 235, and 240 can be trained to detect lane lines on a road on which vehicle 100 is traveling, one or more heads 230, 235, and 240 can be trained to detect road signs, one or more heads 230, 235, and 240 can be trained to detect other vehicles, etc. Heads 230, 235, and 240 can be trained to perform object detection based on features from feature extraction (i.e., based on the output of common part 215). Different heads 230, 235, and 240 can be trained to perform object detection at different corresponding resolutions (e.g., using different feature maps 245 from the feature pyramid). Heads 230, 235, and 240 can detect objects of the corresponding type at the corresponding resolution using any machine learning technique suitable for object detection, such as: knowledge-based techniques, such as multi-resolution rule-based methods; feature-invariant techniques, such as edge grouping, spatial gray-level correlation matrix, or Gaussian mixture; template matching techniques, such as shape templates or active shape models; or appearance-based techniques, such as decomposition and clustering, Gaussian distribution and multilayer perceptron, neural networks, support vector machines with multinomial kernels, naive Bayes classifiers with joint statistics of local appearance and location, or higher-order statistics with hidden Markov models.

[0043] Computer 105 can be programmed to generate corresponding bounding boxes around objects in the detected sensor data 225. For an image frame, each bounding box can be defined by the pixel coordinates of the diagonal of the bounding box, thus specifying a rectangle, or for a point cloud, it can be defined by the spatial coordinates of a specified rectangular prism. For example, computer 105 can generate bounding boxes around regions from object detection, with corresponding heads 230, 235, 240 identifying said regions as detected objects. Computer 105 can generate each bounding box as the minimum size containing the corresponding region (e.g., by constructing pixel coordinate pairs for the bounding box using the highest and lowest vertical pixel coordinates and the leftmost and rightmost horizontal pixel coordinates of the region).

[0044] Computer 105 is programmed to enable and disable a first part 205, a second part 210, and a possible third part 220 of machine learning model 200. When parts 205, 210, and 220 are enabled, computer 105 executes those parts as part of executing machine learning model 200. When parts 205, 210, and 220 are disabled, computer 105 executes machine learning model 200 without executing those parts. Computer 105 can execute machine learning model 200 with first part 205 enabled and second part 210 disabled (and third part 220 enabled or disabled). Computer 105 can execute machine learning model 200 with first part 205 disabled and second part 210 enabled (and third part 220 enabled or disabled). Computer 105 can execute machine learning model 200 with third part 220 enabled and first part 205 and second part 210 enabled or disabled.

[0045] Enabling the first part 205 and the second part 210 can be mutually exclusive. In other words, the computer 105 can disable the second part 210 in response to the first part 205 being enabled, and the computer 105 can disable the first part 205 in response to the second part 210 being enabled.

[0046] Computer 105 can enable and disable the first part 205 and the second part 210 based on the driving situation of vehicle 100. Computer 105 can enable the first part 205 and disable the second part 210 in response to the driving situation being a first driving situation, and enable the second part 210 and disable the first part 205 in response to the driving situation being a second driving situation. Computer 105 can determine the driving situation as described above. Computer 105 can store in memory a table that pairs possible driving situations with identifiers of the first part 205 or the second part 210 of the machine learning model 200. The following table is an example when the driving situation is weather conditions:

[0047] Driving Situation Enable part Disabled parts rain Part 1, page 205 Part Two 210, Part Three 220 Snowy day Part Two 210 Part 1, page 205; Part 3, page 220 sunny Part Three 220 Part 1, page 205; Part 2, page 210

[0048] In the example table, the first driving scenario is rainy weather, the second is snowy weather, and the third is sunny weather. Parts 205, 210, and 220 can be selected based on training for a specific driving scenario, for example. In the example table, the first part 205 may include a first head 230 for detecting lane lines trained in rainy weather, the second part 210 may include a second head 235 for detecting lane lines trained in snowy weather, and the third part 220 may include a third head 240 for detecting lane lines in sunny weather. Alternatively or additionally, parts 205, 210, and 220 can be selected based on training on a solution applicable to a specific driving scenario, which may differ in different operating modes.

[0049] The machine learning model 200 can be trained on a separate training dataset for each of the 205, 210, 220 that can be enabled. For example, the training data may include a first training dataset of images captured during a first driving scenario, a second training dataset of images captured during a second driving scenario, and so on. During training runs, the machine learning model 200 can execute the following: when the first training dataset is received, enable the first part 205 and disable the second part 210; when the second training dataset is received, enable the second part 210 and disable the first part 205, and so on. The training data can be annotated with ground-based detections, and a loss function can be computed that accumulates error when detecting ground-based detections collected across the training datasets. The number of training datasets can be chosen to be balanced relative to each other. The machine learning model 200 can be trained using any suitable training method, for example, supervised reinforcement learning using backpropagation from the loss function.

[0050] Computer 105 is programmed to execute machine learning model 200. As part of executing machine learning model 200, computer 105 accordingly executes enabled portions 205, 210, 220 and does not execute disabled portions 205, 210, 220. Machine learning model 200 may generate outputs from enabled portions 205, 210, 220 but not from disabled portions 205, 210, 220 (e.g., detections 250 from each header 230, 235, 240 in enabled portions 205, 210, 220 but not from headers 230, 235, 240 in disabled portions 205, 210, 220). For example, machine learning model 200 may output detections 250 of objects from each header 230, 235, 240 in one of enabled portions 205, 210, 220.

[0051] Computer 105 is programmed to actuate components of vehicle 100 based on the output of machine learning model 200. In the context of this disclosure, "actuation" is defined as setting an object to motion via mechanical or electromechanical stimulation. The component may include propulsion system 120, braking system 125, and / or steering system 130. For example, computer 105 may actuate the component when performing ADAS. Computer 105 may actuate the component based on detections 250 (e.g., identifiers and bounding boxes of detected objects) returned by heads 230, 235, 240 in the enable portions 205, 210, 220 of machine learning model 200. For example, computer 105 may actuate braking system 125 to stop vehicle 100 before reaching one of the detected objects. As another example, computer 105 may actuate steering system 130 to steer vehicle 100 within detected lane lines. In another example, computer 105 can autonomously operate vehicle 100; in other words, it can actuate propulsion system 120, braking system 125, and steering system 130 based on detected objects. Computer 105 can execute path planning algorithms to navigate vehicle 100 around detected objects and within detected lane lines.

[0052] Figure 3 This is a flowchart illustrating an example process 300 for executing a machine learning model 200 based on a driving context of vehicle 100. The memory of computer 105 stores executable instructions for executing the steps of process 300, and / or can be programmed using structures such as those mentioned above. As a general overview of process 300, computer 105 receives sensor data 225 and input from an operator, determines the driving context of vehicle 100, executes machine learning model 200 based on the driving context, receives the output of machine learning model 200, and actuates components of vehicle 100 based on the output of machine learning model 200. Process 300 continues as long as vehicle 100 remains running.

[0053] Process 300 begins at box 305, where computer 105 receives sensor data 225 from sensor 115 and may receive input from an operator who sets the operating mode of vehicle 100, as described above.

[0054] Next, in box 310, computer 105 determines the driving situation based on the data from box 305, as described above.

[0055] Next, in box 315, computer 105 enables and disables portions 205, 210, and 220 of machine learning model 200 based on the driving scenario from box 310, and executes machine learning model 200 as described above. Computer 105 executes machine learning model 200 when the first portion 205 is enabled and the second portion 210 is disabled in response to a first driving scenario; and executes machine learning model 200 when the first portion 205 is disabled and the second portion 210 is enabled in response to a second driving scenario.

[0056] Next, in box 320, computer 105 receives the output generated by machine learning model 200 in box 315 (e.g., detection 250 from enabled heads 230, 235, 240), as described above.

[0057] Next, in box 325, computer 105 actuates the components of vehicle 100 based on the output of machine learning model 200 from box 320, as described above.

[0058] Next, in decision box 330, computer 105 determines whether vehicle 100 is still started (i.e., in the starting state). For the purposes of this disclosure, "starting state" is defined as a state in which all electrical power is supplied to the electrical components of vehicle 100 and vehicle 100 is ready to be driven (e.g., the engine is running); "off state" is defined as a state in which a small amount of electrical power is supplied to selected electrical components of vehicle 100 (typically used when vehicle 100 is stored); and "accessory energized state" is defined as a state in which all electrical power is supplied to more electrical components compared to the off state and vehicle 100 is not ready to be driven. Typically, the operator puts vehicle 100 in the starting state when the operator is about to operate vehicle 100, puts vehicle 100 in the off state when the operator is about to leave vehicle 100, and puts vehicle 100 in the accessory energized state when the operator is about to sit in vehicle 100 but not operate vehicle 100. In response to vehicle 100 being in the started state, process 300 returns to block 305 to continue actuating vehicle 100 based on sensor data 225. In response to vehicle 100 being in the off state or accessories being powered on, process 300 ends.

[0059] Generally, the described computing system and / or device may employ any of a variety of computer operating systems, including but not limited to the following versions and / or types: Ford Applications; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (e.g., released by Oracle Corporation of Redwood Coast, California). Operating systems: AIX UNIX (published by International Business Machines, Inc., Armonk, New York); Linux; Mac OSX and iOS (published by Apple Inc., Inc., Cupertino, California); BlackBerry OS (published by BlackBerry Ltd., Inc., Waterloo, Canada); Android (developed by Google and the Open Handset Alliance); or provided by QNX Software Systems. CAR infotainment platform. Examples of computing devices include, but are not limited to, onboard computers, computer workstations, servers, desktop computers, laptops, mobile computers or handheld computers, or other computing systems and / or devices.

[0060] Computing devices typically include computer-executable instructions, which can be executed by one or more computing devices such as those listed above. Computer-executable instructions can be compiled or interpreted from computer programs created using a variety of programming languages ​​and / or technologies, which, individually or in combination, include, but are not limited to, Java. TM Languages ​​such as C, C++, Matlab, Simulink, Stateflow, Visual Basic, JavaScript, Python, Perl, and HTML are used. Some of these applications can be compiled and executed on virtual machines such as the Java Virtual Machine and the Dalvik Virtual Machine. Generally, a processor (e.g., a microprocessor) receives instructions (e.g., from memory, computer-readable media, etc.) and executes those instructions to perform one or more processes, including one or more of those described herein. Such instructions and other data can be stored and transferred using a variety of computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0061] Computer-readable media (also known as processor-readable media) include any non-transitory (e.g., tangible) medium that contributes to providing data (e.g., instructions) that can be read by a computer (e.g., by the computer's processor). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals that constitute a system bus coupled to the computer's processor. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or magnetic tape, or any other medium from which a computer can read.

[0062] The databases, data repositories, or other data stores described herein can include various mechanisms for storing, accessing / retrieving a variety of data, including hierarchical databases, file sets in file systems, application databases in proprietary formats, relational database management systems (RDBMS), NoSQL databases, graph databases (GDB), and so on. Each such data store is typically contained within a computing device employing a computer operating system such as those mentioned above, and is accessed via a network in any one or more of various ways. File systems can be accessed from the computer operating system and can include files stored in various formats. In addition to languages ​​used to create, store, edit, and execute the stored programs (such as PL / SQL as described above), RDBMS typically employs Structured Query Language (SQL).

[0063] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) and stored on computer-readable media (e.g., disks, storage, etc.) associated therewith. Computer program products may include such instructions stored on computer-readable media for performing the functions described herein.

[0064] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, inspirations, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a certain ordered order, such processes can be practiced by performing the steps in a different order than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with the applicable owner / user manual and / or safety guidelines.

[0065] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to describe the nature of the words, not to be restrictive. The adjectives “first,” “second,” and “third” are used throughout this document as identifiers and are not intended to indicate importance, order, or quantity. The use of “in response to,” “after determining,” etc., indicates a causal relationship, not just a temporal one. In light of the foregoing teachings, many modifications and variations of this disclosure are possible, and this disclosure may be practiced in ways other than those specifically described.

[0066] According to the present invention, a computer is provided having a processor and a memory, wherein the memory stores instructions executable by the processor to: execute a machine learning model on the vehicle with a first part of the machine learning model enabled and a second part of the machine learning model disabled, in response to a first driving situation of the vehicle; and execute the machine learning model with the first part disabled and the second part enabled, in response to a second driving situation.

[0067] According to one embodiment, the instructions may further include instructions for actuating vehicle components based on the output of a machine learning model.

[0068] According to one embodiment, the output of the machine learning model includes the detection of objects in the environment surrounding the vehicle.

[0069] According to one embodiment, the first portion includes at least one first head; and the second portion includes at least one second head.

[0070] According to one embodiment, the machine learning model includes a common part; and the at least one first head and the at least one second head are arranged in the machine learning model to receive input from the common part.

[0071] According to one embodiment, the common part is trained to perform feature extraction on sensor data.

[0072] According to one embodiment, the at least one first head and the at least one second head are trained to perform object detection based on features extracted from the feature extraction.

[0073] According to one embodiment, the first part is trained to perform object detection, and the second part is trained to perform object detection.

[0074] According to one embodiment, the machine learning model is a deep neural network.

[0075] According to one embodiment, the driving scenario is the operating mode of the vehicle.

[0076] According to one embodiment, the operating mode indicates whether the components of the vehicle are controlled by the computer or by the vehicle operator.

[0077] According to one embodiment, the driving scenario refers to the environmental conditions experienced by the vehicle.

[0078] According to one embodiment, the first driving scenario is daytime, and the second driving scenario is nighttime.

[0079] According to one embodiment, the driving scenario is weather conditions.

[0080] According to one embodiment, the driving scenario is the vehicle's position.

[0081] According to the present invention, a method includes: executing a machine learning model on the vehicle while enabling a first part of the machine learning model and disabling a second part of the machine learning model in response to a driving situation of the vehicle being a first driving situation; and executing a machine learning model while disabling the first part and enabling the second part in response to a driving situation being a second driving situation.

[0082] According to one embodiment, the invention is further characterized in that the components of the vehicle are actuated based on the output of the machine learning model.

[0083] According to one embodiment, the first portion includes at least one first head; the second portion includes at least one second head; the machine learning model includes a common part; and the at least one first head and the at least one second head are arranged in the machine learning model to receive input from the common part.

[0084] According to one embodiment, the common part is trained to perform feature extraction on sensor data; and the at least one first head and the at least one second head are trained to perform object detection based on features from the feature extraction.

[0085] According to one embodiment, the driving situation is either the vehicle's operating mode or the environmental conditions experienced by the vehicle.

Claims

1. A method comprising: In response to the driving situation of the vehicle, which is a first driving situation, the machine learning model is executed on the vehicle with the first part of the machine learning model enabled and the second part of the machine learning model disabled. as well as In response to the driving scenario being a second driving scenario, the machine learning model is executed with the first part disabled and the second part enabled.

2. The method of claim 1, further comprising a component of the vehicle actuated based on the output of the machine learning model.

3. The method of claim 2, wherein the output of the machine learning model includes the detection of objects in the environment surrounding the vehicle.

4. The method of claim 1, wherein: The first part includes at least one first head; and The second part includes at least one second head.

5. The method of claim 4, wherein: The machine learning model includes a common part; and The at least one first head and the at least one second head are arranged in the machine learning model to receive input from the common part.

6. The method of claim 5, wherein the common part is trained to perform feature extraction on sensor data.

7. The method of claim 6, wherein the at least one first head and the at least one second head are trained to perform object detection based on features from the feature extraction.

8. The method of claim 1, wherein the first portion is trained to perform object detection, and the second portion is trained to perform object detection.

9. The method of claim 1, wherein the driving scenario is the operating mode of the vehicle.

10. The method of claim 9, wherein the operating mode indicates whether the components of the vehicle are controlled by the computer or by the operator of the vehicle.

11. The method of claim 1, wherein the driving situation is the environmental conditions experienced by the vehicle.

12. The method of claim 11, wherein the first driving scenario is daytime and the second driving scenario is nighttime.

13. The method of claim 1, wherein the environmental conditions are weather conditions.

14. The method of claim 1, wherein the driving situation is the position of the vehicle.

15. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method as claimed in any one of claims 1 to 14.

Images