Using neural networks to perform edge processing on sensor data can reduce data traffic on communication networks.

Abstract

Methods, systems, and apparatus related to edge processing of sensor data using neural networks to reduce network traffic to and / or from servers. In one method, a cloud server uses an artificial neural network (ANN) to process sensor data from a vehicle. The ANN has several layers. Based on analyzing at least one characteristic of the sensor data received from the vehicle and / or the context associated with processing the sensor data, the cloud server determines to send one or more of the layers of the ANN for edge processing on the vehicle itself. In other cases, the cloud server decides to send the one or more layers to an edge server device located on the communication path between the vehicle and the cloud server. The edge processing reduces network data traffic.

Description

[0001] Related applications

[0002] This application claims priority to U.S. Patent Application No. 16 / 942,317, filed July 29, 2020, entitled “Edge Processing of Sensor Data Using a Neural Network to Reduce Data Traffic on a Communication Network,” the entire disclosure of which is hereby incorporated by reference. Technical Field

[0003] At least some of the embodiments disclosed herein generally relate to neural network processing devices, and more specifically, but not limited to, using neural networks to perform edge processing on sensor data to reduce data traffic. Background Technology

[0004] Recent developments in autonomous driving technology allow computing systems to operate the control elements of motor vehicles, at least in certain situations, without the assistance of a human operator.

[0005] For example, sensors (e.g., cameras and radar) may be mounted on a motor vehicle to detect the conditions of the environment surrounding the vehicle as it travels in a lane. With or without any input from the vehicle's operator, a computing system mounted on the vehicle analyzes the sensor input to identify the conditions and generates control signals or commands for autonomous adjustments to the vehicle's direction and / or speed.

[0006] In some configurations, when the computing system identifies a situation where it may be unable to continue operating the vehicle safely, it alerts the human operator and requests the human operator to take over control of the vehicle and drive it manually, rather than allowing the computing system to drive the vehicle autonomously.

[0007] Autonomous driving and / or advanced driver assistance systems (ADAS) can use artificial neural networks (ANNs) to identify events and / or objects captured in sensor inputs. Examples of sensor inputs include images from digital cameras, LiDAR, radar, ultrasonic sonar, etc.

[0008] Generally, artificial neural networks (ANNs) use a network of neurons to process inputs to the network and generate outputs from the network. For example, each neuron in the network receives a set of inputs. Some inputs to a neuron may be the outputs of some neurons in the network; and some inputs to a neuron may be inputs provided to the neural network. The input / output relationships among the neurons in the network represent the neuronal connectivity in the network.

[0009] For example, each neuron can have a bias, activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network can have different activation functions. The relationship between the input and output of an ANN is generally defined by the ANN model, which contains data representing the connectivity of neurons in the network, as well as the bias, activation function, and synaptic weights of each neuron.

[0010] For example, input to an ANN network can be generated based on camera input; and the output from an ANN network can be an identifier for an item such as an event or object. Attached Figure Description

[0011] Embodiments are illustrated by way of example rather than limitation in the accompanying drawings, in which similar reference numerals indicate similar elements.

[0012] Figure 1 This describes a data processing system that uses edge processing to reduce data traffic on a network, according to one embodiment.

[0013] Figure 2 The description describes a computing device, according to one embodiment, that performs processing on a portion of an ANN on a vehicle to reduce data traffic.

[0014] Figure 3 The present invention describes a computing device, according to one embodiment, that is separate from the vehicle and used to process data from the vehicle to reduce data traffic.

[0015] Figure 4 A method for processing sensor data using a vehicle's computing device is demonstrated according to one embodiment.

[0016] Figure 5 This invention illustrates a method, according to one embodiment, for a computing device to process sensor data using a portion of an ANN received from another computing device.

[0017] Figure 6 A method for processing sensor data from sensors of a vehicle is shown according to one embodiment. Detailed Implementation

[0018] The following disclosure describes various embodiments for edge processing (e.g., processing at a vehicle or edge server) of sensor data using a portion of a neural network to reduce data traffic to remote computing devices (e.g., cloud servers). At least some embodiments herein relate to processing sensor data using a computing device (e.g., a neural network processor) located in a vehicle. The processing is performed using a portion of an artificial neural network (ANN) received from a remote computing device, which performs further processing using other portions of the ANN. In one example, the vehicle is an autonomous vehicle (e.g., a car, boat, aircraft, or drone).

[0019] In previous methods, mobile devices (e.g., autonomous vehicles) may store and / or generate sensor data that needs to be processed in the cloud or other servers (e.g., remotely located from the vehicle). For example, an autonomous vehicle may have an ANN configured to identify objects captured in camera images and / or classify said objects. Such images may be uploaded to the cloud or other servers for processing, training, etc. Processing in the cloud or on the server is typically performed via multiple layers of artificial neurons. After an image has been processed by one or more layers, the output of those layers may be much smaller than the output of the image itself.

[0020] However, sending image data for layer processing by an ANN in the cloud or on a server requires sending large data files or image data streams (e.g., image streams from a vehicle's camera). This, for example, increases data traffic on the network (e.g., cellular or other wireless networks) between the vehicle collecting the data and the remote cloud server. This significantly degrades network performance because a large amount of bandwidth is required to transmit image or other sensor data. It can also cause mobile devices to consume excessive battery power, which may be limiting for some smaller mobile devices.

[0021] Various embodiments of this disclosure provide technical solutions to one or more of the above-described technical problems. In one embodiment, a cloud server uses an ANN to process sensor data from a vehicle. The ANN has several layers. In some cases, based on analysis of the nature of the sensor data received from the vehicle (e.g., the type or characteristics of the data) and / or other factors associated with the context of processing the sensor data (e.g., bandwidth consumption attributable to data traffic to or from the cloud server, and / or the environment in which the vehicle is operating (e.g., daytime, nighttime, and / or fog or rain)), the cloud server decides to send one or more layers of the ANN for edge processing at the vehicle itself. In other cases, the cloud server decides to send one or more layers to an edge server device located between the vehicle and the cloud server. By pushing processing to one or more layers closer to the vehicle (or other data sources to be processed by the ANN), data traffic on one or more networks of the networked system can be reduced.

[0022] In one embodiment, an ANN comprises a network of neurons stored in memory on a computing device. A layer of neurons generates an output that serves as the input to the next layer of neurons. Each neuron takes one or more inputs to generate an output. In one instance, a layer of neurons can be considered as one ANN, and another layer of neurons can be considered as another ANN. The connection layers then form a combined ANN. The computation of the neuron layers can be tuned to include input vectors that multiply the matrix to generate the output vector. In some cases, this matrix operation can be decomposed into smaller matrix operations based on the hardware acceleration capabilities of one or more ANN accelerators.

[0023] In one embodiment, given the above, the computing device is configured to intelligently partition the computation of an ANN with multiple layers across multiple devices (e.g., cloud, edge server, vehicle, mobile device, etc.). For example, a vehicle causes sensors to generate a first input vector. The first input vector is multiplied by the matrix of the first layer of neurons to generate a first output vector of the second layer of neurons. Initially, all layers reside in the cloud / server. If the cloud / server determines that it is advantageous for the edge server (rather than the cloud / server) to process the first layer, the model of the first layer is pushed to the edge server, such that the cloud / server receives the first output vector from the edge server instead of the first input vector from the vehicle or the edge server.

[0024] In one embodiment, the cloud / server monitors its ANN to determine which of one or more layers can be pushed to one or more edge servers configured on the communication path from the mobile device to the cloud / server. When the processing performed by the push layer via the ANN is sufficient, the original image does not need to be uploaded to the cloud / server. Otherwise, the original image can be further uploaded from the edge server to the cloud / server for further processing, as may be necessary in some cases.

[0025] In one embodiment, the ANN at the cloud / server can estimate the probability that the edge server will upload an image after processing with one or more ANN layers specified by the cloud / server, in the case that the ANN layer is pushed to the vehicle or edge server for processing at a data source closer to the image in the network. In one embodiment, the cloud / server pushes processing to the edge server based on a balance between the cost and benefits of processing at the edge server and the data traffic required to push the layer to the edge server, as well as the data traffic required to upload the processed result and the original image to the edge server when the processing result is insufficient.

[0026] In one embodiment, one or more parts of the ANN are selected by the server and pushed to the vehicle. In one instance, each of the parts is a layer of the ANN. In another instance, each of the parts is a portion of the data for the neuron model of the ANN.

[0027] In one embodiment, the networking system (e.g., a network providing wireless and / or wired communication paths between a vehicle and a cellular base station and / or server) includes at least one processing device (e.g., the CPU of a cloud server) and a memory containing instructions configured to instruct the at least one processing device to perform a method. The method includes: receiving data via a network from a computing device (e.g., a controller of a memory device located in the vehicle), wherein the received data is processed using an artificial neural network (ANN), the ANN including a neuron model for processing the data, and the computing device providing the data from processing sensor data obtained from at least one sensing device of the vehicle (e.g., a camera of the vehicle); sending a first portion of the neuron model to the computing device, wherein the first portion is selected based on predicted data traffic associated with processing the sensor data, the predicted data traffic including at least one of data traffic to or from the computing device; receiving intermediate results from the computing device, wherein the computing device provides intermediate results based on processing the sensor data using the first portion (e.g., output from processing the sensor data using a first layer of the ANN); and providing an output by processing the intermediate results using a second portion of the neuron model (e.g., the intermediate results are input to a second layer of the ANN).

[0028] In one instance, the output is used to control a vehicle system. In one instance, a cloud server sends the output to the vehicle to induce control actions on the vehicle. In one instance, the controlled system may be an electronically controllable engine, steering, braking, or other component of the vehicle.

[0029] In one embodiment, a set of sensors is configured on a vehicle, and an edge server may be located on the vehicle or in a base station (e.g., in a 5G base station or connected to the base station via a local area network), and the edge server communicates with a cloud server. Sensor data goes to the edge server and then to the cloud server. The edge server and cloud server are network-separated and work together to process the sensor data using an ANN. The output from the ANN at the cloud server may or may not be returned to the vehicle that provided the input sensor data.

[0030] In one instance, a layer is an assembly of artificial neurons, and intermediate results are the outputs of the artificial neurons in the layer that are downloaded to the edge server. An artificial neuron is a dataset specifying the behavior and / or state of a neuron. In one instance, layers of an ANN that can be pushed to the edge server are identified (e.g., discovering an improved or optimal way to split the ANN into two parts, such that the part as the first layer can be pushed to the edge server for improved or optimal results).

[0031] Figure 1 This describes a data processing system according to one embodiment that uses edge processing of data (e.g., sensor or other data) to reduce data traffic over a network (e.g., data traffic between a mobile device in a vehicle and the cloud using a wireless network, where data traffic is associated with processing sensor data collected by the mobile device). Vehicle 102 includes one or more sensing devices 104. A computing device 110 within vehicle 102 processes sensor data 106 from sensing devices 104 and outputs intermediate results 130. Intermediate results 130 are sent to computing device 138 via network 136. Computing device 138 typically uses an artificial neural network (ANN) 150 to process intermediate results 130 to produce an output 162. Output 162 can be used to control the movement 166 of vehicle 102. To reduce data traffic 134, computing device 138 may divide the ANN 150 into multiple parts (e.g., part 154, part 156, part 158, etc.) and send portions of the ANN 150 to other devices (e.g., computing device 110) to process the sensor data 106. In one instance, each part is a portion of the neuron model of ANN 150.

[0032] Vehicle 102 may be, for example, an autonomous vehicle. Sensing device 104 may be, for example, an image sensor, LiDAR, radar, GPS, or other types of sensors. Sensing device 104 outputs the collected sensor data 106 (e.g., images, GPS location, etc.) to computing device 110.

[0033] The computing device 110 may be, for example, a computer controlled by a processing device, an embedded computer, a controller in a vehicle, a server, a laptop computer, a tablet computer, or other electronic devices. In some embodiments, the computing device 110 may be described as an edge processing device.

[0034] In one embodiment, computing device 110 includes processing device 116 and memory 118. Processing device 116 may be, for example, a microprocessor, graphics processor, embedded processor, embedded controller, central processing unit (CPU), system-on-a-chip (SoC), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other processing device. Processing device 116 may consist of a single processor having a single processing core, a single processor having multiple processing cores, or multiple processors. Processing device 116 and memory 118 may be configured on a printed circuit board. In some cases, processing device 116 and memory 118 are packaged together in a system-on-a-chip (SoC).

[0035] The memory 118 may include volatile memory (e.g., DRAM and / or SRAM) and / or non-volatile memory. The memory 118 may be separate from the processing device 116 (e.g., MTF module, SIMM module, NVMe module, etc.) or may be embedded within the processing device 116 (e.g., high-bandwidth memory (HBM)).

[0036] Examples of non-volatile memories include flash memory, memory cells formed based on NAND logic gates, NOR logic gates, phase-change memory (PCM), magnetic RAM (MRAM), resistive random access memory (RRAM), crosspoint memory, and other memory devices. Crosspoint memory devices can use transistorless memory elements, each having memory cells and selectors stacked together in a column. The column of memory elements is connected via two wiring arrangements laid out in a vertical direction, wherein one wiring arrangement in one direction of a layer is above the column of memory elements, and the other wiring arrangement is in the opposite direction and in a layer below the column of memory elements. Each memory element can be individually selected at the intersection of a wiring arrangement on each of the two layers. Crosspoint memory devices are fast and non-volatile and can be used as a unified memory pool for processing and storage. Other examples of non-volatile memory include read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and electronically erasable programmable read-only memory (EEPROM).

[0037] The memory 118 contains instructions 120 that instruct the processing device 116 to perform tasks, such as receiving a portion 154 of ANN 150 from the computing device 138, processing sensor data 106 using ANN 122, and sending intermediate results 130 to the computing device 138 after processing the sensor data 106.

[0038] Processing device 116 sends intermediate results 130 to computing device 138 via network 136. Network 136 may include a wireless network or a combination of wireless and wired networks. Network 136 may also include networking services such as cellular telephone, satellite, or Internet services, and networking devices such as satellites, cellular telephone towers, cellular telephone base stations, network switches, and network routers. The volume of data transmitted between computing device 138 and computing device 110 (e.g., megabytes per second or gigabytes per second) is... Figure 1 This is represented as data flow 134. Data flow 134 may include, for example, intermediate results 130 sent to computing device 138, sensor data 106 sent to computing device 138 for further processing, a portion 154 of ANN 150 sent to computing device 110, a portion 158 of ANN 150 sent to sensing device 104, etc.

[0039] The computing device 138 receives input data 140 (e.g., intermediate result 130) from the network 136. The computing device 138 may be, for example, a computer, embedded computer, server, laptop computer, tablet computer, or other electronic device controlled by a processing device.

[0040] In one embodiment, computing device 138 includes processing device 144 and memory 146. Processing device 144 may be, for example, a microprocessor, graphics processor, embedded processor, embedded controller, central processing unit (CPU), system-on-a-chip (SoC), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other processing device. Processing device 144 and memory 146 may be configured on a printed circuit board. In some cases, processing device 144 and memory 146 are packaged together in a system-on-a-chip (SoC).

[0041] Memory 146 may include volatile memory (e.g., DRAM and / or SRAM) and / or non-volatile memory (e.g., NAND flash, NOR flash, PCM, MRAM, RRAM, cross-point memory, ROM, PROM, EPROM, EEPROM, etc.). Memory 146 may be separate from processing device 144 (e.g., DIMM module, SIMM module, NVMe module, etc.) or may be embedded within processing device 144 (e.g., HBM memory).

[0042] The memory 146 contains instructions 148 that instruct the processing device 144 to perform tasks, such as receiving a request for a portion 154 of the ANN 150 from the computing device 110, sending a portion 154 of the ANN 150 to the computing device 110, processing input data 140 (e.g., intermediate results 130) using at least a portion 156 of the ANN 150, and providing output 162 based on processing the intermediate results 130 from the computing device 110.

[0043] The processing device 144 includes an ANN 150 (e.g., stored in memory). The ANN 150 uses a network of neurons to process inputs to the network and generate outputs from the network.

[0044] For example, each neuron in a network receives a set of inputs. Some inputs to a neuron may be external inputs provided to the neural network, and some inputs to a neuron may be the outputs of some neurons in the network. The input / output relationships among the neurons in the network represent the neuronal connectivity in the network.

[0045] For example, each neuron can have a bias, an activation function, and a set of synaptic weights for its input. The activation function can be a step function, a linear function, a log-sigmoid function, etc. Different neurons in the network can have different activation functions.

[0046] For example, each neuron can generate a weighted sum of its input and its bias, and then produce an output as a function of the weighted sum, which is computed using the neuron's activation function. The input to an ANN can be described as an input vector, and the output can be described as an output vector.

[0047] The relationship between the inputs and outputs of an ANN is generally defined by a neuron model, which contains data representing the connectivity of neurons in the ANN, as well as the biases, activation functions, and synaptic weights of each neuron. Using a given neuron model 152, a computing device 138 computes the output 162 of the ANN 150 from a given set of input data 140 leading to the ANN 150.

[0048] For example, input data 140 to ANN 150 may be generated based on camera output from sensing device 104. Output 162 from ANN 150 may be an identifier for an item such as an event or object.

[0049] In one embodiment, ANN 150 is a spiking neural network (SNN). An SNN is a type of ANN that closely mimics a natural neural network. When the activation level of a neuron is sufficiently high, the SNN neuron produces a spike as its output. The activation level of an SNN neuron mimics the membrane potential of a natural neuron. The output / spiking of an SNN neuron can alter the activation level of other neurons that receive the output. The current activation level of an SNN neuron, which varies over time, is typically modeled using differential equations and is considered the state of the SNN neuron. Incoming spikes from other neurons can push the activation level of those neurons higher, reaching a threshold for producing a spike. Once a neuron produces a spike, its activation level is reset. Before producing a spike, the activation level of an SNN neuron can decay over time, as controlled by differential equations. The temporal element in the behavior of SNN neurons makes SNNs suitable for processing spatiotemporal data. The connectivity of SNNs is typically sparse, which helps reduce computational workload.

[0050] Generally, supervised methods can be used to train ANNs, where the parameters in the ANN are tuned to minimize or reduce the error between the known output produced by the corresponding input and the computed output generated by applying the input to the ANN. Examples of supervised learning / training methods include reinforcement learning and learning with error correction.

[0051] Alternatively or in combination, unsupervised methods can be used to train ANNs, where the exact output produced by a given set of inputs is unknown until training is complete. ANNs can be trained to classify items into multiple categories or data points into clusters. Multiple training algorithms can be used for complex machine learning / training paradigms.

[0052] Computing device 110 takes input data 112 (e.g., sensor data 106) and generates intermediate results 130. In one embodiment, computing device 110 passes input data 112 as intermediate results 130 without processing input data 112 using an ANN. Computing device 138 takes intermediate results 130 as input data 140. ANN 150 uses input data 140 as input to neuron model 152 and computes output 162.

[0053] The computing device 138 may determine that the data flow 134 between the computing device 110 and the computing device 138 can be reduced by processing a portion of the sensor data 106 within the computing device 110. The computing device 138 may select a set of neurons (e.g., a portion 154 of ANN 150) and send the set of neurons to the computing device 110 to form ANN 122 within the computing device 110.

[0054] In one instance, ANN 122 is a subset of ANN 150. In one instance, ANN 122 contains neuron model data that is not present in ANN 150. In one instance, ANN 122 is maintained by computing device 138 to contain some or all of ANN 150 (e.g., the included portions may be pre-defined or determined in real time based on the context of vehicle 102).

[0055] The computing device 138 may select the set of neurons to be sent to the computing device 110 based on factors such as: comparing the predicted data size 114 of the input data 112 to the computing device 110 with the predicted data size 132 of the intermediate result 130 output from the computing device 110; or comparing the predicted data size 114 of the input data 112 to be processed by the portion 154 of the ANN 150 with the data size 160 of the portion 154 of the ANN 150; or predicting whether sensor data 106 will need to be uploaded to the computing device 138 for further processing after the intermediate result 130 is provided.

[0056] For example, computing device 110 may receive sensor data 106 (e.g., image data from an image sensor). Computing device 138 may predict the data size 114 of the sensor data 106 input to computing device 110 (e.g., 10 MB input vector per second). Computing device 138 may predict that if some processing performed by ANN 150 is performed by computing device 110 (e.g., detecting objects in an image), the data size 132 of the intermediate result 130 output from computing device 110 will be small (e.g., 1 MB output vector per second). Computing device 138 may select a portion 154 of neuron model 152 (e.g., neurons associated with detecting objects in an image) and send the portion 154 of neuron model 152 to computing device 110 to be implemented as neuron model 124 within ANN 122. ANN 122 takes sensor data 106 as an input vector, processes the data, and outputs the intermediate result 130 as an output vector.

[0057] In one instance, sensor data 106 from sensing device 104 (e.g., image data from an image sensor) may need to be sent to computing device 138 for further processing after intermediate results 130 have been provided. This could be due to, for example, glare that could lead to uncertain intermediate results 130. Computing device 138 may include predicting whether sensor data 106 will need to be uploaded to computing device 138 for further processing as part of determining whether to perform processing on portion 154 of ANN 150 on computing device 110.

[0058] In one example, computing device 138 may predict a data size 108 from sensing device 104. Computing device 138 may determine that a portion 154 of ANN 150 used for processing sensor data 106 is smaller than the predicted data size 108 of the sensor data 106 from sensing device 104. For example, computing device 138 may predict a data size 108 of 1TB from sensing device 104. Computing device 138 may determine, for example, that a portion 154 of ANN 150 has a data size 160 of 50MB. Computing device 138 may select a portion 154 of ANN 150 to send to computing device 110 based on the fact that the data size 160 of a portion 154 of ANN 150 is smaller than the predicted data size 108 of the sensor data 106 from sensing device 104.

[0059] In one embodiment, processing device 116 predicts data traffic 134 generated between computing device 110 and computing device 138 due to processing sensor data 106. Based on the predicted data traffic 134, processing device 116 may request a portion 154 of ANN 150 from computing device 138. Computing device 138 sends portion 154 of ANN 150 to processing device 116. Processing device 116 uses portion 154 of neuron model 152 from computing device 138 to create neuron model 124 within ANN 122. Processing device 116 uses ANN 122 to process sensor data 106 and outputs intermediate results 130.

[0060] For example, sensor data 106 could be image data from a camera. Processor 116 can predict that sending the image data as an intermediate result 130 to computing device 138 will generate a large amount of data traffic 134 (e.g., 15 MB per second data size 132). Processing device 116 can request a portion 154 of ANN 150 and implement the received portion 154 in ANN 122. Processor 116 uses ANN 122 to process the image data (e.g., identify or classify objects or events). After processing the image data, the data size 132 of the intermediate result 130 can be smaller (e.g., 2 MB per second), thus generating less data traffic 134.

[0061] The computing device 138 takes input data 140 (e.g., intermediate result 130) and processes the data using portion 156 of ANN 150 to generate output 162. Output 162 can be sent back to vehicle 102. Vehicle 102 can use output 162 to control vehicle 102's actions 166. For example, intermediate result 130 can identify objects or events (e.g., pedestrians entering the vehicle's path). Output 162 can instruct vehicle 102's actions 166 (e.g., stopping or changing lanes). Vehicle 102 can control vehicle 102's actions 166 based on output 162 (e.g., applying brakes or steering the vehicle).

[0062] In various embodiments, edge processing of sensor data as described above can be implemented in a networked system that allows communication between a vehicle and a cloud server (or other computing device). The networked system includes at least one vehicle and at least one computing device (e.g., vehicle 102 and computing devices 110, 138). The networked system is configured to implement edge processing using a portion of an ANN selected and sent from the computing device to reduce data traffic within the networked system.

[0063] The networked system is connected via one or more communication networks. The communication network (e.g., network 136) may include at least a local device network such as Bluetooth, a wide area network (WAN), a local area network (LAN), an intranet, a mobile wireless network such as 4G or 5G, an extranet, the Internet, and / or any combination thereof. Nodes in the networked system (e.g., vehicles and / or other mobile / computing devices) may each be part of a peer-to-peer network, a client-server network, a cloud computing environment, etc. Furthermore, any of the devices, computing devices, vehicles, sensors or cameras, and / or user interfaces in the networked system may contain some type of computing system. The computing system may include network interfaces to other devices in the LAN, intranet, extranet, and / or the Internet. The computing system may also operate as a server or client machine in a client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server or client machine in a cloud computing infrastructure or environment.

[0064] Figure 2 The description describes an embodiment where computing device 138 determines processing of a portion 158 of ANN 150 on vehicle 102 (e.g., in sensing device 204). Computing device 138 sends a portion 158 of ANN 150 to sensing device 204. Sensing device 204 uses a portion 158 of the neuron model 152 from computing device 138 to create a neuron model 270 within ANN 268. Sensing device 204 uses ANN 268 to process data collected from sensing device 204 and outputs the processed data as sensor data 206. Computing device 110 takes the sensor data 206 as input data 112 and further processes the data using ANN 122 to output an intermediate result 130. In this configuration, sensing device 204 can be described as an edge processing device.

[0065] For example, sensing device 204 may be an image sensor. Computing device 138 may determine that it is advantageous to process data from the image sensor within the image sensor itself (e.g., by performing object identification). Computing device 138 sends a portion 158 of ANN 150 to sensing device 204. Sensing device 204 uses a portion 158 of neuron model 152 to create neuron model 270 within ANN 268. Sensing device 204 uses ANN 268 to process the data collected from the image sensor and output sensor data 206 (e.g., objects identified as pedestrians).

[0066] Processing device 116 takes input data 112 (e.g., sensor data 206), further processes the input data 112 using ANN 122, and outputs intermediate result 130. For example, intermediate result 130 may be event detection (e.g., the trajectory of a pedestrian identified in sensor data 206 intersects with the trajectory of vehicle 102). Performing processing of portion 158 of ANN 150 within sensing device 204 reduces the data size 208 of sensor data 206 from sensing device 204 and reduces the data traffic 134 to computing device 138 because sensing device 204 will not need to send all image data out for processing.

[0067] Figure 3 This describes an embodiment where the computing device 310 is separated from the vehicle 302. In one embodiment, both the computing device 310 (e.g., a 5G base station) and the computing device 138 (e.g., a cloud server) are servers. The computing device 310 is on the communication path between the vehicle 302 and the computing device 138. The vehicle 302 outputs sensor data 206 to the network 336, and the network 336 routes the sensor data 206 to the computing device 310. This is similar to... Figure 2 As described, computing device 310 processes sensor data 206. Computing device 310 can be described as an edge processing device. In one embodiment, network 336 is configured to... Figure 1 The network described is the same as network 136.

[0068] Figure 4 This illustrates a method for processing sensor data implemented in a vehicle according to one embodiment. For example, Figure 4 The method can be combined Figure 1 The network 136 and computing device 138 are implemented in Figure 1 Among the vehicles, 102.

[0069] Figure 4 The method can be executed by processing logic, which may include hardware (e.g., processing device, circuit system, special-purpose logic, programmable logic, microcode, device hardware, integrated circuit, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, Figure 4 The method is performed at least in part by one or more processing devices.

[0070] Although shown in a specific order or sequence, the order of processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in different orders, and some processes may be performed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0071] At box 401, the vehicle's computing device receives data from the vehicle's sensing devices. In one example, the computing device 110 of vehicle 102 receives sensor data 106 from the sensing device 104 of vehicle 102.

[0072] At box 403, data traffic to and / or from the computing device is predicted. In one example, computing device 138 predicts data traffic 134 to or from computing device 110.

[0073] At box 405, a portion of the neuron model from the artificial neural network (ANN) is received in the computing device based on predicted data traffic to or from the computing device (e.g., data traffic between the computing device and a cloud server). In one example, a portion 154 of the neuron model 152 from the ANN 150 is received in the computing device 110 based on predicted data traffic 134 to and / or from the computing device 110.

[0074] At box 407, a portion of the neuron model stored within the computing device is used to process data from the sensing device within the computing device. In one example, computing device 110 uses a portion of the neuron model 124 within computing device 110 to process sensor data 106 from sensing device 104.

[0075] At box 409, intermediate results are sent from the computing device to the processing device storing the ANN across the network. In one instance, computing device 110 sends intermediate results 130 to processing device 144 containing ANN 150 across network 136 (e.g., in a cloud network).

[0076] At box 411, output is provided from the processing device by processing intermediate results from the computing device using a portion of the ANN. In one example, processing device 144 uses portion 156 of ANN 150 to process intermediate results 130 from computing device 110 and provides output 162. Output 162 can be used to control the actions 166 of vehicle 102 (and / or control another vehicle).

[0077] Figure 5 This illustrates a method for processing sensor data implemented in a vehicle according to one embodiment. For example, Figure 5 The method can be combined Figure 2 The network 136 and computing device 138 are implemented in Figure 2 Among the vehicles, 102.

[0078] Figure 5The method can be executed by processing logic, which may include hardware (e.g., processing device, circuit system, special-purpose logic, programmable logic, microcode, device hardware, integrated circuit, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, Figure 5 The method is performed at least in part by one or more processing devices.

[0079] Although shown in a specific order or sequence, the order of processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in different orders, and some processes may be performed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0080] At box 501, the first computing device determines to perform processing on a portion of the artificial neural network (ANN) on the vehicle. In one example, the first computing device 138 determines to perform processing on portion 158 of ANN 150 on vehicle 102.

[0081] At box 503, the vehicle's sensing device receives a portion of the neuron model from the ANN of the first computing device. In one example, sensing device 204 receives a portion 158 of neuron model 152 from ANN 150 of the first computing device 138.

[0082] At box 505, the vehicle's second computing unit receives a portion of the neuron model from the first computing unit's ANN based on predicted data flow to and / or from the first computing unit associated with processing sensor data from the vehicle. In one example, a portion 154 of the neuron model 152 from the ANN 150 is received in the second computing unit 110 of the vehicle 102 based on predicted data flow 134 to or from the first computing unit 138 associated with processing sensor data 206 from sensing devices 204 of the vehicle 102.

[0083] At block 507, the second computing device receives data from the sensing device that has been partially processed using a neuron model stored within the sensing device. In one example, computing device 110 receives sensor data 206 that has been partially processed in sensing device 204 using a neuron model 270 within sensing device 204.

[0084] At box 509, a portion of the neuron model stored within the second computing device is used to process data from the sensing device in the second computing device. In one example, computing device 110 uses a portion of the neuron model 124 within computing device 110 to process sensor data 206 from sensing device 204.

[0085] At box 511, intermediate results are sent across the network from a second computing device to a processing device that stores the ANN in a first computing device. In one example, computing device 110 sends intermediate results 130 across network 136 to a processing device 144 containing ANN 150.

[0086] At box 513, output is provided from the processing device by processing intermediate results from the second computing device using a portion of the ANN. In one example, processing device 144 uses portion 156 of ANN 150 to process intermediate results 130 from computing device 110 and provides output 162. Output 162 can be used to control the movement 166 of vehicle 102.

[0087] Figure 6 A method for processing sensor data from sensors of a vehicle is illustrated according to one embodiment. For example, Figure 6 The method can be combined Figure 3 Network 336, computing device 310 and computing device 138 are implemented in Figure 3 Among the vehicles, 302.

[0088] Figure 6 The method can be executed by processing logic, which may include hardware (e.g., processing device, circuit system, special-purpose logic, programmable logic, microcode, device hardware, integrated circuit, etc.), software (e.g., instructions that run or execute on the processing device), or a combination thereof. In some embodiments, Figure 6 The method is performed at least in part by one or more processing devices.

[0089] Although shown in a specific order or sequence, the order of processes may be modified unless otherwise specified. Therefore, the illustrated embodiments should be understood as examples only, and the illustrated processes may be performed in different orders, and some processes may be performed in parallel. Furthermore, one or more processes may be omitted in various embodiments. Therefore, not all processes are required in every embodiment. Other process flows are possible.

[0090] At block 601, the first computing device determines to perform processing on a portion of an artificial neural network (ANN) outside the first computing device. In one example, the first computing device 138 determines to perform processing on a portion 158 of ANN 150 on vehicle 302.

[0091] At box 603, the vehicle's sensing device receives a portion of the neuron model from the ANN. In one example, sensing device 204 receives a portion 158 of the neuron model 152 from the ANN 150 of the first computing device 138.

[0092] At box 605, a second computing device detached from the vehicle receives a portion of the neuron model from the ANN of the first computing device based on predicted data flow to and / or from the first computing device associated with processing sensor data from the vehicle. In one example, a portion 154 of the neuron model 152 from the ANN 150 is received in the second computing device 310 detached from the vehicle 302 based on predicted data flow 134 to or from the first computing device 138 associated with processing sensor data 206 from sensing devices 204 of the vehicle 302.

[0093] At box 607, the second computing device receives data from the sensing device that has been partially processed using a neuron model stored within the sensing device. In one example, the second computing device 310 receives sensor data 206 that has been processed in the sensing device 204 using a portion of the neuron model 270 within the sensing device 204.

[0094] At box 609, a portion of the neuron model stored within the second computing device is used to process data from the sensing device within the second computing device. In one example, the second computing device 310 uses a portion of the neuron model 124 within the second computing device 310 to process sensor data 206 from the sensing device 204.

[0095] At box 611, intermediate results are sent across the network from the second computing device to the processing device that stores the ANN. In one example, the second computing device 310 sends intermediate results 130 across network 336 to the processing device 144 containing the ANN 150.

[0096] At box 613, output from the processing device is provided by processing intermediate results from the second computing device using a portion of the ANN. In one example, processing device 144 uses portion 156 of ANN 150 to process intermediate results 130 from the second computing device 310 and provides output 162. Output 162 can be used to control the movement 166 of vehicle 302.

[0097] In some embodiments, edge processing of data as described above (e.g., sensor or other data) is implemented using a cloud system. In one instance, a computing environment operates in conjunction with embodiments of this disclosure. Components of the computing environment can be implemented using any desired combination of hardware and software components.

[0098] The exemplary computing environment may include client computing devices, provider servers, authentication servers and / or cloud components that communicate with each other via a network (e.g., network 136).

[0099] The client computing device can be any computing device, such as a desktop computer, laptop computer, tablet computer, PDA, smartphone, mobile phone, smart appliance, wearable device, IoT device, in-vehicle device, etc. According to various embodiments, the client computing device accesses services at a provider server (e.g., computing device 138).

[0100] A client computing device (e.g., computing device 110) may include one or more input devices or interfaces for a user of the client computing device. For example, the one or more input devices or interfaces may include one or more of the following: a keyboard, mouse, trackpad, trackball, stylus, touchscreen, hardware buttons on the client computing device, etc. The client computing device may be configured to execute various applications (e.g., web browser applications) to access the network.

[0101] The provider server can be any computing device configured to host one or more applications / services. In some embodiments, the provider server may require security authentication before authorizing access to the services and / or resources offered on it. In some embodiments, the application / service may include an online service that can be accessed after the device has authenticated its access. In some embodiments, the provider server may be configured with an authentication server for authenticating users and / or devices. In other embodiments, the authentication server may be configured remotely and / or independently relative to the provider server.

[0102] The network can be any type of network configured to provide communication between components of the cloud system. For example, the network can be any type of network (including infrastructure) that provides communication, exchanges information, and / or facilitates the exchange of information, such as the Internet, local area network, wide area network, personal area network, cellular network, near field communication (NFC), optical code scanner, or other suitable connections that enable the sending and receiving of information between components of the cloud system. In other embodiments, one or more components of the cloud system may communicate directly via a dedicated communication link.

[0103] In various embodiments, the cloud system may also include one or more cloud components. Cloud components may include one or more cloud services, such as software applications (e.g., queues, etc.), one or more cloud platforms (e.g., web front-ends, etc.), cloud infrastructure (e.g., virtual machines, etc.), and / or cloud storage devices (e.g., cloud databases, etc.). In some embodiments, one or both of the provider server and the authentication server may be configured to operate in or with cloud computing / architecture, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and / or Software as a Service (SaaS).

[0104] In one embodiment, a system for edge processing to reduce data traffic on a network includes: at least one processing device (e.g., 144); and a memory containing instructions configured to instruct the at least one processing device to: receive data from a computing device (e.g., 110) via a network, wherein the received data is processed using an artificial neural network (ANN), the ANN including a neuron model (e.g., 152) for processing the data, and the computing device provides the data from processing sensor data (e.g., 106) obtained from at least one sensing device (e.g., 104) of a vehicle; send a first portion (e.g., 154) of the neuron model to the computing device, wherein the first portion is selected based on a predicted data flow (e.g., 134) associated with processing the sensor data, the predicted data flow including at least one of data flow to or from the computing device; receive intermediate results (e.g., 130) from the computing device, wherein the computing device provides intermediate results based on processing the sensor data using the first portion; and provide output by processing the intermediate results using a second portion (e.g., 156 or 158) of the neuron model.

[0105] In one embodiment, selecting a first part of a neuron model based on predicted data flow includes selecting a set of neurons from the neuron model, and selecting the set of neurons is based on at least one of the following: comparing the data size of the input data to be processed using the first part with the data size of the output data to be obtained from processing using the first part, or comparing the data size of the input data to be processed using the first part with the data size of the first part.

[0106] In one embodiment, the instructions are further configured to instruct at least one processing device to: predict, using at least a portion of the ANN, that the size of the sensor data processed using the first portion is greater than the size of the first portion; wherein the first portion is sent to a computing device in response to the prediction that the size of the sensor data processed using the first portion is greater than the size of the first portion.

[0107] In one embodiment, a method includes: processing data received from a second computing device (e.g., 110) via a network by a first computing device (e.g., 138) and using an artificial neural network (ANN), wherein the ANN includes a neuron model for processing the data, the second computing device processes sensor data from at least one sensing device of a vehicle (e.g., 102), and the second computing device provides received data from processing the sensor data; determining, by the first computing device, to perform processing on a first portion of the neuron model on the second computing device; in response to determining to perform processing on the first portion of the neuron model on the second computing device, transmitting the first portion via the network to the second computing device, wherein the second computing device provides intermediate results based on processing the sensor data using the first portion; receiving the intermediate results from the second computing device; and processing the intermediate results by the first computing device using a second portion of the neuron model to provide an output.

[0108] In one embodiment, the first computing device is a first server, and the second computing device is a second server on the communication path between the vehicle and the first server (e.g., 310).

[0109] In one embodiment, determining that processing is to be performed on the first portion of the second computing device includes at least one of the following: comparing the data size of the sensor data with the data size of the intermediate result, comparing the data size of the sensor data with the data size of the first portion, or comparing the data size of the intermediate result with the data size of the first portion.

[0110] In one embodiment, sensor data is a first input vector, and intermediate results are a first output vector.

[0111] In one embodiment, determining to perform processing on a first portion of the neuron model on a second computing device further includes predicting at least one of the data size of a first input vector or the data size of a first output vector.

[0112] In one embodiment, determining to perform processing on the first portion on the second computing device is based on determining whether sensor data will be uploaded to the first computing device after providing intermediate results.

[0113] In one embodiment, at least one sensing device includes an image sensor.

[0114] In one embodiment, the method further includes: determining, by a first computing device, to perform processing on a third portion of the ANN on a vehicle; and in response to determining that processing is to be performed on the third portion on a vehicle, sending the third portion via a network to at least one sensing device (e.g., 204), wherein: the at least one sensing device uses the third portion to generate sensor data to process at least one image from an image sensor; and the at least one sensing device sends the generated sensor data to a second computing device for determining intermediate results.

[0115] In one embodiment, the method further includes sending an output to a vehicle to control actions on the vehicle.

[0116] In one embodiment, the second computing device is a controller in the vehicle's computer system.

[0117] In one embodiment, the sensor data corresponds to an image collected by the vehicle's image sensors, and the intermediate results include at least one of the identification of objects in the image or the classification of objects.

[0118] In one embodiment, determining that processing is performed on a first portion of a neuron model on a second computing device includes evaluating the data flow between the first and second computing devices associated with providing output.

[0119] In one embodiment, the data flow is associated with at least one of the following: receiving sensor data from a second computing device after determining that the sensor data needs to be further processed by the ANN at a first computing device, sending a first portion to the second computing device, or receiving intermediate results from the second computing device.

[0120] In one embodiment, the method further includes selecting a first portion of the neuron model for transmission to a second computing device, wherein the selection is based on predicted data flow between the first and second computing devices.

[0121] In one embodiment, a system includes: at least one processing device (e.g., 116); and a memory containing instructions configured to instruct the at least one processing device to: process sensor data obtained from at least one sensing device of a vehicle using at least one portion of an artificial neural network (ANN) (e.g., 106); transmit results from processing the sensor data to a computing device via a network (e.g., 138); receive a first portion of the ANN from the computing device, the first portion being selected based on a prediction of data flow associated with processing the sensor data, the predicted data flow including at least one of data flow to or from the computing device; and transmit an intermediate result (e.g., 130) determined based on processing the sensor data using the first portion to the computing device, wherein the computing device provides an output for controlling the vehicle, and the output is obtained by processing the intermediate result using a second portion of the ANN.

[0122] In one embodiment, the instructions are further configured to instruct at least one processing device to: compare the data size of the sensor data (e.g., 108) with the data size (e.g., 160) of a first portion (e.g., 154) of the ANN; and send a request for the first portion to a computing device based on the comparison of the data size of the sensor data with the data size of the first portion; wherein the first portion is received from the computing device in response to the request.

[0123] In one embodiment, when using an ANN to process sensor data, a first portion is selected to reduce the flow of predicted data to and / or from the computing device.

[0124] This disclosure includes various means for performing methods and implementing the systems described above, including a data processing system for performing these methods, and a computer-readable medium containing instructions that, when executed on the data processing system, cause the system to perform these methods.

[0125] The descriptions and figures are illustrative and should not be construed as limiting. Many specific details are described to provide a thorough understanding. However, in some cases, well-known or conventional details have not been described to avoid obscuring the description. References to one or more embodiments in this disclosure do not necessarily refer to the same embodiment; and such references imply at least one.

[0126] In this specification, references to "an embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of this disclosure. The phrase "in one embodiment" appearing in various places throughout this specification does not necessarily refer to all of the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. Furthermore, various features that may be present in some embodiments but not in others are described. Similarly, various requirements are described, which may be requirements for some embodiments but not for others.

[0127] In this specification, various functions and / or operations may be described as being executed by or caused by software code for the sake of simplicity. However, those skilled in the art will recognize that this expression means that the functions and / or operations are caused by one or more processing devices executing code, such as microprocessors, application-specific integrated circuits (ASICs), graphics processors, and / or field-programmable gate arrays (FPGAs). Alternatively or in combination, functions and operations may be implemented using dedicated circuit systems (e.g., logic circuit systems) with or without software instructions. Embodiments may be implemented using hardwired circuit systems without software instructions or in combination with software instructions. Therefore, the techniques described are neither limited to any particular combination of hardware circuit systems and software, nor to any particular source of instructions executed by a computing device.

[0128] While some embodiments may be implemented in fully functional computers and computer systems, various embodiments are capable of being distributed as computing products in various forms and can be applied regardless of the specific type of computer-readable medium in which the distribution is actually implemented.

[0129] At least some of the disclosed aspects may be embodied, at least in part, in software. That is, the technology may be implemented in a computing device or another system in response to its processing device (e.g., a microprocessor) executing a sequence of instructions contained in memory (e.g., ROM, volatile RAM, non-volatile memory, cache, or remote storage device).

[0130] The routines used to implement the embodiments described may be implemented as part of an operating system, middleware, service delivery platform, software development kit (SDK) component, network service, or other specific application, component, program, object, module, or instruction sequence (sometimes referred to as a computer program). The calling interface of these routines may be exposed to the software development community as an application programming interface (API). Computer programs typically include one or more sets of instructions stored in various memories and storage devices within a computer at various times, and these sets of instructions, when read and executed by one or more processors in the computer, cause the computer to perform operations necessary for carrying out elements involving various aspects.

[0131] Computer-readable media can be used to store software and data that, when executed by a computing device, cause the device to perform various methods. Executable software and data can be stored in various locations including, for example, ROM, volatile RAM, non-volatile memory, and / or caches. A portion of this software and / or data can be stored in any of these storage devices. Furthermore, data and instructions can be obtained from a centralized server or a peer-to-peer network. Different portions of the data and instructions can be obtained at different times and in different communication sessions or in the same communication session from different centralized servers and / or peer-to-peer networks. Data and instructions can be obtained in their entirety before the application is executed. Alternatively, portions of data and instructions can be obtained dynamically and as needed for execution. Therefore, it is not required that all data and instructions be on the computer-readable media at any given time.

[0132] Examples of computer-readable media include, but are not limited to, recordable and non-recordable media, such as volatile and non-volatile memory devices, read-only memory (ROM), random access memory (RAM), flash memory devices, solid-state drive storage media, removable disks, disk storage media, optical storage media (e.g., optical disc read-only memory (CDROM), digital versatile optical disc (DVD), etc.), and other media. Computer-readable media can store instructions. Other examples of computer-readable media include, but are not limited to, non-volatile embedded devices using NOR flash or NAND flash architectures. Media used in these architectures may include unmanaged NAND devices and / or managed NAND devices, including, for example, eMMC, SD, CF, UFS, and SSD.

[0133] Generally, non-transitory computer-readable media includes any entity that provides (e.g., stores) information in a form accessible by computing devices (e.g., computers, mobile devices, network devices, personal digital assistants, manufacturing tools with controllers, any device having a collection of one or more processors, etc.).

[0134] In various embodiments, hardwired circuitry systems can be used in combination with software and firmware instructions to implement the technology. Therefore, the technology is neither limited to any particular combination of hardware circuitry systems and software, nor to any particular source of instructions executed by a computing device.

[0135] A wide variety of computing devices of different types can be used to implement the various embodiments described herein. As used herein, examples of "computing device" include, but are not limited to, servers, centralized computing platforms, systems with multiple computing processors and / or components, mobile devices, user terminals, vehicles, personal communication devices, wearable digital devices, electronic self-service terminals, general-purpose computers, electronic document readers, tablet computers, laptop computers, smartphones, digital cameras, home appliances, televisions, or digital music players. Additional examples of computing devices include devices that are part of what is called the "Internet of Things (IoT)." Such "things" may interact incidentally with their owners or administrators who may monitor or modify settings on them. In some cases, such owners or administrators act as users of the "thing" device. In some instances, a user's primary mobile device (e.g., an Apple iPhone) may be an administrator server for paired "thing" devices worn by the user (e.g., an Apple Watch).

[0136] In some embodiments, the computing device may be a computer or a host system, such as a desktop computer, laptop computer, web server, mobile device, or another computing device including memory and processing means. The host system may include or be coupled to a memory subsystem, such that the host system can read data from or write data to the memory subsystem. The host system may be coupled to the memory subsystem via a physical host interface. Generally, the host system may access multiple memory subsystems via the same communication connection, multiple separate communication connections, and / or a combination of communication connections.

[0137] In some embodiments, the computing device is a system comprising one or more processing devices. Examples of processing devices may include a microcontroller, a central processing unit (CPU), a dedicated logic circuit system (e.g., a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), etc.), a system-on-a-chip (SoC), or another suitable processor.

[0138] In one example, the computing device is a controller for a memory system. The controller includes a processing unit and a memory containing instructions executed by the processing unit to control various operations of the memory system.

[0139] While some diagrams illustrate several operations in a specific order, non-orderly dependent operations can be reordered and other operations can be combined or decomposed. Although some reorderings or other groupings are specifically mentioned, other reorderings or groupings will be obvious to those skilled in the art, and therefore an exhaustive list of alternatives is not provided. Furthermore, it should be recognized that stages can be implemented using hardware, firmware, software, or any combination thereof.

[0140] In the foregoing description, this disclosure has been described with reference to specific exemplary embodiments thereof. It will be apparent that various modifications may be made thereto without departing from the broader spirit and scope set forth in the appended claims. Therefore, the specification and drawings should be viewed in an illustrative rather than restrictive sense.

Claims

1. A system comprising: At least one processing device; as well as A memory containing instructions configured to instruct the at least one processing device: Data is received from a computing device via a network, wherein the received data is processed using an artificial neural network (ANN), the ANN including a neuron model for processing the data, and the computing device provides the data from processing sensor data obtained from at least one sensing device of the vehicle; A first portion of the neuron model is sent to the computing device, wherein the first portion is selected based on a predicted data flow associated with processing the sensor data, the predicted data flow including at least one of data flow to or from the computing device; Receive intermediate results from the computing device, wherein the computing device provides the intermediate results based on processing the sensor data using the first portion; and The intermediate results are processed using the second part of the neuron model to provide an output for controlling the vehicle.

2. The system of claim 1, wherein selecting the first portion of the neuron model based on predicted data flow comprises selecting a set of neurons from the neuron model, and wherein selecting the set of neurons is based on at least one of: comparing the data size of input data to be processed using the first portion with the data size of output data to be obtained from processing using the first portion, or comparing the data size of input data to be processed using the first portion with the data size of the first portion.

3. The system of claim 1, wherein the instructions are further configured to instruct the at least one processing device: Using at least a portion of the ANN, the predicted data size of the sensor data processed using the first portion is greater than the data size of the first portion; The first portion is sent to the computing device in response to the prediction that the size of the sensor data processed using the first portion is greater than the size of the first portion.

4. A method comprising: The data received from a second computing device via a network is processed by a first computing device using an artificial neural network (ANN), wherein the ANN includes a neuron model for processing the data, the second computing device processes sensor data from at least one sensing device of the vehicle, and the second computing device provides the received data obtained from processing the sensor data. The first computing device determines that processing will be performed on a first portion of the neuron model on the second computing device; In response to determining that processing is to be performed on the first portion of the neuron model on the second computing device, the first portion is sent to the second computing device via the network, wherein the second computing device provides intermediate results based on processing the sensor data using the first portion; Receive the intermediate results from the second computing device; as well as The intermediate results are processed by the first computing device using a second part of the neuron model to provide an output for controlling the vehicle.

5. The method of claim 4, wherein determining to perform the processing on the first portion on the second computing device is based on determining whether the sensor data will be uploaded to the first computing device after the intermediate result is provided.

6. The method of claim 4, further comprising: The first computing device determines that processing will be performed on the third part of the ANN on the vehicle; as well as In response to determining that processing is performed on the third part on the vehicle, the third part is transmitted via a network to the at least one sensing device, wherein: The at least one sensing device uses the third part to generate the sensor data to process at least one image from the image sensor; and The at least one sensing device sends the generated sensor data to the second computing device to determine the intermediate result.

7. The method of claim 4, wherein the sensor data corresponds to an image collected by an image sensor of the vehicle, and the intermediate result includes at least one of an identifier of an object in the image or a classification of the object.

8. The method of claim 4, wherein determining to perform the processing on the first portion of the neuron model on the second computing device includes evaluating the data flow between the first computing device and the second computing device associated with providing the output.

9. A system comprising: At least one processing device; as well as A memory containing instructions configured to instruct the at least one processing device: At least one part of an artificial neural network (ANN) is used to process sensor data obtained from at least one sensing device of the vehicle. The results from processing the sensor data are sent to a computing device via a network; The first portion of the ANN is received from the computing device, the first portion being selected based on a prediction of data traffic associated with processing the sensor data, the predicted data traffic including at least one of data traffic to or from the computing device; and The intermediate results determined based on the processing of the sensor data using the first part are sent to the computing device, wherein the computing device provides an output for controlling the vehicle, and the output is obtained by processing the intermediate results using the second part of the ANN.

10. The system of claim 9, wherein the instructions are further configured to instruct the at least one processing device: The size of the sensor data is compared with the size of the first part of the ANN; and Send a request for the first portion to the computing device based on comparing the data size of the sensor data with the data size of the first portion; The first part is received from the computing device in response to the request.

Images