Shared representation of neural network resources

Abstract

To provide a method for generating a neural network group.SOLUTION: The method includes determining a first layer portion shared by a first neural network subgroup of the neural network group, determining a second layer portion shared by a second neural network subgroup of the neural network group such that different second layer portions are shared by different second neural network subgroups of the neural network group, and determining a third layer portion shared by a third neural network subgroup of the neural network group such that different third layer portions are shared by different third neural network subgroups of the neural network group. Each neural network of the group of neural networks includes a unique combination of layer portions.SELECTED DRAWING: Figure 3

Description

[Background technology]

[0001] Neural networks are used in vehicles for a variety of purposes, including classifying items sensed by sensors associated with the vehicle and providing driving-related responses based on the item classification.

[0002] Neural networks are expected to provide highly accurate responses in changing environments and can be very large, making them resource intensive.

[0003] There is an increasing need to reduce the resource consumption associated with storing neural networks and performing neural network-related processing. Summary of the Invention [Problem to be solved by the invention]

[0004] Presented in this disclosure are methods, systems and non-transitory computer-readable media. [Brief explanation of the drawings]

[0005] The embodiments of the present disclosure can be more fully understood and appreciated from the following detailed description taken in conjunction with the accompanying drawings, in which: [Figure 1] FIG. 1 illustrates an example of a system. [Figure 2] FIG. 1 illustrates an example of a system. [Figure 3] FIG. 10 is a diagram illustrating an example of a neural network group. [Figure 4] 1A-1C illustrate examples of stages in the neural network group generation process. [Figure 5] FIG. 10 illustrates an example of another stage in the neural network group generation process. [Figure 6] FIG. 10 illustrates an example of a neural network group after adding a new neural network. [Figure 7]FIG. 1 illustrates an example of a method. [Figure 8] An example of the method is shown below. DETAILED DESCRIPTION OF THE INVENTION

[0006] The different figures show examples of units and / or software and / or information items and / or steps and / or components. These examples are provided for simplicity of explanation. At least one of the units and / or software and / or information items and / or steps and / or components is optional or required.

[0007] According to an embodiment, a neural network group is provided, the neural network group including a shared layer portion that is shared among different neural networks of the neural network group. The use of the shared layer portion can significantly reduce memory consumption (e.g., by at least 1.1x, 1.2x, 1.5x, 2x, 2.5x, 3x, 3.5x, 4x, 4.5x, 5x, 10x, 20x, 50x, 100x, 200x, or even more).

[0008] In some embodiments, the neural network group is determined based on one or more input constraints, such as one or more dimension parameters, such as the overall dimension, the storage space required to store the neural network group, the number of layers in the neural network group, the number of layer portions in each layer in the neural network group, the dimension weights (bits assigned to weights) and / or biases and / or activation values.

[0009] Depending on the embodiment, the neural network group may exhibit one or more additional input constraints, such as the type of numbers (fixed-point or floating-point representation) and / or the type of activation function.

[0010] In some embodiments, one or more dimension parameters of the neural network group are determined based on at least one rule, at least one model, at least one constraint, at least one optimization parameter, computational consumption associated with implementing the neural network group, memory consumption associated with implementing the neural network group, and a trade-off between the one or more dimension parameters of the neural network group and one or more performance parameters of the neural network group (e.g., accuracy, delay, false positive rate, false negative rate, true positive rate, true negative rate, etc.).

[0011] Depending on the embodiment, the trade-off result is determined by applying a function to two or more factors or parameters or values, which may be linear, non-linear, exponential, logarithmic, a weighted sum, etc.

[0012] By way of example, neural network groups can grow logarithmically rather than exponentially or multiplicatively (when individually trained neural networks are used instead of neural network groups).

[0013] According to an embodiment, a vehicle may store one or more neural network groups, and these neural network groups may be provided to one or more other neural network groups (e.g., via vehicle-to-vehicle communication or communication between the vehicle and another entity). The provision of neural network groups may be dynamic, such that a vehicle may store different neural network groups at different times.

[0014] Depending on the embodiment, the dynamic rendering may depend on one or more parameters, such as the route taken by the vehicle (different routes may be better served by different neural network groups), the driver (automated or human, a particular driver or another particular driver) driving the vehicle, and / or any other scene parameters and / or context parameters and / or environmental parameters and / or safety parameters and / or comfort parameters that may affect the relevance (or suitability) of one or more neural network groups sent to the vehicle.

[0015] Additionally or alternatively, the dynamic provisioning may be based on static and / or dynamic constraints associated with the vehicle, such as the current availability of memory resources and / or the current availability of processing resources and / or the current availability of communication resources (intra-vehicle communication resources and / or extra-vehicle communication resources) and / or the maximum capacity of memory resources and / or the maximum capacity of processing resources and / or the maximum capacity of communication resources (intra-vehicle communication resources and / or extra-vehicle communication resources).

[0016] Additionally or alternatively, the dynamic provisioning is based on a delay constraint. The delay constraint may be set by the driver, the vehicle manufacturer, or another entity. By way of example, the delay constraint may force the neural network processing group to reduce its delay when faced with certain scenes and / or certain environmental conditions, and / or when the complexity of the environment and / or the level of danger associated with driving increases above a threshold. For example, in more dangerous environments and / or under conditions of reduced visibility, and / or when the driver is an inexperienced or accident-prone human driver, the delay should be lower.

[0017] By way of example, neural network groups exhibit perfect connectivity.

[0018] By way of example, neural network groups exhibit only partial connectivity.

[0019] Depending on the embodiment, some layers of the neural network group are fully connected, and some other layers of the neural network group are only partially connected.

[0020] According to the embodiment, the neural network group has a tree structure.

[0021] By way of example, a leaf layer portion of a neural network group is unique in the sense that the leaf layer portion is contained in a single neural network of the neural network group.

[0022] By way of example, all neural networks in a neural network group have the same number of layers.

[0023] By way of example, one or more neural networks of the neural network group are longer than one or more other neural networks of the neural network group.

[0024] Depending on the embodiment, different neural networks of the neural network group may include different combinations of layer portions.

[0025] By way of example, each neural network in the neural network group has a unique layer portion that is not included in any other neural network in the neural network group.

[0026] According to an embodiment, the neural network group includes four or more layers, wherein at least one neural network subgroup of the neural network group includes a different combination of portions of the first three layers of the neural network group.

[0027] According to an embodiment, the neural network group may have at least 5, 10, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1100, 1200, 1300, 1400, 1500, Includes over 2000, 2500, 3000, 3500, 4000, 4500, 5000, 5500, 10000, 11000, 12000, 15000, 20000, 30000, 40000, 50000, 60000, 100000, 150000, 200000, and 250000 neural networks.

[0028] According to an embodiment, a neural network group is provided, which includes: A. The first layer portion is shared by a first neural network subgroup of a neural network group. Depending on the embodiment, there may be multiple first layer portions. B. The second layer portion is shared by the second neural network subgroups, such that different second layer portions are shared by different second neural network subgroups of the neural network group. C. One or more additional layer portions are shared by one or more additional neural network subgroups, such that different neural network groups are shared by different one or more additional neural network subgroups of the neural network group.

[0029] By way of example, the neural network group may include three or more layers, for example, 4, 10, 15, 20, 25, 30, 40, 50, 60, 70, 100, 120, 150, 200, 250, 300, 400, 500, 1000, 1500, 2000 or more layers.

[0030] By way of example, each neural network in the neural network group includes a unique combination of layer portions.

[0031] According to an embodiment, each neural network of the neural network group comprises at least one dedicated layer portion that is not shared by another neural network of said neural network group.

[0032] By way of example, neural networks are trained on different narrow driving-related tasks.

[0033] By way of example, the first layer portion is shared by all neural networks in the neural network group.

[0034] According to an embodiment, at least one layer portion of the neural networks of the neural network group is generated by clustering layers of neural networks that have been trained individually.

[0035] According to various embodiments, clustering may include applying any clustering algorithm, such as at least one of the following: AK-Means Clustering: Partitions the data into K clusters by minimizing the variance within each cluster. B. Hierarchical clustering: Build a tree structure of clusters by sequentially merging or splitting clusters. C. Agglomerative Hierarchical Clustering: Start with individual points and combine them to form clusters. D. Partitional Hierarchical Clustering: Start with the entire dataset and divide it into clusters. E. DBSCAN (Density-Based Spatial Clustering for Applications with Noise): Groups tightly packed points and marks points in low-density regions as outliers. F.OPTICS (Identifying Clustering Structures by Point Ordering): An extension of DBSCAN that handles density variations more efficiently. G. Mean Shift Clustering: Identifies clusters by modes located in the density of data points. H. Gaussian Mixture Model (GMM): Assumes that the data is generated from a mixture of several Gaussian distributions and assigns a probability to each point belonging to a cluster. I. Spectral Clustering: The eigenvalues ​​of the similarity matrix are used in lower dimensions before clustering to reduce the dimension. J. Neighbor Propagation: Sends messages between points to identify samples and identifies clusters based on these samples. K. Birch (Reducing Clustering Hierarchies with Balancing Iterations): Perform hierarchical clustering on large datasets using a combination of other clustering methods. L. Agglomerative Clustering: Creates a hierarchy of clusters by merging clusters based on the shortest distance between them. M. Self-Organizing Map (SOM): An artificial neural network that is trained using unsupervised learning to generate a low-dimensional representation of data. N. Fuzzy C-means (FCM): Allows each data point to belong to multiple clusters with different membership degrees. O. HDBSCAN (Hierarchical Density-Based Spatial Clustering for Noisy Applications): An extension of DBSCAN to discover clusters of different densities.

[0036] Depending on the embodiment, clustering may be performed with or without a constraint on the number of clusters, and it has been discovered that constraints can improve the accuracy and / or time required to perform clustering.

[0037] According to an embodiment, at least some of the neural networks of the neural network group are generated during a period of mutual training of the neural networks.

[0038] According to an embodiment, one neural network of a neural network group is added to the neural network group after forming a subgroup of neural networks including some of the neural networks of the neural network group.

[0039] Depending on the embodiment, neural network groups may be generated in one or more ways.

[0040] Depending on the embodiment, the neural network group is modified in at least one way (eg, by adding a new neural network or by performing any other modification).

[0041] By way of example, a method for creating and / or modifying a neural network group is as follows. A. Includes training. B. Includes retraining. C. Does not include retraining. D. Does not include training. E. Train on another neural network (not included in the neural network group) first. F. Including training other neural networks. G. Includes multiple training iterations. H. Includes training and merging iterations. I. It involves using one or more cost functions. J. Involves multiple iterations in which one or more layer portions are changed during one iteration while maintaining other layer portions.

[0042] According to an embodiment, a method for generating a neural network group is provided, which includes determining first layer portions shared by first neural network subgroups of the neural network group, determining second layer portions shared by second neural network subgroups, where different second layer portions are shared by different second neural network subgroups of the neural network group, and determining third layer portions shared by third neural network subgroups, where different third layer portions are shared by different third neural network subgroups of the neural network group, wherein each neural network in the neural network group further includes a unique combination of layer portions.

[0043] According to an embodiment, the method includes determining connectivity between successive layer portions, and determining layer portions that may be shared by neural network subgroups based on the determined connectivity.

[0044] According to an embodiment, the determined connectivity is based on the partial connectivity between layer portions of a specified neural network layer and layer portions of another neural network layer.

[0045] According to an embodiment, the method includes determining the number of layer portions for each layer based on input constraints. By way of example, each neural network's unique combination of layers is specific to a narrow driving-related task.

[0046] By way of example, the sequence of layers from the first layer to the dedicated layer forms a neural network associated with driving-related tasks.

[0047] According to an embodiment, the method includes adding a new neural network to the neural network group.

[0048] By way of example, the addition is based on weight fitting between the weights of the new neural network's layers and the weights of the layers part of the neural network group.

[0049] By way of example, adding includes retraining a new neural network based on the neural network group.

[0050] By way of example, determining the layer portion of the neural network group is based on an individually trained initial neural network.

[0051] According to an embodiment, the determining includes determining the shared layer portion by clustering layers of the individually trained initial neural networks and merging layers of the clusters that correspond to the shared layer portion.

[0052] By way of example, determining the layer portion of the neural network group is based on the results of multiple training and merging iterations.

[0053] By way of example, the neural network group includes neural networks that are different from the end-to-end neural network.

[0054] By way of example, the neural network group includes an end-to-end neural network.

[0055] An example of an end-to-end neural network is a neural network that receives inputs such as sensing information (or preprocessed sensing information) and outputs driving-related decisions (e.g., commands to perform driving-related operations or suggestions to perform driving-related decisions). The driving-related decisions may be automated driving decisions or advanced driver assistance system decisions, or any decisions that result in configuring or changing the vehicle's behavior.

[0056] A neural network in the neural network group may be a narrow artificial intelligence agent, where narrow may mean that the narrow artificial intelligence agent is responsible for managing only a small portion (e.g., less than 1 percent) of the scenes that the vehicle may encounter.

[0057] In the following text, reference numbers relating to Figure 1 are shown without parentheses, reference numbers relating to Figure 2 are shown within parentheses, and any item associated with a reference number without parentheses may be different from or equal to the corresponding item within parentheses.

[0058] 1 and 2 show example vehicles 100 and 400, which include a communication system 130 (430), one or more memory and / or storage units 120 (420), and a processing system 124 (424) including a processor 126 (426). The communication system 130 (430), the one or more memory and / or storage units 120 (420), and the processing system 124 can belong to a computerized system. The computerized system may be a server, a laptop computer, a desktop computer, or any other computer, and may include and communicate with sensing units and / or controllers.

[0059] According to an embodiment, vehicle 100 (400) communicates with network 132 (432) and one or more other remote computerized systems 134 (434) that communicate with network 132 (432). Examples of remote computerized systems are a server or one or more computers that access a storage system that stores items related to one or more portions of one or more neural network groups, at least some of which are not currently stored on the vehicle.

[0060] Depending on the embodiment, the communication system 130 (430) is configured to enable communication between one or more memory and / or storage units 120 (420) and / or sensing systems 110 (410) and / or any of additional units and / or networks 132 (432) (i.e., communicating with remote computerized systems).

[0061] The memory and / or storage unit 120 (420) is shown as storing software. Any reference to software should be made mutatis mutandis to apply to code and / or firmware and / or instructions and / or commands, etc.

[0062] The processor 126 (426) includes multiple processing units 126(1)-126(J) [426(1)-426(J)], where J is an integer greater than 1. Any reference to one unit or item should be made mutatis mutandis to apply to multiple units or items. For example, any reference to a processor should be made mutatis mutandis to apply to multiple processors, and any reference to a communication system 130 (430) should be made mutatis mutandis to apply to multiple communication systems.

[0063] Depending on the embodiment, one or more memory and / or storage units 120 (420) may include one or more memory units, and each memory unit may include one or more memory groups.

[0064] Depending on the embodiment, the one or more memory and / or storage units 120 (420) may include volatile memory and / or non-volatile memory. The one or more memory and / or storage units 120 (420) may be random access memory (RAM) and / or read-only memory (ROM).

[0065] According to an embodiment, the non-volatile storage unit is a mass storage device that provides non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data to the processor or any other unit of the vehicle. For example, and without limitation, the mass storage device may be a hard disk, a removable disk, a removable optical disk, a magnetic tape cartridge or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD) or other optical storage device, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0066] Any content may be stored in any portion of a memory and / or storage unit or in any type of memory and / or storage unit.

[0067] According to an embodiment, the at least one memory unit stores at least one database, such as any database known in the art, for example, DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, etc.

[0068] The memory and / or storage unit 120 (420) is configured to store firmware and / or software, one or more operating systems, data and metadata necessary to perform any of the methods mentioned in this disclosure.

[0069] The memory and / or storage unit 120 (420) is shown as storing software. Any reference to software should be made mutatis mutandis to apply to code and / or firmware and / or instructions and / or commands, etc.

[0070] The various units and / or components communicate with each other using any communication element and / or protocol. An example of a communication system is shown as 130 (430). Other communication elements may be provided.

[0071] The communications system 130 (430) can communicate with the bus 136 (436). Bus representations include one or more of several possible bus structures, such as a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of a variety of bus architectures. For example, such architectures may include an Industry Standard Architecture (ISA) bus, a MicroChannel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, an Accelerated Graphics Port (AGP) bus, a Peripheral Component Interconnect (PCI), a PCI-Express bus, a Personal Computer Memory Card Industry Association (PCMCIA), a Universal Serial Bus (USB), and the like. This bus, and all buses specified herein, can also be implemented through wired or wireless network connections and their respective subsystems.

[0072] The network 132 (432) is located outside the vehicle and is used for communication between the vehicle and at least one remote computing system. By way of example, the remote computing system may be a personal computer, laptop computer, portable computer, server, router, network computer, peer device, or other public network node. Logical connections between the processor and any remote computing system may be realized through local area networks (LANs) and general wide area networks (WANs). Such network connections may be realized in wired and wireless environments through network adapters (which may belong to the communication system 130 (430)). Such networking environments are common and typical in offices, enterprise-wide computer networks, intranets, and larger networks such as the Internet.

[0073] It should be noted that at least a portion of the content shown as being stored in one or more memory / storage units 120 (420) may be stored external to the vehicle. It should also be noted that the processor may evaluate signatures produced by multiple detectors.

[0074] Examples of signature generation and / or image cropping are provided in US Patent Application No. 18 / 527,701, which is incorporated herein by reference.

[0075] According to an embodiment, the memory and / or storage unit 120 (420) stores at least one of the operating system 194 (494), the information 191 (491), the metadata 192 (492), and the software 193 (493).

[0076] Using this software, the processing system is configured to perform one or more of methods 200 and 500 .

[0077] The vehicle 100 (400) further includes a sensing system 110 (410) and a control unit 125 (425).

[0078] The control unit 125 (425) may cooperate with an advanced driver assistance system (ADAS) control unit (e.g., ADAS control unit 481 in FIG. 2 ), an autonomous driving control unit 482 (e.g., autonomous driving (AD) control unit 482 in FIG. 2 ), and / or control or communicate with other vehicle components (including a vehicle computer, e.g., vehicle computer 421 in FIG. 2 ).

[0079] The ADAS control unit 481 is configured to control the ADAS operation.

[0080] The AD control unit 82 is configured to control the automatic driving of the motor vehicle. The vehicle computer 421 is configured to control the operation of the vehicle, in particular the engine, the transmission, and any other vehicle systems or components.

[0081] The vehicle computer 421 may communicate with an engine control module, a transmission control module, a drivetrain control module, and the like.

[0082] The sensing system 110 (410) may include an optical element, a sensing element group, a readout circuit, and an image signal processor. The optical element is followed by a sensing element group, e.g., a sensing element array forming a sensing element line or a sensing element group. The sensing element group is followed by a readout circuit, which reads out the detection signals generated by the sensing element group. The image signal processor is configured to perform initial processing of the detection signals, e.g., to improve the quality of the detection information and perform noise removal, etc. The sensing system 110 (410) is configured to output one or more sensing information units (SIUs).

[0083] The control unit 125 (425) is configured to control the operation of the sensing system 110 (410), and / or one or more memory and / or storage units 120 (420), and / or one or more additional units (excluding the controller).

[0084] Depending on the embodiment, one or more memory and / or storage units 120 (420) may include one or more memory units, and each memory unit may include one or more memory groups.

[0085] Depending on the embodiment, the one or more memory and / or storage units 120 (420) may include volatile memory and / or non-volatile memory. The one or more memory and / or storage units 120 (420) may be random access memory (RAM) and / or read-only memory (ROM).

[0086] According to an embodiment, the non-volatile storage unit is a mass storage device that provides non-volatile storage of computer code, computer-readable instructions, data structures, program modules, and other data to the processor or any other unit of the vehicle. For example, and without limitation, the mass storage device may be a hard disk, a removable disk, a removable optical disk, a magnetic tape cartridge or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD) or other optical storage device, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), etc.

[0087] By way of example, and not limitation, computer-readable media may include "computer storage media" and "communications media." "Computer storage media" includes volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, or other data. Exemplary computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVDs) or other optical storage devices, magnetic tape cartridges, magnetic tape, magnetic disk storage devices, or any other medium used to store the desired information and accessible by a computer.

[0088] Any content may be stored in any portion of a memory and / or storage unit or in any type of memory and / or storage unit.

[0089] According to an embodiment, the at least one memory unit stores at least one database, such as any database known in the art, for example, DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, mySQL, PostgreSQL, etc.

[0090] The various units and / or components communicate with each other using any communication element and / or protocol. An example of a communication system is shown as 430. Other communication elements may be provided.

[0091] FIG. 2 illustrates one or more memory and / or storage units as including a cache memory 420A, which may be used to store the most relevant information and / or instructions and / or metadata, such as one or more shared layer portions (particularly the weights and / or biases and / or connectivity of the shared layer portions).

[0092] FIG. 2 further illustrates one or more items (one item, some items, or all items) that may be stored in one or more memories and / or storage units, including: A. The neural network software 496, when executed by the processor, results in neural network processing, which can apply neural network groups to information. B. Neural network metadata 497 stores weights and / or biases and / or connectivity of a neural network group, or weights and / or biases and / or connectivity of at least a portion of a neural network group. C. AD software 498 is configured to control AD ​​operations (e.g., by instructing AD control unit 482 to execute instructions), suggest AD operations to AD control unit 482, or implement at least some of the functions of AD control unit 482. D. AD software 499 is configured to control AD ​​operations (e.g., by instructing AD control unit 491 to execute instructions), suggest AD operations to AD control unit 491, or implement at least some of the functions of AD control unit 491. E. Neural network management software 495 is configured to manage the storage of layer portions of neural network groups in one or more memory / storage units 420, e.g., these portions are stored in a cache or otherwise accessible by processor 426.

[0093] FIG. 3 shows an example of a neural network group 60 that includes K neural networks 62-1 through 62-K, where K is an integer greater than four.

[0094] The first layer includes a first number N1 of first layer portions LP(1,1)-LP(1,N1) 70(1,1)-70(1,N1), a second number N2 of second layer portions LP(2,1)-LP(2,N2) 70(2,1)-70(2,N2), a third number N3 of third layer portions LP(3,1)-LP(3,N3) 70(3,1)-70(3,N3), and one or more additional layers, where each neural network includes a unique combination of layer portions, such as LP(U,1)-LP(U,K) 70(U,1)-70(U,K), where U represents unique. The unique portions of different neural networks may belong to the same layer of the neural network group or to different layers of the neural network group.

[0095] 4-5 show example stages in the process of generating one or more neural network groups.

[0096] According to an embodiment, the one or more generation processes include clustering layers of the individually trained neural networks and generating layer parts based on the clusters (e.g., one layer part for each cluster) to generate at least one layer part of a neural network of the neural network group.

[0097] According to an embodiment, the one or more generation processes include generating at least one layer portion of the neural network during a co-training period of the neural network.

[0098] 4 shows multiple (J) individually trained neural networks 70-1 through 70-J. These neural networks may also be referred to as initial neural networks because they are processed to generate a group of neural networks, each of which is different from the initial neural network.

[0099] Each individually trained neural network includes an input layer {L(1,in)-L(J,in), 72(1,in)-72(J,in)}, one or more hidden layers {e.g., {L(1,hidden)-L(J,hidden), 72(1,hidden)-72(J,hidden)}, and an output layer {L(1,out)-L(J,out), 72(1,out)-72(J,out)}. The first cluster mapping 75 includes representations 77(1,in)-77(J,in) of the input layers L(1,in)-L(J,in) of J individually trained neural networks, which are clustered to provide N1 clusters, where N1 equals 3 in FIG. 3, as shown, for example, by the three clusters 78(1), 78(2), and 78(3).

[0100] These representations may be vectors, embeddings, embedded signatures, etc. Examples of embeddings are given in U.S. Patent Application No. 18 / 527,701, which is incorporated herein by reference.

[0101] 5 shows a second cluster mapping 80, which includes representations 82(1,hidden)-82(J,hidden) of the first hidden layers of J individually trained neural networks, which are clustered to provide N2 clusters, where in FIG. 4 N2 equals 10, as shown, for example, by the ten clusters 84(1)-84(10).

[0102] FIG. 6 shows a modified neural network group 60-1 in which a new neural network is added after the creation of neural network group 60 of FIG. 3. The new neural network 62-L+1 includes a unique combination of layer portions and a unique layer portion LP(U,K+1) 70(U,K+1).

[0103] FIG. 7 illustrates an example method 200 for generating a group of neural networks.

[0104] According to an embodiment, the method 200 begins with an initialization step 210 .

[0105] Depending on the embodiment, the initialization step 210 includes receiving or determining one or more input constraints, such as size constraints or any other input constraints, where at least some of the input constraints are set forth above.

[0106] According to an embodiment, step 210 includes determining a plurality of layer portions for each layer based on input constraints, such as but not limited to any of the input constraints described above.

[0107] According to an embodiment, step 210 includes determining connectivity between successive layer portions, and determining layer portions that may be shared by neural network subgroups based on the determined connectivity.

[0108] According to an embodiment, the determined connectivity is based on the partial connectivity between the layer portion of the specified neural network layer and the layer portion of another neural network layer.

[0109] According to an embodiment, the connectivity determination is based on the connectivity of the individually trained neural networks that are processed to generate the neural network group. For example, suppose a first layer portion is generated based on a set of first layers of individually trained neural networks, the first layer portion is connected to one or more second layer portions, and these second layer portions are generated based on one or more sets of second layers of individually trained neural networks that are connected to the first layers in the set of first layers.

[0110] See cluster 78(1) in FIG. 4, which represents a collection of first layers of individually trained neural networks.

[0111] The first layer sets are connected to the second layer sets represented by the cluster sets of the second cluster mapping 80 (eg, clusters 84(1), 84(5), and 84(9)).

[0112] The set of clusters of the second cluster mapping (e.g., clusters 84(1), 84(5), and 84(9)) is used to generate a set of three second layer portions of the neural network group, and the first layer portion associated with cluster 78(1) is connected to the set of three second layer portions of the neural network group.

[0113] Depending on the embodiment, the connectivity determination is performed during or between any of steps 220, 230 and 240.

[0114] According to an embodiment, step 210 is followed by step 220, which determines a first layer portion shared by a first neural network subgroup of the neural network group. According to an embodiment, there are multiple first layer portions.

[0115] According to an embodiment, step 220 is followed by step 230, in which different second layer portions are shared by different second neural network subgroups of the neural network group by determining second layer portions that can be shared by second neural network subgroups.

[0116] According to an embodiment, step 230 is followed by step 240, in which different third layer portions are shared by different third neural network subgroups of the neural network group by determining third layer portions that can be shared by the third neural network subgroups.

[0117] According to an embodiment, step 240 is followed by step 250, which determines one or more additional layer portions that may be shared by one or more additional neural network subgroups, where different one or more additional layer portions are shared by different one or more additional neural network subgroups of the neural network group.

[0118] According to an embodiment, determining at least one of steps 220, 230, 240, and 250 includes determining the shared layer portion by clustering layers of the individually trained initial neural networks and merging layers of the clusters that correspond to the shared layer portion.

[0119] By way of example, the determination of at least one of steps 220, 230, 240 and 250 is based on the results of multiple training and merging iterations.

[0120] Depending on the embodiment, merging may involve averaging the weights and biases of the layers of the cluster. Depending on the embodiment, merging may involve applying a weighted sum or any function to the weights and biases of the layers of the cluster.

[0121] According to an embodiment, the method 200 provides a group of neural networks, where each neural network in the group of neural networks includes a unique combination of layer portions.

[0122] By way of example, the unique combination of layer portions used in each neural network is specific to a narrow driving-related task.

[0123] By way of example, the sequence of layers from the first layer to the dedicated layer forms a neural network associated with driving-related tasks.

[0124] According to an embodiment, the method 200 includes a step 260 of updating the neural network group. For example, the step 260 includes adding a new neural network to the neural network group.

[0125] By way of example, the addition is based on weight fitting between the weights of the new neural network's layers and the weights of the layers part of the neural network group.

[0126] By way of example, adding includes retraining a new neural network based on the neural network group.

[0127] FIG. 8 illustrates an example method 500 for using a group of neural networks in connection with a driving-related application.

[0128] According to an embodiment, the method 500 begins with an initialization step 510 .

[0129] According to an embodiment, the initialization step 510 includes acquiring the neural network group. Acquiring may include receiving the neural network group by the vehicle from a computerized system external to the vehicle and / or loading the neural network group into a memory unit of the vehicle. The memory unit may be a cache memory and / or may be included in one or more memories / memory units (or one or more memories / memory units). Acquiring may include requesting the neural network group from a remote computerized system.

[0130] By way of example, this group of neural networks includes: A. The first layer portion is shared by a first neural network subgroup of a neural network group. There can be multiple first layer portions. B. The second layer portions can be shared by second neural network subgroups, such that different second layer portions are shared by different second neural network subgroups of the neural network group. C. The one or more additional layer portions are shared by one or more additional neural network subgroups, such that different neural network groups are shared by different one or more additional neural network subgroups of the neural network group. The one or more additional layer portions include a third layer portion, and the third layer portion is shared by a third neural network subgroup, such that different third layer portions are shared by different third neural network subgroups of the neural network group.

[0131] By way of example, each neural network in the neural network group includes a unique combination of layer portions.

[0132] By way of example, the acquisition is based on one or more input constraints, such as a size constraint or any other input constraint, where at least some of the input constraints are set forth above. Additionally or alternatively, the acquisition is based on the expected path of the vehicle, weather conditions, availability of resources for running and / or storing the neural network group, etc.

[0133] According to an embodiment, step 510 is followed by step 520, in which the group of neural networks (or at least some of the neural networks in the group of neural networks) are applied to the input information to generate driving-related decisions and / or classification decisions and / or lane detection decisions, etc.

[0134] Depending on the embodiment, multiple iterations of steps 510 and 520 may be performed, obtaining different neural network groups, for example, if the vehicle reaches a different location than where the vehicle was in the previous iteration of step 520.

[0135] By way of example, a neural network group is part of a collection of narrow AI agents, with different AI agents being realized by different neural networks in the neural network group. A neural network group may be preceded by a perception router that sends perception information to the associated narrow AI agents.

[0136] By way of example, the collection includes a large number (eg, at least 50,000) of neural networks implemented by a neural network group.

[0137] In some embodiments, the relevant narrow AI agent (trained to perform tasks related to the received sensory information and / or vehicle task and / or vehicle route) is activated, while at least some of the other narrow AI agents are not. Use of a shared network layer may require activating at least some of the unrelated neural networks (associated with the unrelated narrow AI agents).

[0138] An example of a narrow set of AI agents is given in US patent application Ser. No. 18 / 036,150, which is incorporated herein by reference.

[0139] By way of example, this aggregation uses the neural network group to reduce resource consumption and promote resources at a relatively accessible level, ensuring rapid activation with low latency.

[0140] By way of example, a group of neural networks is in the form of a branched tree of shared layers / representations between these networks, thus requiring less memory than the original structure of independent networks.

[0141] For example, suppose there are to be 400,000 neural networks, and a neural network group is provided, which includes: A. Instead of 400,000 first layers with 400,000 different neural networks, use a smaller first number N1 (e.g., 3) of first layer portions. B. Instead of 400,000 second layers with 400,000 different neural networks, use a smaller second number N2 (e.g., 10) of second layer portions. C. Instead of 400,000 third layers with 400,000 different neural networks, use a smaller third number N3 (e.g., 100) of third layer portions.

[0142] Neural network groups can increase the number of neural networks by using a shared layer portion that grows logarithmically, rather than an exponential or multiplicative number of layers.

[0143] By way of example, neural network groups are constructed during the generation process using kernels / filters / weights similar to a shared representation, taking into account the shared representation across different layers of the neural network.

[0144] By way of example, even though different neural networks are assigned to different skills or tasks, this group of neural networks can be operated as a liquid state neural network, which may include selectively activated and deactivated neural networks.

[0145] Depending on the implementation, activation of each layer portion (particularly when selectively activating (only or mostly) correlated neural networks and decreasing in layer order) may increase and / or there may be more layer portions in each layer. For example, the first layer portion may be activated 50% of the neural network group's execution time, the second layer portion 10% of its execution time, the third layer portion 1% of its execution time, etc.

[0146] According to an embodiment, generating a neural network group may be based on multi-task training, as different neural networks within the neural network group are assigned to different tasks.

[0147] By way of example, generating neural network groups can include, for example, weight sharing during multitask training. This can improve performance by utilizing domain-specific information contained in the training signals of related tasks, as well as allowing for more efficient parameterization of the models. By way of example, weight sharing or merging can utilize weight sharing techniques such as those described in "Multitask learning," Caruana, R., Machine learning volume 28, pages 41-75 (1997), which is incorporated herein by reference.

[0148] By way of example, neural network groups are generated using a two-stage approach: (a) independently training M independent networks (M narrow agents), and (b) combining earlier stages of the networks to have a shared representation for efficient runtime. This can be achieved by finding similar learning weights among the parallel narrow networks.

[0149] By way of example, the neural network group is generated using neural network distillation, an example of which is provided in "Distilling the Knowledge in a Neural Network" by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (2015): https: / / arxiv.org / abs / 1503.02531, which is incorporated herein by reference.

[0150] According to an embodiment, a group of neural networks is trained using an end-to-end method, which involves training the group of neural networks with a shared architecture from the beginning, where the agents learn each narrow task while being constrained to share weights for effective learning.

[0151] According to an embodiment, a group of neural networks is trained using a hypernetwork method, which involves using a scene-based hypernetwork to predict scene-specific weights, where some layers are fixed and shared among all agents, but a separate hypernetwork predicts the scene-specific weights.

[0152] By way of example, the initial network layer part captures low level information, which is more likely to be shared between (more) different agents.

[0153] By way of example, the selection of layer portions to share after training is performed as mentioned by the above method via network distillation and information similarity between different networks, see, for example, the following sentences, which are incorporated herein by reference: A. Maximillian Schmidt, Marvin Giese, and Eric J. Wederman, "Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation" (2018) by Maximillian Schmidt, Marvin Giese, and Eric J. Wederman. B. Romain Lopez-Roig, Janis Lizarraguirre-Rodriguez, and Armand Joubert (2019) "Similarity of Neural Network Representations Revisited": https: / / arxiv.org / pdf / 2305.06329

[0154] In some embodiments, where a collection of agents is provided, the hypernetwork method is used in conjunction with a routing mechanism.

[0155] By way of example, sharing of layer portions is done in narrow scenarios and not on a per-function basis.

[0156] Depending on the embodiment, the neural network group proposal and / or neural network group creation may be unknown to the training of the initially trained neural network, and the initially trained neural network may be trained separately, autonomously and independently, or alternatively, cross-network training may be used.

[0157] Any combination of any steps of any method set forth in this disclosure is provided.

[0158] In the foregoing detailed description, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be understood by those skilled in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the present invention.

[0159] While the subject matter regarded as the present invention is particularly pointed out and distinctly claimed at the concluding portion of the specification, the organization and method of operation, together with objects, features and advantages of the present invention, may best be understood by reference to the following detailed description when read in conjunction with the accompanying drawings.

[0160] It will be understood that elements shown in the figures have not necessarily been drawn to scale for simplicity and clarity of illustration. For example, the dimensions of some elements may be exaggerated relative to other elements for clarity. Also, where considered appropriate, reference numerals may be repeated among the figures to indicate corresponding or analogous elements.

[0161] Because the illustrated embodiments of the present invention may be implemented using electronic components and circuits that are, for the most part, known to those skilled in the art, it is not necessary to go into greater detail than set forth above in order to understand the basic concepts of the invention and not to confuse or distract from the teachings of the present invention.

[0162] Any reference in the specification to a method should be applied mutatis mutandis to a device or system capable of performing the method and / or a non-transitory computer-readable medium storing instructions for performing the method.

[0163] Any reference in the specification to a system or device should be applied mutatis mutandis to a method executable by the system and / or to a non-transitory computer-readable medium storing instructions executable by the system.

[0164] Any reference in the specification to a non-transitory computer-readable medium should apply mutatis mutandis to a device or system capable of executing instructions stored on the non-transitory computer-readable medium and / or to a method of executing the instructions.

[0165] Any combination of any modules or units described in any part of the drawings, the specification and / or any claims may be provided.

[0166] The transformation module, active learning module, clustering module, or any other module described herein may be implemented by code, instructions and / or commands stored on hardware and / or non-transitory computer-readable media, may be included in the vehicle, outside the vehicle, in a mobile device, in a server, etc.

[0167] The vehicle may be any type of vehicle, for example, a ground transport vehicle, an aircraft, or a watercraft.

[0168] The description and / or drawings may be images. An image is an example of sensed information. Any reference to an image may apply mutatis mutandis to any type of natural signal, such as signals generated by nature, signals representing human behavior, signals representing operations related to the stock market, medical signals, financial series, geodetic signals, geophysical, chemical, molecular, textual and numerical signals, time series, etc. Any reference to a media unit may also apply mutatis mutandis to sensed information. The sensed information may be of any type, sensed by any type of sensor, such as a visible light camera, an audio sensor, a sensor sensitive to infrared light, radar imaging, ultrasound, electro-optical, radiographic, LIDAR (light detection and ranging), etc. Sensing may include generating samples (e.g., pixels, audio signals) representing transmitted or otherwise arriving signals at the sensor.

[0169] The description and / or figures may depict a conceptual structure. The conceptual structure may include one or more clusters. Each cluster may include a signature and associated metadata. Each reference to one or more clusters may apply to a reference to the conceptual structure.

[0170] The specification and / or drawings may depict a processor. The processor may be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a fully custom integrated circuit, or the like, or a combination of these integrated circuits.

[0171] Any combination of any steps of any method illustrated in the specification and / or figures may be provided.

[0172] Any combination of subject matter of any of the claims may be provided.

[0173] Any combination of the systems, units, components, processors, sensors illustrated in the specification and / or drawings may be provided.

[0174] Any reference to an object may also apply to a pattern, and therefore any reference to object detection may also apply mutatis mutandis to pattern detection.

[0175] A situation may be a single location / characteristic combination at a point in time. A scenario is a sequence of events that follows logically within a causal frame of reference. Any reference to a scenario should apply mutatis mutandis to a situation.

[0176] The sensing information unit may be sensed by one or more sensors of one or more types, and the one or more sensors may belong to the same device or system or to different devices in the system.

Claims

1. A neural network group, a first layer portion, a second layer portion, and a third layer portion; the first layer portion is shared by a first neural network subgroup of the neural network group; the second layer portions may be shared by second neural network subgroups, such that different second layer portions are shared by different second neural network subgroups of the neural network group; the third layer portion may be shared by a third neural network subgroup, such that different third layer portions are shared by different third neural network subgroups of the neural network group; each neural network in the group of neural networks further includes a unique combination of layer portions; A neural network group characterized by:

2. each neural network of the neural network group includes at least one dedicated layer portion that is not shared by another neural network of the neural network group; 2. The neural network group of claim 1.

3. The neural network is trained across different narrow driving-related tasks.

2. The neural network group of claim 1.

4. the first layer portion is shared by all the neural networks in the neural network group; 2. The neural network group of claim 1.

5. At least one layer portion of the neural network of the neural network group is generated by clustering layers of neural networks that have been individually trained.

2. The neural network group of claim 1.

6. At least some of the neural networks of the group of neural networks are generated during a period of mutual training of the neural networks.

2. The neural network group of claim 1.

7. one of the neural networks in the neural network group is added to the neural network group after a subgroup of neural networks is formed, the subgroup of neural networks including a portion of the neural networks in the neural network group; 2. The neural network group of claim 1.

8. 1. A method for generating a neural network group, comprising: determining a first layer portion shared by a first neural network subgroup of the neural network group; determining second layer portions shared by second neural network subgroups such that different second layer portions are shared by different second neural network subgroups of the neural network group; and determining third layer portions shared by third neural network subgroups such that different third layer portions are shared by different third neural network subgroups of the neural network group; each neural network in the group of neural networks further includes a unique combination of layer portions; A method for generating a neural network group, comprising:

9. determining connectivity between successive layer portions to determine layer portions that may be shared by neural network subgroups based on the connectivity; 9. The method for generating a neural network group according to claim 8.

10. The connectivity is based on partial connectivity between a layer portion of a specified neural network layer and a layer portion of another neural network layer.

10. The method for generating a neural network group according to claim 9.

11. determining a number of layer portions for each layer based on the input constraints; 9. The method for generating a neural network group according to claim 8.

12. Each neural network's unique combination of layers is tailored to a narrow range of driving-related tasks.

9. The method for generating a neural network group according to claim 8.

13. 1. A non-transitory computer-readable medium for generating a neural network group, comprising: The non-transitory computer-readable medium includes a method for determining a first layer portion shared by a first neural network subgroup of the neural network group; determining second layer portions shared by second neural network subgroups such that different second layer portions are shared by different second neural network subgroups of the neural network group; and and storing instructions executable by a processor for determining third layer portions shared by third neural network subgroups, wherein different third layer portions are shared by different third neural network subgroups of the neural network group; each neural network in the group of neural networks further includes a unique combination of layer portions; 10. A non-transitory computer-readable medium for generating a neural network group, comprising:

Images