Shared representation of neural network resources

Abstract

To provide a method for generating a neural network group.SOLUTION: The method includes determining a first layer portion shared by a first neural network subgroup of the neural network group, determining a second layer portion shared by a second neural network subgroup of the neural network group such that different second layer portions are shared by different second neural network subgroups of the neural network group, and determining a third layer portion shared by a third neural network subgroup of the neural network group such that different third layer portions are shared by different third neural network subgroups of the neural network group. Each neural network of the group of neural networks includes a unique combination of layer portions.SELECTED DRAWING: Figure 3

Description

Background Art

[0001] Neural networks are used for various purposes in vehicles, including classifying items sensed by sensors related to the vehicle and providing driving-related responses based on the item classifications.

[0002] Neural networks are expected to provide highly accurate responses in changing environments and, because of this very high likelihood, consume large amounts of resources.

[0003] There is an ever-growing need to reduce the resource consumption associated with the memory of neural networks and the execution of neural network-related processing.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure is a method, system, and non-transitory computer-readable medium.

Brief Description of the Drawings

[0005] The embodiments of the present disclosure can be more comprehensively understood and grasped by combining the following detailed description with the accompanying drawings. In the drawings, [Figure 1] is a diagram showing an example of a system. [Figure 2] is a diagram showing an example of a system. [Figure 3] is a diagram showing an example of a neural network group. [Figure 4] is a diagram showing an example of a stage of a neural network group generation process. [Figure 5] is a diagram showing an example of another stage of a neural network group generation process. [Figure 6] is a diagram showing an example of a neural network group after adding a new neural network. [Figure 7]This figure shows an example of the method. [Figure 8] Here is an example of the method. [Modes for carrying out the invention]

[0006] Different drawings illustrate examples of units and / or software and / or information items and / or steps and / or parts. These examples are provided for brevity of explanation. At least one of the units and / or software and / or information items and / or steps and / or parts may be optional or required.

[0007] The embodiment provides a neural network group, which includes a shared layer portion shared among different neural networks within the neural network group. The use of the shared layer portion can significantly reduce memory consumption (for example, by at least 1.1x, 1.2x, 1.5x, 2x, 2.5x, 3x, 3.5x, 4x, 4.5x, 5x, 10x, 20x, 50x, 100x, 200x, and even more).

[0008] In the embodiment, a neural network group is determined based on one or more input constraints. These input constraints are, for example, one or more dimensional parameters, which are, for example, the overall dimensions, the memory required to store the neural network group, the number of layers in the neural network group, the number of layer parts in each layer of the neural network group, the weights of the dimensions (bits assigned to the weights), and / or the bias and / or activation values.

[0009] In the examples, the neural network group exhibits one or more additional input constraints. These additional input constraints include, for example, the type of number (fixed-point or floating-point representation) and / or the type of activation function.

[0010] In the embodiment, one or more dimensional parameters of a neural network group are determined based on at least one rule, at least one model, at least one constraint, at least one optimization parameter, computational consumption associated with realizing the neural network group, memory consumption associated with realizing the neural network group, and a trade-off between one or more dimensional parameters of the neural network group and one or more performance parameters of the neural network group (e.g., accuracy, delay, false positive rate, false negative rate, true positive rate, true negative rate, etc.).

[0011] As demonstrated in the examples, the outcome of a trade-off is determined by applying a function to two or more factors, parameters, or values. This function may be linear, nonlinear, exponential, logarithmic, weighted sum, or the like.

[0012] As demonstrated by the examples, neural network groups can grow logarithmically rather than exponentially or multiplicatively (when using individually trained neural networks instead of neural network groups).

[0013] In this embodiment, a vehicle stores one or more neural network groups, and these neural network groups may be provided to one or more other neural network groups (for example, through vehicle-to-vehicle communication or vehicle-to-other-entity communication). The provision of neural network groups can be done dynamically so that the vehicle stores different neural network groups at different times.

[0014] In the embodiment, the dynamic delivery depends on one or more parameters. These parameters include, for example, the path the vehicle takes (different paths may better serve different neural network groups), the driver operating the vehicle (automatic or human, a specific driver or another specific driver), and / or any other scene parameters and / or context parameters and / or environmental parameters and / or safety parameters and / or comfort parameters that may influence the relevance (or suitability) of one or more neural network groups sent to the vehicle.

[0015] Additionally or alternatively, dynamic provisioning is based on static and / or dynamic constraints related to the vehicle, such as the current availability of memory resources and / or processing resources and / or communication resources (in-vehicle and / or external communication resources) and / or maximum capacity of memory resources and / or maximum capacity of processing resources and / or maximum capacity of communication resources (in-vehicle and / or external communication resources).

[0016] Additionally or alternatively, dynamic delivery is based on delay constraints. Delay constraints may be set by the driver, vehicle manufacturer, or another entity. By example, when faced with a certain scene and / or environmental conditions, and / or when the complexity of the environment and / or the level of danger associated with driving increases above a threshold, delay constraints can force a reduction in the delay of the neural network processing group. For example, in a more dangerous environment, and / or under conditions of low visibility, and / or when the driver is an inexperienced or accident-prone human driver, the delay should be lower.

[0017] In the example, the neural network group exhibits complete connectivity.

[0018] According to the embodiments, the neural network group exhibits only partial connectivity.

[0019] According to the embodiments, some layers of the neural network group are fully connected, and some other layers of the neural network group are only partially connected.

[0020] According to the embodiments, the neural network group has a tree structure.

[0021] According to the embodiments, the leaf layer part of the neural network group is unique in that the leaf layer part is included in a single neural network of the neural network group.

[0022] According to the embodiments, all neural networks in the neural network group have the same number of layers.

[0023] According to the embodiments, one or more neural networks of the neural network group are longer than one or more other neural networks of the neural network group.

[0024] According to the embodiments, different neural networks of the neural network group include different combinations of layer parts.

[0025] According to the embodiments, each neural network of the neural network group has a unique layer part, and this layer part is not included in any other neural network of the neural network group.

[0026] According to the embodiments, the neural network group includes four or more layers, where at least one neural network subgroup of the neural network group includes different combinations of parts of the first three layers of the neural network group.

[0027] In the embodiment, the neural network groups are at least 5, 10, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, 550, 600, 650, 700, 750, 800, 850, 900, 950, 1000, 1100, 1200, 1300, 1400, 1500, Includes 2000, 2500, 3000, 3500, 4000, 4500, 5000, 5500, 10000, 11000, 12000, 15000, 20000, 30000, 40000, 50000, 60000, 100000, 150000, 200000, and over 250000 neural networks.

[0028] The examples provide a group of neural networks that include the following: A. The first layer portion is shared by the first neural network subgroup of the neural network group. Depending on the embodiment, there may be multiple first layer portions. B. The second layer portion is shared by the second neural network subgroup, so that different second layer portions are shared by different second neural network subgroups within the neural network group. C. One or more additional layer portions are shared by one or more additional neural network subgroups, so that different neural network groups are shared by one or more different additional neural network subgroups of a neural network group.

[0029] In the embodiment, the neural network group may include three or more layers, for example, 4, 10, 15, 20, 25, 30, 40, 50, 60, 70, 100, 120, 150, 200, 250, 300, 400, 500, 1000, 1500, 2000 or more layers.

[0030] As demonstrated in the examples, each neural network in the neural network group includes a unique combination of layers.

[0031] In this embodiment, each neural network in a neural network group includes at least one dedicated layer portion that is not shared by other neural networks in the same neural network group.

[0032] In the examples, the neural network is trained on different narrow driving-related tasks.

[0033] In this embodiment, the first layer is shared by all neural networks in the neural network group.

[0034] In this embodiment, at least one layer portion of the neural network in a neural network group is generated by clustering layers of individually trained neural networks.

[0035] In various embodiments, clustering may include applying any clustering algorithm, for example, at least one of the following: AK-mean clustering: This method divides data into K clusters by minimizing the variance within each cluster. B. Hierarchical clustering: A tree structure of clusters is constructed by sequentially merging or splitting clusters. C. Agglomerative hierarchical clustering: Starting from individual points, these are joined together to form clusters. D. Partitioned hierarchical clustering: Start with the entire dataset and divide it into clusters. E. DBSCAN (Density-based spatial clustering for noisy applications): Groups tightly packed points and marks points in low-density regions as outliers. F.OPTICS (Identification of clustering structure by ordering points): An extension of DBSCAN that handles changes in density more efficiently. G. Mean-shift clustering: Clusters are identified by the modes arranged in the density of data points. H. Gaussian Mixture Model (GMM): Assumes that the data is generated from a mixture of several Gaussian distributions, and assigns a probability to each point belonging to a cluster. I. Spectral clustering: Reduce the dimensionality by using eigenvalues ​​of the similarity matrix in a lower dimension before clustering. J. Neighbor propagation: Messages are sent between points to identify samples, and clusters are identified based on these samples. K.Birch (Reducing clustering hierarchy with balancing iteration): Performs hierarchical clustering on large datasets using a combination of other clustering methods. L. Agglomerative clustering: Creates a cluster hierarchy by merging clusters based on the shortest distance between them. M. Self-organizing Map (SOM): An artificial neural network that is trained using unsupervised learning to generate a low-dimensional representation of data. N. Fuzzy C-mean (FCM): Allows each data point to belong to multiple clusters with different degrees of belonging. O.HDBSCAN (Hierarchical Density-Based Spatial Clustering of Noisy Applications): An extension of DBSCAN that discovers clusters of different densities.

[0036] Examples show that clustering may be performed with or without constraints on the number of clusters. It has been found that constraints can improve the accuracy and / or time of clustering.

[0037] In the embodiment, at least a portion of the neural networks in the neural network group are generated during the neural network mutual training period.

[0038] In this embodiment, one neural network in a neural network group is added to the neural network group after a subgroup of neural networks is formed that includes a portion of the neural networks in the neural network group.

[0039] As demonstrated in the examples, neural network groups are generated in one or more ways.

[0040] In the examples, the neural network group is modified in at least one way (for example, by adding a new neural network or by performing any other modifications).

[0041] The following are methods for generating and / or modifying neural network groups, as illustrated by the examples. A. Includes training. B. Includes retraining. C. Does not include retraining. D. Does not include training. E. First, train on another neural network (not included in the neural network group). F. Includes training other neural networks. G. Includes multiple training iterations. H. Includes training and merging. I. This includes using one or more cost functions. J. Includes multiple iterations in which one or more layers are modified while other layers remain unchanged during a single iteration.

[0042] An embodiment provides a method for generating a neural network group. The method for generating a neural network group includes determining a first layer portion shared by a first neural network subgroup of the neural network group, determining a second layer portion shared by a second neural network subgroup so that different second layer portions are shared by different second neural network subgroups of the neural network group, and determining a third layer portion shared by a third neural network subgroup so that different third layer portions are shared by different third neural network subgroups of the neural network group, wherein each neural network in the neural network group further includes a unique combination of layer portions.

[0043] In one embodiment, the method includes determining the connectivity between consecutive layer portions, thereby determining which layer portions can be shared by neural network subgroups, and this determination is based on the determined connectivity.

[0044] In the embodiment, the determined connectivity is based on the partial connectivity between a specified neural network layer and another neural network layer.

[0045] In one embodiment, the method includes determining the number of layer portions of each layer based on input constraints. As demonstrated by the examples, the unique combination of layers in each neural network limits the task to a narrow range of driving-related tasks.

[0046] In this embodiment, a series of layers from the first layer to the dedicated layer form a neural network related to driving-related tasks.

[0047] In one embodiment, the method includes adding a new neural network to a group of neural networks.

[0048] In the embodiment, the addition is based on weight fitting between the weights of the new neural network layer and the weights of the layer portion of the neural network group.

[0049] In the example, the addition involves retraining a new neural network based on the neural network group.

[0050] In this example, determining the layer portion of a neural network group is based on individually trained initial neural networks.

[0051] In the embodiment, the decision involves clustering the layers of individually trained initial neural networks and determining the shared layer portion by merging the layers of the clusters corresponding to the shared layer portion.

[0052] In the example, determining the layer portion of the neural network group is based on the results of multiple training and merge iterations.

[0053] In the examples, the neural network group includes neural networks that are different from end-to-end neural networks.

[0054] In the examples, the neural network group includes an end-to-end neural network.

[0055] An example of an end-to-end neural network is a neural network that receives input such as sensing information (or pre-processed sensing information) and outputs driving-related decisions (e.g., commands to perform driving-related operations or suggestions to perform driving-related decisions). Driving-related decisions may be autonomous driving decisions or advanced driver assistance system decisions, or any decisions that will set or modify vehicle propagation.

[0056] The neural network in a neural network group may be a narrow artificial intelligence agent. Narrow might mean that a narrow AI agent is responsible for managing only a small portion (e.g., less than 1 percent) of the scenes that a vehicle might encounter.

[0057] In the following text, reference numbers related to Figure 1 are shown without parentheses, and reference numbers related to Figure 2 are shown in parentheses. Any item related to a reference number without parentheses may be different from or equal to the corresponding item in parentheses.

[0058] Figures 1 and 2 show examples of vehicles 100 and 400, which include a communication system 130 (430), one or more memory and / or storage units 120 (420), and a processing system 124 (424) including a processor 126 (426). The communication system 130 (430), one or more memory and / or storage units 120 (420), and the processing system 124 may belong to a computerized system. The computerized system may be a server, a laptop computer, a desktop computer, or any other computer, and may include sensing units and / or controllers, and may communicate with sensing units and / or controllers.

[0059] In this embodiment, the vehicle 100(400) communicates with network 132(432) and one or more other remote computerized systems 134(434) that communicate with network 132(432). An example of a remote computerized system is a server or one or more computers that access a storage system, which stores items related to one or more parts of one or more neural network groups, and at least some of the neural networks are not currently stored in the vehicle.

[0060] In one embodiment, the communication system 130 (430) is configured to enable communication between one or more memory and / or storage units 120 (420) and / or sensing systems 110 (410) and / or additional units (i.e., communicating with a remote computerized system) and / or networks 132 (432).

[0061] Memory and / or storage unit 120 (420) is shown as for storing software. Any reference to the software should be modified as necessary to apply to code and / or firmware and / or instructions and / or commands, etc.

[0062] The processor 126(426) includes multiple processing units 126(1)-126(J)[426(1)-426(J)], where J is an integer greater than 1. Any reference to one unit or item should be modified to apply to multiple units or items. For example, any reference to a processor should be modified to apply to multiple processors, and any reference to a communication system 130(430) should be modified to apply to multiple communication systems.

[0063] In the embodiment, one or more memory and / or storage units 120(420) include one or more memory units, and each memory unit may include one or more memory groups.

[0064] In the embodiment, one or more memories and / or storage units 120(420) include volatile memory and / or non-volatile memory. One or more memories and / or storage units 120(420) may also be random access memory (RAM) and / or read-only memory (ROM).

[0065] In some embodiments, the non-volatile storage unit is a mass storage device that provides non-volatile storage for computer code, computer-readable instructions, data structures, program modules, and other data to a processor or any other unit of a vehicle. For example, but not limited to, the mass storage device may be a hard disk, a removable disk, a removable optical disk, a magnetic tape cartridge, or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD), or other optical storage device, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

[0066] Any content may be stored in any part of memory and / or storage units, or in any type of memory and / or storage unit.

[0067] In one embodiment, at least one memory unit stores at least one database, for example, any database known in the art, such as DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, etc.

[0068] The memory and / or storage unit 120 (420) is configured to store firmware and / or software, one or more operating systems, and data and metadata necessary to perform any method referred to in this disclosure.

[0069] Memory and / or storage unit 120 (420) is shown as for storing software. Any reference to the software should be modified as necessary to apply to code and / or firmware and / or instructions and / or commands, etc.

[0070] Various units and / or components communicate with each other using arbitrary communication elements and / or protocols. An example of a communication system is shown as 130(430). Other communication elements may be provided.

[0071] The communication system 130 (430) can communicate with the bus 136 (436). The bus representation includes one or more of several possible bus structures, such as a memory bus or memory controller, a peripheral bus, an accelerated graphics port, and a processor or local bus using any of the various bus architectures. For example, such architectures may include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Enhanced ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, Accelerated Graphics Port (AGP) bus, and Peripheral Component Interconnect (PCI), PCI-Express bus, Personal Computer Memory Card Industry Association (PCMCIA), Universal Serial Bus (USB), etc. This bus and all buses specified herein can also be implemented through wired or wireless network connections and each subsystem.

[0072] Network 132 (432) is located outside the vehicle and is used for communication between the vehicle and at least one remote computing system. For example, the remote computing system may be a personal computer, laptop computer, portable computer, server, router, network computer, peer device, or other public network node. The logical connection between the processor and any remote computing system can be achieved through a local area network (LAN) and a general-purpose wide area network (WAN). Such network connections can be achieved in wired and wireless environments through network adapters (which may belong to communication system 130 (430)). Such networking environments are common and typical in larger networks such as offices, enterprise-scale computer networks, intranets, and the internet.

[0073] It should be noted that at least a portion of the content indicated as being stored in one or more memory / storage units 120 (420) may be stored outside the vehicle. It should also be noted that the processor may evaluate signatures generated by multiple detectors.

[0074] Examples of signature generation and / or image cropping are provided in U.S. Patent Application No. 18 / 527,701, which is incorporated herein by reference.

[0075] In this embodiment, the memory and / or storage unit 120(420) stores at least one of the following: the operating system 194(494), information 191(491), metadata 192(492), and software 193(493).

[0076] Using this software, the processing system is configured to perform one or more of methods 200 and 500.

[0077] The vehicle 100 (400) further includes a sensing system 110 (410) and a control unit 125 (425).

[0078] The control unit 125 (425) can cooperate with an advanced driver assistance system (ADAS) control unit (e.g., the ADAS control unit 481 in Figure 2), an autonomous driving control unit 482 (e.g., the autonomous driving (AD) control unit 482 in Figure 2), and / or control other vehicle components (including a vehicle computer, e.g., the vehicle computer 421 in Figure 2), and communicate with other vehicle components.

[0079] The ADAS control unit 481 is configured to control ADAS operations.

[0080] The AD control unit 82 is configured to control the autonomous driving of the vehicle. The vehicle computer 421 is configured to control the operation of the vehicle, in particular the engine, transmission, and any other vehicle systems or components.

[0081] The vehicle computer 421 can communicate with the engine control module, transmission control module, drivetrain control module, and the like.

[0082] The sensing system 110 (410) may include an optical element, a sensing element group, a readout circuit, and an image signal processor. Following the optical element is a sensing element group, such as a sensing element line or a sensing element array forming a sensing element group. Following the sensing element group is a readout circuit, which reads out the detection signal generated by the sensing element group. The image signal processor is configured to perform initial processing of the detection signal, such as improving the quality of the detection information and performing noise reduction. The sensing system 110 (410) is configured to output one or more sensing information units (SIUs).

[0083] The control unit 125 (425) is configured to control the operation of the sensing system 110 (410) and / or one or more memory and / or storage units 120 (420) and / or one or more additional units (excluding the controller).

[0084] In this embodiment, one or more memory and / or storage units 120(420) include one or more memory units, and each memory unit may include one or more memory groups.

[0085] In the embodiment, one or more memories and / or storage units 120(420) include volatile memory and / or non-volatile memory. One or more memories and / or storage units 120(420) may also be random access memory (RAM) and / or read-only memory (ROM).

[0086] In some embodiments, the non-volatile storage unit is a mass storage device that provides non-volatile storage for computer code, computer-readable instructions, data structures, program modules, and other data to a processor or any other unit of a vehicle. For example, but not limited to, the mass storage device may be a hard disk, a removable disk, a removable optical disk, a magnetic tape cartridge, or other magnetic storage device, a flash memory card, a CD-ROM, a digital versatile disk (DVD), or other optical storage device, random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and the like.

[0087] For example, but not limited to, computer-readable media may include “computer storage media” and “communication media.” “Computer storage media” includes volatile and non-volatile media, removable and non-removable media, implemented by any method or technique for storing information such as computer-readable instructions, data structures, program modules, or other data. Exemplary computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical storage devices, magnetic tape cartridges, magnetic tapes, magnetic disk storage devices, or any other media used to store desired information and accessible by a computer.

[0088] Any content may be stored in any part of memory and / or storage units, or in any type of memory and / or storage unit.

[0089] In one embodiment, at least one memory unit stores at least one database, for example, any database known in the art, such as DB2®, Microsoft® Access, Microsoft® SQL Server, Oracle®, MySQL, PostgreSQL, etc.

[0090] Various units and / or components communicate with each other using arbitrary communication elements and / or protocols. An example of a communication system is shown as 430. Other communication elements can be provided.

[0091] Figure 2 shows one or more memory and / or storage units, including a cache memory 420A, which may be used to store the most relevant information and / or instructions and / or metadata, such as one or more shared layer portions (in particular the weights and / or biases and / or connectivity of the shared layer portions).

[0092] Figure 2 further illustrates one or more items (one item, several items, or all items) that may be stored in one or more memories and / or storage units, and these one or more items include: A. Neural network software 496, when executed by a processor, results in neural network processing. Neural network processing allows for the application of neural network groups to information. B. Neural network metadata 497 stores the weights and / or biases and / or connectivity of the neural network group, or the weights and / or biases and / or connectivity of at least some of the neural network group. The C.AD software 498 is configured to control AD ​​operations (for example, by instructing the AD control unit 482 to execute commands), propose AD operations to the AD control unit 482, or implement at least some functions of the AD control unit 482. The D.AD software 499 is configured to control AD ​​operations (for example, by instructing the AD control unit 491 to execute commands), propose AD operations to the AD control unit 491, or implement at least some functions of the AD control unit 491. E. The neural network management software 495 is configured to manage the memory of one or more layer portions of the neural network group in one or more memory / storage units 420, for example, these portions are stored in a cache or otherwise accessible by the processor 426.

[0093] Figure 3 shows an example of a neural network group 60 containing K neural networks 62-1 to 62-K, where K is an integer greater than 4.

[0094] The first layer includes a first number of N1 first layer parts LP(1,1)-LP(1,N1)70(1,1)-70(1,N1), a second number of N2 second layer parts LP(2,1)-LP(2,N2)70(2,1)-70(2,N2), a third number of N3 third layer parts LP(3,1)-LP(3,N3)70(3,1)-70(3,N3), and one or more additional layers. Each neural network includes a unique combination of layer parts, and each neural network includes a unique layer part, for example LP(U,1)-LP(U,K)70(U,1)-70(U,K), where U represents unique. Unique parts of different neural networks may belong to the same layer of the neural network group or to different layers of the neural network group.

[0095] Figures 4 and 5 show examples of one or more stages in the generation process of a neural network group.

[0096] In the embodiment, one or more generation processes include clustering layers of individually trained neural networks and generating layer portions (e.g., one layer portion for each cluster) based on the clusters to generate at least one layer portion of a neural network for a group of neural networks.

[0097] In the examples, one or more generation processes include generating at least one layer portion of a neural network during the mutual training period of the neural network.

[0098] Figure 4 shows multiple (J) individually trained neural networks 70-1 to 70-J. These neural networks can also be called initial neural networks because they are processed to generate a group of neural networks. The neural networks in the group of neural networks differ from the initial neural networks.

[0099] Each individually trained neural network includes an input layer {L(1,in)-L(J,in), 72(1,in)-72(J,in)}, one or more hidden layers {e.g., {L(1,hidden)-L(J,hidden), 72(1,hidden)-72(J,hidden)}, and an output layer {L(1,out)-L(J,out), 72(1,out)-72(J,out)}. The first cluster mapping 75 includes representations 77(1,in)-77(J,in) of the input layers L(1,in)-L(J,in) of J individually trained neural networks, and these representations are clustered to provide N1 clusters. In Figure 3, N1 is equal to 3, and is represented, for example, by three clusters 78(1), 78(2), and 78(3).

[0100] These representations may include vectors, embeddings, and signatures of embeddings. An example of an embedding is shown in U.S. Patent Application No. 18 / 527,701, which is incorporated herein by reference.

[0101] Figure 5 shows the second cluster mapping 80, which includes representations 82(1,hidden)-82(J,hidden) of the first hidden layers of J individually trained neural networks, and these representations are clustered to provide N2 clusters. In Figure 4, N2 is equal to 10, and is represented, for example, by 10 clusters 84(1)-84(10).

[0102] Figure 6 shows the modified neural network group 60-1 with the addition of a new neural network, which is added after the generation of neural network group 60 in Figure 3. The new neural network 62-L+1 includes a layer portion and a unique combination of the unique layer portion LP(U,K+1)70(U,K+1).

[0103] Figure 7 shows an example of a method 200 for generating neural network groups.

[0104] In the embodiment, method 200 begins with initialization step 210.

[0105] In some embodiments, the initialization step 210 includes receiving or determining one or more input constraints, such as size constraints or any other input constraints, where at least some input constraints are as shown above.

[0106] In some embodiments, step 210 includes determining multiple layer portions of each layer based on input constraints, and these input constraints are not limited to, for example, any of the aforementioned input constraints.

[0107] In one embodiment, step 210 includes determining the connectivity between consecutive layer portions, thereby determining which layer portions can be shared by neural network subgroups, and that this is based on the determined connectivity.

[0108] In the embodiment, the determined connectivity is based on the partial connectivity between a specified neural network layer and another neural network layer.

[0109] In this embodiment, connectivity determination is based on the connectivity of individually trained neural networks that are processed to generate a neural network group. For example, assuming that a first layer portion is generated based on a set of first layers of individually trained neural networks, the first layer portion is connected to one or more second layer portions, and these second layer portions are generated based on one or more sets of second layers of individually trained neural networks that are connected to the first layers in the set of first layers.

[0110] Refer to cluster 78(1) in Figure 4, which represents a collection of the first layers of individually trained neural networks.

[0111] The set of the first layer is connected to the set of the second layer, which is represented by the cluster set of the second cluster mapping 80 (for example, clusters 84(1), 84(5), and 84(9)).

[0112] The cluster set of the second cluster mapping (e.g., clusters 84(1), 84(5), 84(9)) is used to generate a set of three second layer parts of the neural network group, and the first layer part associated with cluster 78(1) is connected to the set of three second layer parts of the neural network group.

[0113] In the embodiment, the connectivity determination is performed during or between any of steps 220, 230, and 240.

[0114] In this embodiment, step 210 is followed by step 220, which determines the first layer portion shared by the first neural network subgroup of the neural network group. In this embodiment, there are multiple first layer portions.

[0115] In this embodiment, step 230 follows step 220, in which different second layer portions are shared by different second neural network subgroups of the neural network group, by determining which second layer portions can be shared by the second neural network subgroups.

[0116] In this embodiment, step 230 is followed by step 240, in which different third layer portions are shared by different third neural network subgroups of the neural network group, by determining which third layer portions can be shared by different third neural network subgroups of the neural network group.

[0117] In the embodiment, step 240 is followed by step 250, in which one or more additional layer portions are determined to be shared by one or more additional neural network subgroups, so that one or more different additional layer portions are shared by one or more different additional neural network subgroups of the neural network group.

[0118] In the embodiment, at least one of the decisions in steps 220, 230, 240, and 250 includes determining the shared layer portion by clustering the layers of the individually trained initial neural networks and merging the layers of the clusters corresponding to the shared layer portion.

[0119] In the examples, at least one of the decisions in steps 220, 230, 240, and 250 is based on the results of multiple training and merge iterations.

[0120] In some embodiments, merging involves averaging the weights and biases of the cluster layers. In other embodiments, merging involves applying a weighted sum or applying an arbitrary function to the weights and biases of the cluster layers.

[0121] In the embodiment, Method 200 provides a neural network group in which each neural network in the neural network group includes a unique combination of layer portions.

[0122] As demonstrated by the examples, the unique combination of layers used in each neural network limits the task to a narrow range of driving-related tasks.

[0123] In this embodiment, a series of layers from the first layer to the dedicated layer form a neural network related to driving-related tasks.

[0124] In one embodiment, method 200 includes a step 260 of updating a neural network group. For example, step 260 includes adding a new neural network to the neural network group.

[0125] In the embodiment, the addition is based on weight fitting between the weights of the new neural network layer and the weights of the layer portion of the neural network group.

[0126] In the example, the addition involves retraining a new neural network based on the neural network group.

[0127] Figure 8 shows an example of Method 500 for using neural network groups related to driving-related applications.

[0128] In the embodiment, method 500 begins with initialization step 510.

[0129] In some embodiments, the initialization step 510 includes acquiring a neural network group. Acquisition may include the vehicle receiving the neural network group from an external computerized system and / or loading the neural network group into the vehicle's memory unit. The memory unit may be a cache memory and / or may consist of one or more memory / memory units (or one or more memory / memory units). Acquisition may also include requesting the neural network group from a remote computerized system.

[0130] In this example, this neural network group includes the following: A. The first layer portion is shared by the first neural network subgroup of the neural network group. Multiple first layer portions may exist. B. The second layer portion can be shared by the second neural network subgroup, so that different second layer portions are shared by different second neural network subgroups within the neural network group. C. Different neural network groups are shared by different additional neural network subgroups of a neural network group, by one or more additional layer portions being shared by one or more additional neural network subgroups of a neural network group. Different third layer portions are shared by different third neural network subgroups of a neural network group, by one or more additional layer portions including a third layer portion, and the third layer portion being shared by a third neural network subgroup of a neural network group.

[0131] As demonstrated in the examples, each neural network in the neural network group includes a unique combination of layers.

[0132] In the examples, the acquisition is based on one or more input constraints, such as size constraints or any other input constraints, where at least some of the input constraints are shown above. Additionally or alternatively, the acquisition is based on the expected route of the vehicle, weather conditions, the availability of resources to run and / or store the neural network group, etc.

[0133] In the embodiment, step 510 is followed by step 520, in which a neural network group (or at least some neural networks within the neural network group) is applied to the input information to generate driving-related decisions and / or classification decisions and / or lane detection decisions, etc.

[0134] In the embodiment, if steps 510 and 520 are iterated multiple times and, for example, the vehicle reaches a different position than the position it was in during a previous iteration of step 520, a different neural network group can be obtained.

[0135] In this embodiment, a neural network group is part of a narrow collection of AI agents, and different AI agents are realized by different neural networks within the neural network group. In front of the neural network group, there may be a perceptual router that transmits perceptual information to the associated narrow AI agents.

[0136] In the embodiment, the set includes a large number (e.g., at least 50,000) neural networks implemented by a neural network group.

[0137] In the embodiment, relevant narrow AI agents (trained to perform tasks related to received perceptual information and / or vehicle tasks and / or vehicle routes) are activated, while at least some of the other narrow AI agents are not. The use of a shared network layer may require activating at least some of the irrelevant neural networks (related to the irrelevant narrow AI agents).

[0138] An example of a narrow set of AI agents is shown in U.S. Patent Application No. 18 / 036,150, which is incorporated herein by reference.

[0139] In the embodiment, this set uses the neural network group to reduce resource consumption and facilitate resource utilization at a relatively accessible level, thereby ensuring rapid activation with low latency.

[0140] As demonstrated in the example, a neural network group takes the form of a branched tree consisting of shared layers / representations between these networks, thus requiring less memory than the original structure of independent networks.

[0141] For example, assuming there should be 400,000 neural networks, a neural network group is provided, and the neural network group includes the following: A. Instead of 400,000 first layers, each containing 400,000 different neural networks, use a smaller first layer portion of N1 (e.g., 3). B. Instead of 400,000 second layers, each containing 400,000 different neural networks, use a smaller second layer portion with a second number N² (e.g., 10). C. Instead of 400,000 third layers, each containing 400,000 different neural networks, use a smaller number of third layers, N3 (e.g., 100).

[0142] A neural network group can increase the number of neural networks by using a logarithmically growing number of shared layers rather than an exponentially or multiplicatively growing number of layers.

[0143] The example demonstrates how to construct neural network groups during the generation process, taking into account shared representations across different layers of the neural network, using kernels / filters / weights similar to shared representations.

[0144] In the embodiment, even when different neural networks are assigned to different skills or tasks, this group of neural networks can be operated as a liquid neural network, which may include selectively activated and deactivated neural networks.

[0145] The embodiment increases the activation of each layer portion (especially when selectively activating (only or mostly) correlated neural networks and decreasing them in order of layers), and / or when each layer has more layer portions. For example, the first layer portion is activated for 50% of the neural network group's execution time, the second layer for 10% of its execution time, the third layer portion for 1% of its execution time, and so on.

[0146] As demonstrated in the example, generating neural network groups may be based on multi-task training, since different neural networks within the neural network group are assigned to different tasks.

[0147] In some embodiments, the generation of neural network groups may include, for example, weight sharing during multitask training. This not only improves performance by utilizing domain-specific information contained in the training signals of related tasks, but also allows for more efficient parameterization of the model. In some embodiments, weight sharing or merging may utilize weight sharing techniques such as “Multitask learning”, Caruana, R., Machine learning volume 28, pages 41-75(1997), which are incorporated herein by reference.

[0148] In this embodiment, neural network groups are generated using a two-step method: (a) independently training M independent networks (M narrow agents), and (b) combining the earlier stages of the networks to have a shared representation for effective execution time. This can be achieved by finding similar learning weights among the parallel narrow networks.

[0149] In the examples, neural network groups are generated using neural network distillation. An example of neural network distillation is shown in "Distilling the Knowledge in a Neural Network" by Geoffrey Hinton, Oriol Vinyals, and Jeff Dean (2015): https: / / arxiv.org / abs / 1503.02531, which is incorporated herein by reference.

[0150] In this example, a group of neural networks is trained using an end-to-end method, which includes training the group of neural networks with a shared architecture from the beginning. The agents learn their respective narrow tasks while being constrained to share weights for effective learning.

[0151] In this example, a group of neural networks is trained using a hypernetwork method, which includes using a scene-based hypernetwork to predict scene-specific weights. In this case, some layers are fixed and shared among all agents, while separate hypernetworks predict scene-specific weights.

[0152] The example suggests that the initial network layer portion acquires low-level information and is likely to be shared among (more) different agents.

[0153] Examples show that, as described above by network distillation and information similarity between different networks, a selection of layers to be shared after training is performed, as referenced by the methods described above, and is incorporated herein by reference, for example, with reference to the following sentences. "Towards Understanding Learning Representations: To What Extent Do Different Neural Networks Learn the Same Representation" by Maximillian Schmidt, Marvin Giese, and Eric J. Wederman (2018) by A. Maximillian Schmidt, Marvin Giese, and Eric J. Wederman. "Similarity of Neural Network Representations Revisited" by Romain Lopez-Roig, Janis Lizarraguirre-Rodriguez, and Armand Joubert (2019): https: / / arxiv.org / pdf / 2305.06329.

[0154] In the embodiment, if a set of agents is provided, the hyper-networking method is used in conjunction with a routing mechanism.

[0155] The examples show that sharing of layers occurs in narrow scenarios, but not at the functional unit level.

[0156] As demonstrated by the examples, the proposal and / or generation of neural network groups are unknown to the training of the initially trained neural network, and the initially trained neural network may be trained separately. Cross-network training may be autonomous and independent or alternative.

[0157] Any combination of any step of any method shown in this disclosure is also provided.

[0158] In the above detailed description, numerous specific details are given to provide a complete understanding of the present invention. However, as will be understood by those skilled in the art, the present invention may be carried out without these specific details. In other cases, known methods, procedures and components are not described in detail so as not to obscure the present invention.

[0159] The subject matter relating to the present invention is specifically pointed out and clearly claimed in the concluding section of the specification. However, the organization and method of operation of the present invention, as well as its purpose, features and advantages, may be best understood by referring to the following detailed description, when read in conjunction with the accompanying drawings.

[0160] It should be understood that the elements shown in the drawings are not necessarily drawn to scale for the sake of simplicity and clarity. For example, the dimensions of some elements may be exaggerated to make them clearer in relation to others. Also, where appropriate, reference numbers may be repeated in the drawings to indicate corresponding or similar elements.

[0161] Since the illustrated embodiments of the present invention can be realized using mostly electronic components and circuits known to those skilled in the art, it is not necessary to provide a more detailed explanation than shown above in order to understand the basic concepts of the present invention and to avoid confusing or distracting from the teachings of the present invention.

[0162] Any reference to a method in this specification should be applied mutatis mutandis to a device or system capable of performing that method and / or to a non-temporary computer-readable medium storing instructions for performing that method.

[0163] Any reference in this specification to a system or device should be applied mutatis mutandis to the methods that can be performed by the system and / or to non-temporary computer-readable media that store instructions that can be performed by the system.

[0164] Any reference in this specification to non-temporary computer-readable media should be applied mutatis mutandis to a device or system capable of executing instructions stored on non-temporary computer-readable media, and / or to a method of executing such instructions.

[0165] Any combination of any module or unit described in any part of the drawings, specification, and / or any claim may be provided.

[0166] The conversion module, active learning module, clustering module, or any other module described herein may be implemented by code, instructions, and / or commands stored in hardware and / or non-temporary computer-readable media, and may be contained within a vehicle, outside a vehicle, within a mobile device, within a server, etc.

[0167] The vehicle may be any type of vehicle, such as a ground transport vehicle, an aircraft, or a ship.

[0168] The specification and / or drawings may be images. Images are an example of sensing information. Any reference to images may be applied to any kind of natural signal, such as signals generated by nature, signals representing human behavior, signals representing operations related to stock markets, medical signals, financial series, geodetic signals, geophysical, chemical, molecular, text and numerical signals, and time series. Any reference to media units may be applied to sensing information. Sensing information may be of any kind and may be perceived by any kind of sensor, such as a visible light camera, audio sensor, infrared-sensing sensor, radar image, ultrasound, electro-optics, radiography, and LIDAR (light detection and ranging). Sensing may include generating samples (e.g., pixels, audio signals) that represent transmitted signals or other signals that reach the sensor.

[0169] The description and / or figures may illustrate a conceptual structure. The conceptual structure may include one or more clusters. Each cluster may include a signature and associated metadata. Each reference to one or more clusters may apply to a reference to the conceptual structure.

[0170] The specification and / or drawings may show a processor. The processor may also be a processing circuit. The processing circuit may be implemented as a central processing unit (CPU) and / or one or more other integrated circuits, such as application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), fully custom integrated circuits, or a combination thereof.

[0171] Any combination of any step of any method shown in the specification and / or figures may be provided.

[0172] Any combination of any subject matter of any claim may be provided.

[0173] Any combination of systems, units, components, processors, and sensors shown in the specification and / or drawings may be provided.

[0174] Any reference to an object may be applied to a pattern. Therefore, any reference to object detection may be applied mutatis mutandis to pattern detection.

[0175] A situation may be a single combination of location / characteristics at a given point in time. A scenario is a series of events that logically follow each other within a causal framework. Any reference to a scenario should be applied mutatis mutandis to a situation.

[0176] The sensing information unit may be sensed by one or more sensors of one or more types. The one or more sensors may belong to the same device or system, or to different devices in the system.

Claims

1. A system comprising a neural network group, It includes the first layer, the second layer, and the third layer. The first layer portion is shared by the first neural network subgroup of the neural network group, The second layer portion can be shared by a second neural network subgroup, so that different second layer portions are shared by different second neural network subgroups of the neural network group. The third layer portion can be shared by a third neural network subgroup, so that different third layer portions are shared by different third neural network subgroups of the neural network group. Each neural network in the aforementioned neural network group further includes a unique combination of layer portions. A system characterized by the following features.

2. Each neural network in the neural network group includes at least one dedicated layer portion that is not shared by other neural networks in the neural network group. The system according to feature 1.

3. The aforementioned neural network is trained across different narrow driving-related tasks. The system according to feature 1.

4. The first layer portion is shared by all neural networks in the neural network group. The system according to feature 1.

5. At least one layer portion of the neural network in the aforementioned neural network group is generated by clustering layers of individually trained neural networks. The system according to feature 1.

6. At least a portion of the neural networks in the aforementioned group of neural networks are generated during the neural network mutual training period. The system according to feature 1.

7. One of the neural networks in the neural network group is added to the neural network group after a subgroup of neural networks has been formed, and the subgroup of neural networks includes a portion of the neural networks in the neural network group. The system according to feature 1.

8. A method for generating a neural network group performed by a computer, To determine the first layer portion shared by the first neural network subgroup of the aforementioned neural network group, By determining the second layer portion shared by the second neural network subgroup, different portions of the second layer are shared by different second neural network subgroups of the neural network group. This includes determining which third layer portions are shared by third neural network subgroups, such that different third layer portions are shared by different third neural network subgroups of the neural network group. Each neural network in the aforementioned neural network group further includes a unique combination of layer portions. A method for generating a neural network group characterized by the following:

9. This further includes determining connectivity between consecutive layer portions, thereby determining which layer portions can be shared by neural network subgroups based on said connectivity. The method for generating a neural network group according to feature 8.

10. The aforementioned connectivity is based on partial connectivity between a specified neural network layer and another neural network layer. The method for generating a neural network group according to feature 9.

11. This further includes determining the number of layer parts for each layer based on input constraints. The method for generating a neural network group according to feature 8.

12. The unique combination of layers in each neural network limits it to narrow, driving-related tasks. The method for generating a neural network group according to feature 8.

13. A non-temporary computer-readable medium for generating neural network groups, The non-temporary computer-readable medium determines the first layer portion shared by the first neural network subgroup of the neural network group, By determining the second layer portion shared by the second neural network subgroup, different portions of the second layer are shared by different second neural network subgroups of the neural network group. By determining the third layer portion shared by the third neural network subgroup, the instructions used to be shared by different third neural network subgroups of the neural network group and executed by the processor are stored. Each neural network in the aforementioned neural network group further includes a unique combination of layer portions. A non-temporary, computer-readable medium for generating neural network groups characterized by the following:

Images