Method and device for processing data

Abstract

The present disclosure relates to a method for processing data, a method for training a neural network, a device for processing data, an electronic device, and a computer-readable storage medium. The method comprises: obtaining shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers; determining the parameters of each of the multiple neural network layers based at least in part on the shared parameters and the private parameters; and determining the output vector of the stack based on the parameters of each of the multiple neural network layers and the input vector input to the neural network layer. The present disclosure replaces the parameters of one or more original neural networks in the stack with shared parameters and private parameters to reduce memory usage and improve inference speed while maintaining the accuracy of the neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence services, and more specifically to a method for processing data, a device for processing data, an electronic device, and a computer-readable storage medium. Background Art

[0002] In the field of artificial intelligence, research on neural network models with large-scale parameters has made gratifying progress, and it has broad prospects in application fields such as natural language processing, computer vision, and machine translation. However, neural network models with large-scale parameters usually have billions to trillions of parameters, resulting in high training costs, inefficient inference speed, and complex deployment processes.

[0003] In order to overcome the above defects, a variety of methods have been proposed to compress the parameter size of neural network models. These methods mainly involve parameter pruning, parameter quantization, parameter sharing, low-rank decomposition, etc. Although these methods can effectively reduce the number of model parameters, they may also cause performance degradation.

[0004] Therefore, it is necessary to improve the parameter compression method of the neural network model to reduce the number of parameters while maintaining the model performance, thereby achieving a more efficient and feasible application of large-scale parameter neural network models. Summary of the invention

[0005] The embodiments of the present disclosure provide a method for processing data, a method for training a neural network, an apparatus for processing data, an electronic device, and a computer-readable storage medium.

[0006] An embodiment of the present disclosure provides a method for processing data, the method comprising: obtaining shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers; determining parameters of each of the multiple neural network layers based at least in part on the shared parameters and the private parameters; and determining an output vector of the stack based on the parameters of each of the multiple neural network layers and an input vector input to the neural network layer, wherein the shared parameters include at least one of a dynamic shared parameter and a static shared parameter, the private parameters include at least one of a dynamic private parameter and a static private parameter, and for each of the multiple neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0007] An embodiment of the present disclosure provides a method for processing data, the method comprising: determining static parameters corresponding to a first neural network layer based on static shared parameters shared by multiple neural network layers in a stack and static private parameters private to a first neural network layer among the multiple neural network layers; determining dynamic shared parameters shared by the multiple neural network layers and dynamic private parameters private to the first neural network layer based on an input vector input to the first neural network layer; determining dynamic parameters corresponding to the first neural network layer based on the dynamic shared parameters and the dynamic private parameters; determining an output vector of the first neural network layer based on the input vector, static parameters and dynamic parameters of the first neural network layer; and determining the presence of a detected object based at least in part on the output vector.

[0008] An embodiment of the present disclosure provides a method for training a neural network, wherein the neural network includes a stack, and the stack includes a plurality of stacked neural network layers. The method includes: determining an input vector input to each of the plurality of neural network layers and an output vector output by the neural network based on training data and the neural network in training; and adjusting shared parameters shared by the plurality of neural network layers in the stack and private parameters private to each of the plurality of neural network layers using the input vector of each neural network layer and the output vector of the neural network output, until the output vector of the neural network output meets a preset condition; wherein the shared parameters include at least one of a dynamic shared parameter and a static shared parameter, and the private parameters include at least one of a dynamic private parameter and a static private parameter, and for each of the plurality of neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0009] An embodiment of the present disclosure provides a device for processing data, the device comprising: an acquisition module, configured to acquire shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers; a determination module, configured to determine parameters of each of the multiple neural network layers based at least in part on the shared parameters and the private parameters; and an inference module, configured to determine an output vector of the stack based on the parameters of each of the multiple neural network layers and an input vector input to the neural network layer; wherein the shared parameters include at least one of a dynamic shared parameter and a static shared parameter, the private parameters include at least one of a dynamic private parameter and a static private parameter, and for each of the multiple neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0010] An embodiment of the present disclosure discloses an electronic device, including: one or more processors; and one or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the above method is performed.

[0011] An embodiment of the present disclosure provides a computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a processor, the above method is implemented.

[0012] According to another aspect of the present disclosure, a computer program product or a computer program is provided, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various aspects or various optional implementations of the above-mentioned various aspects.

[0013] The disclosed embodiment replaces the original parameters of one or more neural networks in the stack with shared parameters and private parameters to reduce memory usage and improve reasoning speed while maintaining the accuracy of the neural network. Thus, the overall parameter usage is reduced while maintaining the performance of the neural network. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solution of the embodiment of the present disclosure, the following briefly introduces the drawings required for describing the embodiment. The drawings described below are only exemplary embodiments of the present disclosure.

[0015] Figure 1 is an example schematic diagram showing a scenario according to an embodiment of the present disclosure.

[0016] Figure 2 A schematic diagram showing a stack in an exemplary neural network.

[0017] FIG. 3A to FIG. 3C A schematic diagram of an improved neural network model according to an embodiment of the present disclosure is shown.

[0018] Figure 4A A flowchart of a method 40 for processing data according to an embodiment of the present disclosure is shown. FIG. 3A to FIG. 3C The improved neural network model in is used for inference.

[0019] Figure 4B Pseudo code for “statically” calculating parameters of a neural network layer according to an embodiment of the present disclosure is shown.

[0020] Figure 4CPseudo code for “dynamically” calculating parameters of a neural network layer according to an embodiment of the present disclosure is shown.

[0021] Figure 5 A flowchart of a method for training a neural network according to an embodiment of the present disclosure is shown.

[0022] Figure 6 Another schematic diagram of a neural network model according to an embodiment of the present disclosure is shown.

[0023] Figure 7 Graphs showing test results according to embodiments of the present disclosure.

[0024] Figure 8 A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0025] Fig. 9 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0026] Fig.10 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present disclosure more obvious, the exemplary embodiments according to the present disclosure will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described here.

[0028] In this specification and the accompanying drawings, operations and elements having substantially the same or similar features are denoted by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. Meanwhile, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the descriptions, and cannot be understood as indicating or implying relative importance or ranking.

[0029] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics and other technologies. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0032] Machine Learning (ML) is a multi-disciplinary interdisciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning by teaching. The pre-trained model is the latest development of deep learning, which integrates the above technologies.

[0033] After pre-training and fine-tuning, the artificial intelligence model disclosed in the present invention shows a level of understanding close to that of humans, and can still achieve the performance level of large-scale neural networks, especially when data is limited, while improving reasoning efficiency and reducing deployment costs.

[0034] Optionally, the models that can be used in the embodiments of the present disclosure below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, the neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating outputs in the output layer. The nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layer, activation layer, pooling layer, convolution layer, etc. The input and output of the neural network model can take various forms, and the present disclosure is not limited to this.

[0035] The solutions provided in the embodiments of the present disclosure involve technologies such as artificial intelligence and / or machine learning, which are specifically described through the following embodiments.

[0036] First, refer to Figure 1 The application scenarios of the method for processing data and the corresponding device according to the embodiments of the present disclosure are described. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically shown.

[0037] The neural network model of the embodiment of the present disclosure can be integrated into various electronic devices, for example, Figure 1 The server 110 and any electronic device in the multiple terminals 120. For example, the neural network model can be integrated in the terminal 120. The terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC, Personal Computer), a smart speaker or a smart watch, etc., but is not limited to this. For another example, the neural network model can also be integrated in the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this disclosure.

[0038] It is understandable that the device for reasoning using the neural network model of the embodiment of the present disclosure can be a terminal, a server, or a system composed of a terminal and a server. The method for processing data of the embodiment of the present disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0039] The artificial intelligence model provided by the embodiments of the present disclosure may also involve artificial intelligence cloud services in the field of cloud technology. Among them, cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and network in a wide area network or a local area network to realize the calculation, storage, processing, and sharing of data. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the application of cloud computing business model, which can form a resource pool, which is used on demand and is flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites, drug research websites, and more portal websites. With the high development and application of the Internet industry, each item may have its own identification mark in the future, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately, and all kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0040] It is worth noting that the terminal 110 and the server 120 according to the embodiment of the present disclosure comply with the data protection principle, respect the user's data rights, and protect the user's data security and privacy. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will clearly inform the user of the purpose, method and scope of collecting, using, storing, transmitting and deleting the user's data, and obtain the user's consent. According to the embodiment of the present disclosure, the terminal 110 and the server 120 will take reasonable technical and management measures to prevent the user's data from being leaked, tampered with, damaged or lost. According to the embodiment of the present disclosure, the terminal 110 and the server 120 provider will regularly review and update the user's data and delete expired or useless data in a timely manner. In addition, the cloud service provider using the embodiment of the present disclosure respects the user's rights to access, correct, delete, withdraw consent, complain and claim data, and provides convenient channels and procedures so that users can effectively exercise these rights.

[0041] Furthermore, the process of using artificial intelligence technology to analyze data in the terminal 120 or the server 110 is based on the principles of legality, rationality and transparency. The data collected and processed by the artificial intelligence model according to the embodiment of the present disclosure is relevant, necessary and appropriate for the prediction purpose and does not contain any personal identity information or sensitive information. The neural network model according to the embodiment of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent the data from being accessed, used or leaked without authorization.

[0042] The artificial intelligence-based neural network model according to the embodiments of the present disclosure will comply with relevant data protection regulations and ethical principles. The neural network model is trained based on a large amount of anonymized and de-identified data and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also been rigorously tested and evaluated to ensure that its output results are accurate and reliable and will not cause any misleading or discrimination. The artificial intelligence model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. In addition, the neural network model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.

[0043] In the current era of booming large neural network models (hereinafter referred to as large models or large models), the model parameters of neural network models are increasing significantly, which brings challenges to training, inference and deployment. Various methods have been proposed to reduce the number of model parameters, such as knowledge distillation, parameter pruning, parameter quantization, parameter sharing, low-rank decomposition, etc. The knowledge distillation scheme compresses the large model into a smaller model to maintain performance as much as possible, but it may lead to a reduction in model capacity. The pruning scheme removes redundant parameters from the large model, but this may affect the stability of the model. The quantization scheme considers reducing the numerical precision of the model parameters to reduce storage and computing requirements, but it may lead to a decrease in model accuracy. The parameter sharing scheme reduces the number of parameters by sharing parameters between different layers, but this scheme may limit the expressiveness of the model. The low-rank decomposition scheme decomposes the parameter matrix into multiple smaller matrices to reduce the number of parameters, but it may also lead to performance loss.

[0044] In order to maintain the performance of large models while reducing the number of parameters, further improvements need to be made to the neural network model. In the currently widely used neural network models, a large number of stacked neural network layers are used, and these stacked neural network layers can also be called stacks. A stack can generally refer to the superposition of multiple modules of the same or similar structure in any number or configuration. For example, common convolutional neural networks (CNNs), recurrent neural networks (RNNs), neural networks based on self-attention mechanisms, etc. may have stacks.

[0045] Each stack is usually composed of neural network layers of the same or similar structure, which are stacked together in sequence. By stacking multiple neural network layers, the neural network model can learn more complex feature representations and improve the expressiveness of the neural network model. For example, in a neural network model based on a transformer, both the encoder and the decoder contain stacks of multiple layers of the same structure, forming a stacked structure. This design helps capture different levels of abstract features of the input data and improve the performance of the neural network model.

[0046] However, the widespread use of stacks has also led to a significant increase in the number of parameters. Based on this, the disclosed embodiment adopts shared parameter modules and private parameter modules to replace one or more original neural network layers in the stack to reduce memory usage and improve reasoning speed while maintaining the accuracy of the neural network model. Such improvements are intended to reduce the overall parameter usage while maintaining the performance of the neural network model.

[0047] The embodiment of the present disclosure proposes an improved neural network model by improving the stack structurally. The improved neural network model includes: a shared parameter module and a private parameter module. In these two modules, the shared parameter module stores the shared parameters shared by each neural network layer in the stack, while the private parameter module stores the private parameters private to each neural network layer. Among them, the private parameters can be divided into two types: static private parameters and dynamic private parameters. Static private parameters refer to private parameters that do not change with the input vector. And dynamic private parameters refer to private parameters that change with the input vector, so dynamic private parameters can also be called "adaptively" changeable parameters. Similarly, shared parameters can also be divided into two types: static shared parameters and dynamic shared parameters. Static shared parameters refer to shared parameters that do not change with the input vector. And dynamic shared parameters refer to shared parameters that change with the input vector. Static shared parameters, dynamic shared parameters, static private parameters and dynamic private parameters are all parameters that can be learned during the training process. Of course, the present disclosure is not limited to this.

[0048] For the improved neural network model, the embodiment of the present disclosure provides a method for processing data to perform reasoning based on such an improved neural network model. The method includes: obtaining shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers; determining the parameters of each of the multiple neural network layers based at least in part on the shared parameters and the private parameters; and determining the output vector of the stack based on the parameters of each of the multiple neural network layers and the input vector input to the neural network layer, wherein the shared parameters include at least one of a dynamic shared parameter and a static shared parameter, the private parameters include at least one of a dynamic private parameter and a static private parameter, and for each of the multiple neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0049] In the process of reasoning using the improved neural network, the complete parameter data corresponding to each neural network layer can be restored based on the shared parameters stored in the shared parameter module and the private parameters stored in the private parameter module, thereby ensuring the reasoning performance of the neural network model. Since the shared parameters of multiple neural network layers are extracted and stored in the shared parameter module, the number of parameters of the neural network model can be greatly reduced, and the storage space occupied will also be greatly reduced.

[0050] Similarly, for the improved neural network model, the embodiment of the present disclosure also provides a method for training a neural network to train the improved neural network model. The neural network includes a stack, and the stack includes multiple neural network layers stacked. The method for training a neural network includes: based on the training data and the neural network in training, determining the input vector input to each of the multiple neural network layers and the output vector output by the neural network; and using the input vector of each neural network layer and the output vector of the neural network output, adjusting the shared parameters shared by the multiple neural network layers in the stack and the private parameters private to each of the multiple neural network layers in the stack until the output vector of the neural network output meets the preset conditions; wherein the shared parameters include at least one of the dynamic shared parameters and the static shared parameters, and the private parameters include at least one of the dynamic private parameters and the static private parameters, and for each of the multiple neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0051] In the process of training the improved neural network model, since only shared parameters and private parameters need to be trained, the amount of parameters that need to be trained is greatly reduced.

[0052] It can be seen that the embodiments of the present disclosure help solve the problem of too many parameters in large models, improve memory usage efficiency and reasoning speed, while maintaining model performance.

[0053] Refer to the following Figures 2 to 10 A method for processing data, a method for training a neural network, and an apparatus are described according to embodiments of the present disclosure.

[0054] Figure 2 A schematic diagram showing a stack in an exemplary neural network.

[0055] Currently, many neural network models partially or extensively adopt stacking in their architecture to improve their performance. For example, the CNN-based neural network model uses stacking as its main architecture to adapt to the characteristics of various computer vision tasks (such as classification, detection, segmentation, etc.). The main body of the multi-layer perceptron (MLP) neural network (for example, MLP-Mixer) is also composed of dozens of identical stacked mixing layers (Mixer). In addition, the transformer (Transformer) neural network, which is widely used in various fields (including computer vision, natural language processing, multimodal learning, etc.), also has the following characteristics: Figure 2 The encoder or decoder shown is composed of a stack, wherein the encoder has a plurality of stacked encoding layers, and the corresponding decoder has a plurality of stacked decoding layers. Each encoding layer / decoding layer is structurally the same, but may have different parameters. The embodiments of the present disclosure may be used for Figure 2 The stacked coding layers or the stacked decoding layers shown are improved. Figure 2 However, those skilled in the art should understand that the embodiments of the present disclosure can improve stacks of any structure.

[0056] Specifically, the disclosed embodiments effectively reduce the overall parameter usage of the encoder or decoder by splitting the encoder or decoder into a shared parameter module and a private parameter module, making the improved encoder or decoder more concise and efficient. The current encoder or decoder includes multiple neural network layers (e.g., decoding layers or encoding layers) that are similar in structure and function, and there are certain parameter commonalities between these neural network layers. However, since different neural network layers may process different input and output distributions, the parameters of each neural network layer in the encoder or decoder are also unique. Therefore, by combining shared parameter modules and private parameter modules, the disclosed embodiments aim to fully express the commonality of shared parameters while retaining the uniqueness of the private parameters of each layer in the stack.

[0057] To further illustrate the wide adaptability of the embodiments of the present disclosure, two other example neural network models with stacks are given below: a query object-based detection neural network model and a query-based detector model. The internal stack structure modules in these two example neural networks can also be improved using the embodiments of the present disclosure.

[0058] Among them, the query object-based detection neural network model is suitable for the task of detecting specific objects in images or videos. Unlike traditional detectors, the query object-based detection neural network model uses learnable query features to interact with image feature maps. This interaction process can be formalized as formula (1) through the attention function.

[0059] Q updated =Attention(Q,K(V)) (1)

[0060] where Q, K, and V represent query features, key features, and value features, respectively. By using a decoder with stacked decoding layers and / or attention layers of the same structure (the architecture of the decoder is similar to Figure 2 The query object-based detection neural network model gradually refines the query feature Q by interacting with the key features and value features. The final learned query feature Q can be used for object category prediction. However, the number of parameters of the decoder with a stacked decoding layer and an attention layer may be huge. According to an embodiment of the present disclosure, such a decoder can be improved by using a shared parameter module and a private parameter module. This can greatly reduce the number of parameters required for the query object-based detection neural network model.

[0061] The query-based detector model (e.g., AdaMixer) uses the adaptive channel mixing (ACM) method and the adaptive spatial mixing (ASM) method to improve the performance of the detector. There are also multiple stacked decoding layers inside the query-based detector model. Its working principle can be briefly described as follows: given the sampled features Where C = d feat / g, g represents the number of sampling groups. The sampled features x are obtained by the group sampling operation, which converts the feature space channel d feat Divide into g groups, and perform separate sampling on each group. Then, the query-based detector model uses the ACM method to transform the sampled feature x in the channel dimension based on the weight matrix that dynamically changes based on the query feature Q to improve the representation ability of the channel semantic feature. The process of using ACM can be expressed by formulas (2) to (3).

[0062]

[0063] ACM(x) = ReLU(LayerNorm(xM c )) (3)

[0064] Among them, LayerNorm represents the regularization of the layer.

[0065] Next, the query-based detector model uses the ASM method to enhance the adaptability of the query feature Q to the spatial structure of the sampled features by applying an adaptive transformation to the spatial dimension of the sampled features.

[0066] The process of using ASM can be expressed by equations (4) to (5).

[0067]

[0068] ASM(x) = ReLU(LayerNorm(x) T M c )) (5)

[0069] ACM and ASM train independent parameters for each sampling group, and the final output features Through the linear layer L outpu Flatten and convert to d q The query feature Q of the dimension. The ACM process, the ASM process and the output linear transformation process all require the use of a large number of learnable parameters, and these parameters are all located in the decoder stack (that is, in the stacked decoding layer). According to the implementation rate of the present disclosure, these three processes can be improved through a shared parameter module and a private parameter module to significantly reduce the number of parameters in the query-based detector model.

[0070] Specifically, the embodiment of the present disclosure improves the traditional query-based detector model. The method for performing the task of reasoning about target detection using the improved query-based detector model may include: determining static parameters corresponding to the first neural network layer based on static shared parameters shared by multiple neural network layers in the stack and static private parameters private to the first neural network layer among the multiple neural network layers; determining dynamic shared parameters shared by the multiple neural network layers and dynamic private parameters private to the first neural network layer based on the input vector input to the first neural network layer; determining dynamic parameters corresponding to the first neural network layer based on the dynamic shared parameters and the dynamic private parameters; determining the output vector of the first neural network layer based on the input vector, static parameters and dynamic parameters of the first neural network layer; and determining the presence of the detected object based at least in part on the output vector. Compared with traditional solutions, the embodiment of the present disclosure greatly reduces the amount of parameters.

[0071] FIG. 3A to FIG. 3CA schematic diagram of an improved neural network model according to an embodiment of the present disclosure is shown.

[0072] FIG. 3A to FIG. 3C The improved neural network model according to the embodiment of the present disclosure can be replaced by a shared parameter module and a private parameter module. Figure 2 The stack shown. FIG. 3A to FIG. 3C As shown. It can be restored through the shared parameters in the shared parameter module and the private parameters in the private parameter module. Figure 2 Each neural network layer in the stack described in . The shared parameter module stores the shared parameters shared by each neural network layer. The private parameter module stores the private parameters private to each neural network layer.

[0073] Assume that there are N neural network layers stacked in a stack, and the structure of each neural network layer is the same. Depending on the design of the neural network model, the neural network layers in the stack can be restored using only static parameters, only dynamic parameters, or both static and dynamic parameters. Static parameters refer to parameters that do not change with the input vector, while dynamic private parameters refer to parameters that change with the input vector. The following describes these three cases separately.

[0074] like Figure 3A As shown, for the solution of restoring the neural network layer using only static parameters, the parameters of each neural network layer can be associated only with the static shared parameters and the static private parameters. That is, the parameters of each neural network layer can be solved by a function with both the static shared parameters and the static private parameters as independent variables. For example, the parameters of each neural network layer can be the linear sum of the static shared parameters and the static private parameters. Of course, the present disclosure is not limited to this.

[0075] For example, the static shared parameters can be mathematically represented by the static shared parameter matrix W shared The static shared parameter matrix W is expressed in the form of shared The dimension is d×h. Each neural network layer shares the static shared parameter matrix W shared .exist Figure 3A In, d=in_dim, h=out_dim.

[0076] Static private parameters can also be expressed in the form of a static private parameter matrix. For example, for the i-th neural network layer, its static private parameter matrix can be expressed as W i private To indicate. i private It can be calculated by formula (6).

[0077]

[0078] Among them, the first low-rank matrix First projection matrix At the same time, r<<d,h. k is an integer greater than 0 and less than or equal to K. B ik The rank of is r, which captures W shared The main patterns and information in A ik As the first projection matrix, it transforms B ik Restore to W i private By changing W i private Split into B ik and A ik The product of and only train B during training ik and A ik , which can further reduce the amount of parameters to be trained.

[0079] K is used to calculate W i private That is, to restore a neural network layer, K first low-rank matrices and K first projection matrices are required. Specifically, for accuracy considerations, the static private parameter matrix W i private Can be designed as K B ik A ik The larger the number of K, the higher the precision of the static private parameter matrix, and the smaller the number of K, the lower the precision of the static private parameter matrix.

[0080] If the parameter matrix W of a neural network layer i It can be restored only by the static private parameter matrix and the static shared parameter matrix, so the parameter matrix W i It can be calculated by formula (7).

[0081] W i =W shared +W i private (7)

[0082] The method of calculating the static private parameter matrix based on formula (6)-formula (7) is also called the method of "statically" calculating the private parameters. Of course, the present disclosure is not limited to this.

[0083] like Figure 3B As shown, for the scheme of restoring the neural network layer using only dynamic parameters, the parameters of each neural network layer can be a function of the dynamic shared parameters and the dynamic private parameters. For example, it can be a linear sum of the dynamic shared parameters and the dynamic private parameters.

[0084] For example, the dynamic shared parameter can be mathematically represented by a dynamic shared parameter matrix The dimension of the dynamic shared parameter matrix is Each neural network layer shares this dynamic shared parameter matrix The dynamic shared parameters can be adaptively changed with the input vector q. In a specific embodiment of the present disclosure, the dynamic shared parameter matrix is ​​a linear function with the input vector q as an independent variable, which linearly expands the input vector q into a d×h matrix. At this time, the dynamic shared parameter matrix can be calculated by formula (8):

[0085]

[0086] That is, for each of the multiple neural network layers, a first linear transformation is performed on the input vector q input to the neural network layer to determine the dynamic shared parameter.

[0087] Similarly, dynamic private parameters refer to private parameters that change with the input vector q, so dynamic private parameters can also be called "adaptively" changeable parameters. Dynamic private parameters can also be expressed in the form of a dynamic private parameter matrix. For example, for the i-th neural network layer, its dynamic private parameter matrix can be expressed as To express. It can be calculated by formula (9).

[0088]

[0089] Among them, the second low-rank matrix Second projection matrix At the same time, r<<d,h. k' is an integer greater than 0 and less than K'. Feature matrix is a linear function with the input vector q as the independent variable. Optionally, It can be calculated using formula (10).

[0090]

[0091] That is, for each of the plurality of neural network layers, a second linear transformation is performed on the input vector q input to the neural network layer, and the feature matrix corresponding to the neural network layer can be determined. Similarly, K′ is used to calculate Specifically, for accuracy reasons, the dynamic private parameter matrix Can be designed as K′ The larger the number of K′ is, the higher the precision of the dynamic private parameter matrix is, and the smaller the number of K′ is, the lower the precision of the dynamic private parameter matrix is.

[0092] If the parameter matrix of a neural network layer It can be restored only by the dynamic private parameter matrix and the dynamic shared parameter matrix, so the parameter matrix of the restored neural network layer is It can be calculated by formula (11).

[0093]

[0094] For the scheme of restoring the neural network layer using only dynamic parameters, the parameters of each neural network layer can be a function of the dynamic shared parameters and the dynamic private parameters. For example, it can be a linear sum of the dynamic shared parameters and the dynamic private parameters.

[0095] The method of calculating the dynamic private parameter matrix based on formula (8) to formula (11) is also called the method of "dynamically" calculating the private parameters. Of course, the present disclosure is not limited to this.

[0096] For the scheme of restoring the neural network layer using both static parameters and dynamic parameters, the parameters of each neural network layer Then it can be the parameter matrix obtained above and W i For example, Can be a parameter matrix and W i The weighted sum of and W i The product of , the present disclosure is not limited to this.

[0097] Assume that in each neural network layer in the stack of unmodified neural networks there is For the solution of restoring the neural network layer using only static parameters, each neural network layer contains dxh parameters. Figure 3A For the solution of restoring the neural network layer using only dynamic parameters, the linear transformation of the input vector q requires d q ×d×h parameters, where d q is the dimension of the input vector q. Therefore, Figure 3B The improvement is to reduce the dxh parameters to K×(d q × 2 +d×r+r×h) parameters. It can be seen that no matter which scheme is used to restore the parameters of the neural network layer, the purpose of greatly reducing the number of neural network parameters while maintaining the performance of the neural network can be achieved.

[0098] Based on this, Figure 4A A flowchart of a method 40 for processing data according to an embodiment of the present disclosure is shown. FIG. 3A to FIG. 3C The improved neural network model in is used for inference. Figure 4B Pseudo code for “statically” calculating parameters of a neural network layer according to an embodiment of the present disclosure is shown. Figure 4C Pseudo code for “dynamically” calculating parameters of a neural network layer according to an embodiment of the present disclosure is shown.

[0099] Method 40 can be performed on a terminal device or a server (such as Figure 1 The method 40 includes the following operations S401 to S403. Of course, the method 40 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0100] In operation S401, shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers are obtained.

[0101] In operation S402, parameters of each of the plurality of neural network layers are determined based at least in part on the shared parameters and the private parameters.

[0102] In operation S403, an output vector of the stack is determined based on a parameter of each of the plurality of neural network layers and an input vector input to the neural network layer.

[0103] As described above, a stack is formed by stacking neural network layers of the same or similar structure, and these neural network layers are stacked in a certain order. This "ordered" stacking method can be expressed as: the output vector of the previous neural network layer is used as the input vector of the next neural network layer; or the output vectors of the first m neural network layers are used as the input vectors of the next m neural network layers (m is an arbitrary real number); or when the same input vector is input, the output vector is calculated sequentially through different calculation orders. The present disclosure is not limited to this.

[0104] Optionally, the shared parameters shared by multiple neural network layers in the stack reflect the commonality between the parameters of the neural network layers. Specifically, the shared parameters represent the common characteristics between these neural network layers, which can be regarded as an abstract representation of the commonality of these neural network layers. Among them, the shared parameters include at least one of dynamic shared parameters and static shared parameters. For each of the multiple neural network layers, the dynamic shared parameters are associated with the input vector input to the neural network layer. During the reasoning process, the low-rank matrix obtained through training is used to adjust the input vector input to the neural network layer, thereby obtaining the value of the dynamic shared parameters. In contrast, the static shared parameters are fixed after training and will not be adaptively changed due to the input vector input to the neural network layer. This means that in the reasoning stage, by using the input vector to flexibly adjust the dynamic shared parameters, the adaptability of the common characteristics of each neural network layer in the stack to real-time input data is reflected, while the static shared parameters maintain their fixed values ​​obtained during the training process and are not affected by the input data during reasoning. Of course, the present disclosure is not limited to this.

[0105] Optionally, the private parameters of each neural network layer reflect the uniqueness of the parameters of that specific neural network layer. Although the structures of multiple neural network layers in the stack are the same or similar, different neural network layers may process input vectors with different distributions, or attempt to extract different information from the input vectors. Therefore, the embodiments of the present disclosure restore the parameters of the neural network layer by superimposing the private parameters of each neural network layer with the shared parameters, thereby ensuring the performance and accuracy of the neural network model. Similarly, the private parameters include at least one of a dynamic private parameter and a static private parameter. Of course, the present disclosure is not limited to this.

[0106] For each of the multiple neural network layers, the dynamic private parameters are associated with the input vector input to the neural network layer. During the reasoning process, the low-rank matrix obtained through training is used to adjust the input vector input to the neural network layer, thereby obtaining the value of the dynamic private parameters. In contrast, the static private parameters are fixed after training and will not adaptively change the input vector input to the neural network layer. This means that in the reasoning stage, the dynamic private parameters are flexibly adjusted by using the input vector, which reflects the adaptability of the unique characteristics of each neural network layer in the stack to real-time input data, while the static private parameters maintain their fixed values ​​obtained during the training process and are not affected by the input data during reasoning. Of course, the present disclosure is not limited to this.

[0107] Optionally, the low-rank matrix and private parameters involved in the shared parameters are determined in the process of training the neural network model, and will be stored in the shared parameter module and the private parameter module respectively, so as to facilitate calling these parameters during reasoning and thus improve the reasoning speed. Of course, the present disclosure is not limited to this.

[0108] In the scheme of restoring the parameters of the neural network layer using only static parameters (i.e., static shared parameters and static private parameters), continue with Figure 3A For example, combined with Figure 4B Operation S402 will be described in detail. Figure 4B The pseudo code in briefly illustrates the process of restoring the neural network parameters.

[0109] Optionally, in a scheme where only static parameters are used to restore the parameters of the neural network layer, first in operation S401, based on a first low-rank matrix (B ik ) and the first projection matrix (A ik ), determine the static private parameters of at least one neural network layer in the plurality of neural network layers. Then, in operation S402, based on the static shared parameters (W shared ) and the static private parameter (W i private ), determine parameters of at least one neural network layer among the multiple neural network layers.

[0110] Optionally, the number of the first low-rank matrices and the first projection matrices is K, where K is an integer greater than 1, and determining the static private parameters of at least one of the multiple neural network layers includes: calculating the product of the kth first low-rank matrix and the kth first projection matrix, where k is an integer greater than 0 and less than or equal to K; and accumulating the products of all first low-rank matrices and the first projection matrices to determine the static private parameters of at least one of the multiple neural network layers.

[0111] exist Figure 4B In the process of determining the parameters of each neural network layer in the plurality of neural network layers in operation S402, the non_dynamic_fused_weight(W_shared,A,B) function as defined therein is called. In this function, the input parameters are the static shared parameters W_shared (dimension is (1,d,h)), the first projection matrix A (dimension is (G,r,h)), and the first low-rank matrix B (dimension is (G,d,r)), and the output is the parameters W of the restored neural network layer. G represents the number of groups (i.e., the reference Figure 3A Described for calculating W i private The number of groups K).

[0112] Specifically, W_shared represents a static shared parameter, A and B represent references, Figure 3A The two matrices involved in the static private parameters described (i.e., the low-rank matrix B ik and the projection matrix A ik ), where the dimensions of A are (G, r, h) and the dimensions of B are (G, d, r). By performing matrix multiplication, we can obtain W_private (which is the W described above). i private ), whose dimension is (d, h). Then, the static shared parameters W_shared and the static private weights W_private can be combined by concatenation to obtain the parameter matrix W of the neural network layer. i (i.e. Figure 4B The parameter matrix W of the neural network layer i The grouped dimensions (G = K) are summed in the combined weights, and the final dimension is (d, h). This operation ensures that in the process of restoring the neural network layer, the parameter matrix W of the neural network layer used in the inference phase is obtained by combining the static shared parameters and the static private parameters. i .

[0113] In the scheme of restoring the parameters of the neural network layer using only dynamic parameters (i.e., dynamic shared parameters and dynamic private parameters), continue with Figure 3B For example, combined with Figure 4C Operation S402 will be described in detail. Figure 4C The pseudo code in briefly illustrates the process of restoring the neural network parameters.

[0114] Optionally, in a scheme in which only dynamic parameters are used to restore the parameters of a neural network layer. First, in operation S401, for each of the multiple neural network layers, a linear transformation is performed on the input vector input to the neural network layer to determine the dynamic shared parameters. Then, in operation S402, for each of the multiple neural network layers, a linear transformation is performed on the input vector input to the neural network layer to obtain a feature matrix corresponding to the neural network layer; based on the feature matrix, a second low-rank matrix private to the neural network layer, and a second projection matrix, the dynamic private parameters of at least one neural network layer among the multiple neural network layers are determined. Then, based on the dynamic shared parameters and the dynamic private parameters, the parameters of at least one neural network layer among the multiple neural network layers are determined.

[0115] Optionally, the number of the second low-rank matrices and second projection matrices is K′, K′ is an integer greater than 1, and the determining of the dynamic private parameters of at least one of the multiple neural network layers includes: calculating the product of the k′th second low-rank matrix, the feature matrix and the k′th second projection matrix, k′ is an integer greater than 0 and less than K′; and accumulating the products of all second low-rank matrices, feature matrices and second projection matrices to determine the dynamic private parameters of at least one of the multiple neural network layers. Of course, the present disclosure is not limited to this.

[0116] exist Figure 4C In the example, by calling the function dynamic_fused_weight(W_shared,A,B) as defined therein, the process of determining the parameters of each of the plurality of neural network layers in operation S402 can be implemented. The function "dynamic_fused_weight" adopts a dynamic fusion strategy to adapt to the change of the input vector. First, the input vector q (i.e., pro_feats) is processed by the first linear transformation operation "linear1" to obtain the dynamic shared parameters (“shared_weight”). Next, the input vector q (i.e. Figure 4C pro_feats) in to obtain the feature matrix for dynamic private parameter calculation Then, through matrix multiplication operation, and Combine to get the dynamic private parameter matrix Finally, the dynamic shared parameters and the dynamic private parameter matrix Concatenate and sum in a specific dimension to obtain the fused dynamic parameters as the parameter matrix of the neural network layer used in the inference phase (i.e. Figure 4C of fusedweight).

[0117] For the scheme of restoring the neural network layer using both static parameters and dynamic parameters, the parameters of each neural network layer Then it can be the parameter matrix obtained above and W i For example, Can be a parameter matrix and W i The weighted sum of and W i The product of . Figure 6An example of using both static parameters and dynamic parameters to restore the parameters of a neural network layer is further described. The present disclosure is not limited thereto.

[0118] The disclosed embodiment aims to reduce static parameters and dynamic parameters (ie, adaptive parameters) by splitting the stack to reduce the parameters in the deep learning model. The disclosed embodiment allows parameter sharing of most neural network layers in the stack, while retaining only a small number of private parameters for each neural network layer, thereby significantly reducing the total number of parameters without affecting performance. Through multiple experiments on widely used datasets, the disclosed embodiment was verified in the target detection task, and the results showed that even when the decoder parameters were reduced by 50% to 70%, the model performance was still excellent. It has been shown through multiple experiments that the reasoning method provided by the disclosed embodiment can be applied to more tasks (e.g., natural language processing), different models (e.g., language models), and other neural network components (e.g., backbone networks).

[0119] Figure 5 A flowchart of a method 50 for training a neural network according to an embodiment of the present disclosure is shown, which can be used to train the improved neural network model shown in FIG. 3 .

[0120] Method 50 can be performed on a terminal device or a server (such as Figure 1 The method 50 includes the following operations S501 to S502. Of course, the method 50 may also include more or fewer operations, and the present disclosure is not limited thereto.

[0121] In operation S501, based on training data and the neural network in training, an input vector input to each of the plurality of neural network layers and an output vector output by the neural network are determined.

[0122] In operation S502, the shared parameters shared by multiple neural network layers in the stack and the private parameters private to each neural network layer in the multiple neural network layers are adjusted using the input vector of each neural network layer and the output vector of the neural network output until the output vector of the neural network output meets a preset condition.

[0123] The shared parameter includes at least one of a dynamic shared parameter and a static shared parameter, the private parameter includes at least one of a dynamic private parameter and a static private parameter, and for each of the plurality of neural network layers, the dynamic shared parameter and the dynamic private parameter are associated with an input vector input to the neural network layer. Of course, the present disclosure is not limited thereto.

[0124] Optionally, the preset condition can be understood as a stop condition for neural network training. Figure 2 Taking the Transformer neural network used for English-French translation as an example, the preset conditions may include reaching a pre-set number of training rounds, the loss function converges to a certain threshold, or the performance of the validation set is no longer significantly improved. At this time, the performance of the output vector output by the neural network has basically reached an extreme value. For the target query network, the preset conditions may be that the neural network has achieved the expected performance level on a specific task, or that the neural network parameters have not changed significantly within a certain period of time, or that the loss function converges to a certain threshold. The present disclosure is not limited to this.

[0125] In the process of training the improved neural network model, since only shared parameters and private parameters need to be trained, the amount of parameters that need to be trained is greatly reduced.

[0126] Will refer to Figure 6 An example of improving the query-based detector model (AdaMixer) based on the above-described scheme is further described. In this example, both static parameters and dynamic parameters are used to restore the neural network layer.

[0127] Figure 6 Another schematic diagram of a neural network model according to an embodiment of the present disclosure is shown, which illustrates the calculation process during reasoning within a neural network layer in the stack.

[0128] Figure 6 A method for processing data is shown, which is used to improve the traditional AdaMixer to detect whether a specific object exists in image data or video data using a smaller number of parameters. The method can be briefly described as: based on the static shared parameters shared by multiple neural network layers in the stack and the static private parameters private to the first neural network layer among the multiple neural network layers, determine the static parameters corresponding to the first neural network layer; based on the input vector input to the first neural network layer, determine the dynamic shared parameters shared by the multiple neural network layers and the dynamic private parameters private to the first neural network layer; based on the dynamic shared parameters and the dynamic private parameters, determine the dynamic parameters corresponding to the first neural network layer; based on the input vector, static parameters and dynamic parameters of the first neural network layer, determine the output vector of the first neural network layer; and based at least in part on the output vector, determine the presence of the detected object.

[0129] The above calculation process uses the synergy between static shared parameters, static private parameters, dynamic shared parameters, and dynamic private parameters to input into the neural network layer stack. i The input vector q i Computed as a stack of neural network layersi The output vector q is the output i+1 .

[0130] like Figure 6 As shown, first, the first low-rank matrix B in the upper left dashed box is ik and the first projection matrix A ik Computing neural network layer stack i Static private parameters of Then, from the storage with static shared parameters W shared =W shared In the shared parameter module, get W shared , and W private and W shared Perform element-wise addition of the matrix to obtain the static parameter W fused .

[0131] Next, based on the input to the neural network layer stack i The input vector q i , through Linear i Operation, get the feature matrix Then according to the formula Determine dynamic private parameters based on the input to the neural network layer stack i The input vector q i , through the "Linear Sharing" layer operationLinear share , get the dynamic shared parameters Will and Perform element-wise addition of matrices to obtain dynamic parameters

[0132] According to the principle of the query-based detector model (AdaMixer), it is necessary to first use the adaptive channel mixing (ACM) method and the adaptive spatial mixing (ASM) method to input to the neural network layer stack. i The input vector q i and for the neural network layer stack i The sampling characteristics x i That is, you can use dynamic parameters With the sampling feature x i The mixed eigenvector is obtained by matrix multiplication. The mixed eigenvector is multiplied by the static parameter to obtain the projected vector out i . For the projected vector out i and input to the neural network layer stack i The input vector q i Perform matrix addition to get the neural network layer stack iThe output vector q i+1 .

[0133] Will Figure 6 The experimental results of the improved query-based detector model shown in FIG. 1 are compared with the traditional query-based detector model. Figure 7 As shown. Taking a neural network layer with a dimension of d×h as an example, the embodiments of the present disclosure try different r values ​​to reduce parameters. The bold values ​​represent the default selection. Compared with the parameters of a specific traditional AdaMixer, the embodiments of the present disclosure use less than 10% of the parameters.

[0134] An embodiment of the present disclosure also provides a device for processing data, the device comprising: an acquisition module, configured to acquire shared parameters shared by multiple neural network layers in a stack and private parameters private to each of the multiple neural network layers; a determination module, configured to determine the parameters of each of the multiple neural network layers based at least in part on the shared parameters and the private parameters; and an inference module, configured to determine the output vector of the stack based on the parameters of each of the multiple neural network layers and an input vector input to the neural network layer; wherein the shared parameters include at least one of a dynamic shared parameter and a static shared parameter, the private parameters include at least one of a dynamic private parameter and a static private parameter, and for each of the multiple neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with the input vector input to the neural network layer.

[0135] According to yet another aspect of the present disclosure, an electronic device is provided for implementing the method according to the embodiment of the present disclosure. Figure 8 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0136] like Figure 8 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores a computer-readable code, and when the computer-readable code is run by the one or more processors 2010, the method described above may be executed.

[0137] The processor in the embodiments of the present disclosure may be an integrated circuit chip with signal processing capabilities. The above processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, operations and logic block diagrams in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc., which may be an X86 architecture or an ARM architecture.

[0138] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0139] For example, the method or device according to the embodiment of the present disclosure may also be implemented by Fig. 9 The architecture of the computing device 3000 shown in FIG. Fig. 9 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the method provided in the present disclosure and program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Fig. 9 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Fig. 9 One or more components of a computing device are shown.

[0140] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Fig.10 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0141] like Fig.10As shown, the computer storage medium 4020 stores computer readable instructions 4010. When the computer readable instructions 4010 are executed by the processor, the method according to the embodiment of the present disclosure described with reference to the above figures can be executed. The computer readable storage medium in the embodiment of the present disclosure can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. The non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous connection dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM). It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory. It should be noted that memory of the methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0142] The present disclosure also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method according to the embodiment of the present disclosure.

[0143] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment, or a part of a code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuits, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or methods described herein may be implemented in hardware, software, firmware, dedicated circuits or logic, general purpose hardware or controllers or other computing devices, or some combination thereof as non-limiting examples.

[0145] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. It should be understood by those skilled in the art that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure.

Claims

1. A method for processing image data or video data, the method comprising: Obtaining shared parameters shared by a plurality of neural network layers in a stack and private parameters private to each of the plurality of neural network layers; determining parameters for each of the plurality of neural network layers based at least in part on the shared parameters and the private parameters; as well as determining an output vector of the stack based on parameters of each neural network layer in the plurality of neural network layers and an input vector input to the neural network layer, in, The shared parameter includes at least one of a dynamic shared parameter and a static shared parameter, The private parameter includes at least one of a dynamic private parameter and a static private parameter, For each neural network layer in the plurality of neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with an input vector input to the neural network layer.

2. The method of claim 1, wherein: The obtaining of private parameters of each neural network layer in the plurality of neural network layers comprises: Based on the first low-rank matrix and the first projection matrix private to each neural network layer, a static private parameter of at least one neural network layer among the plurality of neural network layers is determined.

3. The method of claim 1, wherein: The shared parameters include static shared parameters and the private parameters include static private parameters, and determining the parameters of each of the plurality of neural network layers includes: Based on the static shared parameters and the static private parameters, parameters of at least one neural network layer among the plurality of neural network layers are determined.

4. The method of claim 2, wherein: The number of the first low-rank matrix and the first projection matrix is ​​K, K is an integer greater than 1, and determining the static private parameters of at least one neural network layer among the multiple neural network layers includes: Calculate the product of the kth first low-rank matrix and the kth first projection matrix, where k is an integer greater than 0 and less than or equal to K; and The products of all first low-rank matrices and the first projection matrix are accumulated to determine static private parameters of at least one neural network layer among the plurality of neural network layers.

5. The method of claim 1, wherein: The obtaining of shared parameters shared by multiple neural network layers in the stack includes: For each of the multiple neural network layers, a first linear transformation is performed on the input vector input to the neural network layer to determine the dynamic shared parameter.

6. The method of claim 1, wherein: The obtaining of private parameters of each neural network layer in the plurality of neural network layers comprises: For each of the plurality of neural network layers, performing a second linear transformation on the input vector input to the neural network layer to obtain a feature matrix corresponding to the neural network layer; Based on the feature matrix, the second low-rank matrix private to the neural network layer, and the second projection matrix, dynamic private parameters of at least one neural network layer among the multiple neural network layers are determined.

7. The method of claim 6, wherein: The number of the second low-rank matrix and the second projection matrix is ​​K′, K′ is an integer greater than 1, and determining the dynamic private parameters of at least one neural network layer among the multiple neural network layers includes: Calculate the product of the k'th second low-rank matrix, the feature matrix, and the k'th second projection matrix, where k' is an integer greater than 0 and less than K'; and The products of all second low-rank matrices, the feature matrix, and the second projection matrix are accumulated to determine a dynamic private parameter of at least one neural network layer in the plurality of neural network layers.

8. The method of claim 1, wherein: The shared parameters include dynamic shared parameters and the private parameters include dynamic private parameters, and determining the parameters of each of the plurality of neural network layers includes: Based on the dynamic shared parameters and the dynamic private parameters, parameters of at least one neural network layer among the plurality of neural network layers are determined.

9. A method for processing image data or video data, the method comprising: Determine static parameters corresponding to a first neural network layer in the plurality of neural network layers based on static shared parameters shared by the plurality of neural network layers in the stack and static private parameters private to the first neural network layer in the plurality of neural network layers; Determining, based on an input vector input to the first neural network layer, a dynamic shared parameter shared by the plurality of neural network layers and a dynamic private parameter private to the first neural network layer; Determining a dynamic parameter corresponding to the first neural network layer based on the dynamic shared parameter and the dynamic private parameter; Determining an output vector of the first neural network layer based on an input vector, a static parameter, and a dynamic parameter of the first neural network layer; as well as Based at least in part on the output vector, a presence of a detected object is determined in the image data or the video data.

10. A method for training a neural network, wherein: The neural network includes a stack, the stack includes a plurality of stacked neural network layers, and the method includes: Determining an input vector input to each of the plurality of neural network layers and an output vector output by the neural network based on training data and the neural network in training, wherein the training data is image data or video data; and Using the input vector of each neural network layer and the output vector of the neural network output, adjust the shared parameters shared by multiple neural network layers in the stack and the private parameters private to each neural network layer in the multiple neural network layers until the output vector of the neural network output meets the preset condition; in, The shared parameter includes at least one of a dynamic shared parameter and a static shared parameter, The private parameter includes at least one of a dynamic private parameter and a static private parameter, For each neural network layer in the plurality of neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with an input vector input to the neural network layer.

11. A device for processing image data or video data, the device comprising: An acquisition module is configured to: acquire shared parameters shared by multiple neural network layers in a stack and private parameters private to each neural network layer in the multiple neural network layers: A determination module configured to: determine a parameter of each neural network layer in the plurality of neural network layers based at least in part on the shared parameter and the private parameter; as well as an inference module configured to: determine an output vector of the stack based on a parameter of each neural network layer in the plurality of neural network layers and an input vector input to the neural network layer; in, The shared parameter includes at least one of a dynamic shared parameter and a static shared parameter, The private parameter includes at least one of a dynamic private parameter and a static private parameter, For each neural network layer in the plurality of neural network layers, the dynamic shared parameters and the dynamic private parameters are associated with an input vector input to the neural network layer.

12. An electronic device comprising: one or more processors; and One or more memories, wherein a computer executable program is stored in the memory, and when the computer executable program is executed by the processor, the method of any one of claims 1-10 is executed.

13. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions, when executed by a processor, implement the method according to any one of claims 1 to 10.

14. A computer program product, comprising computer instructions, wherein the computer instructions are stored in a computer-readable storage medium, and a processor of a computer device reads and executes the computer instructions from the computer-readable medium, so that the computer device implements the method according to any one of claims 1 to 10.

Images