Processing apparatus and processing method

Abstract

The present disclosure relates to a processing device, a processing method, a first processing apparatus, a second processing apparatus, an electronic device and a computer readable storage medium. The processing device provided by the present disclosure comprises a first processing apparatus and at least one second processing apparatus, wherein the at least one second processing apparatus is arranged separately from the first processing apparatus in a physical manner and is communicatively coupled with the first processing apparatus, wherein the first processing apparatus is configured to: calculate a first vector corresponding to first input information based on model parameters of at least part of a neural network model and the first input information; and calculate a second vector corresponding to second input information based on model parameters of at least part of the neural network model and the second input information; and wherein the second processing apparatus is configured to: calculate a third vector based on the first vector and the second vector.

Description

Technical Field

[0001] This disclosure relates to the fields of artificial intelligence and chips, and more specifically to a processing device, processing method, first processing apparatus, second processing apparatus, electronic device and computer-readable storage medium. Background Technology

[0002] In recent years, artificial intelligence technology has developed rapidly, with AI algorithms based on transformer networks becoming the mainstream architecture. Transformers were first applied to natural language processing tasks, and thanks to their efficient self-attention mechanism and feed-forward network structure, they have quickly gained widespread application in various tasks. Currently, many AI systems, especially generative AI applications such as text generation, image generation, and code generation, are built around transformers as their core algorithms. These applications rely on the stacking and combination of multiple transformer modules to achieve deeper information modeling and the ability to process complex tasks.

[0003] Generative AI, as an important branch of current artificial intelligence, focuses on generating new content based on existing data. Compared to traditional discriminative models, generative models emphasize the creativity and coherence of content, placing higher demands on the implementation of underlying algorithms.

[0004] Transformer models have become a key algorithmic framework in generative AI. However, these models generally suffer from extremely large parameter counts and high computational resource consumption. Especially as the model scales up, the number of transformer modules surges, leading to a significant increase in inference time, difficulty in deployment, and inability to adapt to real-time applications. Summary of the Invention

[0005] This disclosure provides a processing device, a processing method, a first processing apparatus, a second processing apparatus, an electronic device, and a computer-readable storage medium.

[0006] This disclosure provides a processing device, including a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from the first processing unit and communicatively coupled thereto. The first processing unit is configured to: calculate a first vector corresponding to the first input information based on at least a portion of model parameters of a neural network model and first input information; and calculate a second vector corresponding to the second input information based on at least a portion of model parameters of the neural network model and second input information. The second processing unit is configured to: calculate a third vector based on the first vector and the second vector.

[0007] This disclosure provides a processing method executed by a processing device, wherein the processing device includes a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from the first processing unit but communicatively coupled thereto. The processing method includes: calculating a first vector corresponding to the first input information by the first processing unit based on at least a portion of model parameters of a neural network model and first input information; calculating a second vector corresponding to the second input information by the first processing unit based on at least a portion of model parameters of the neural network model and second input information; and calculating a third vector by the second processing unit based on the first vector and the second vector.

[0008] This disclosure provides a first processing device, including a first memory, at least one first processor chip, and a first transceiver. The first memory is configured to store model parameters of at least a portion of the neural network model. The first processor chip is configured to calculate a first vector corresponding to the first input information based on the model parameters of at least a portion of the neural network model and first input information; and to calculate a second vector corresponding to second input information based on the model parameters of at least a portion of the neural network model and second input information. The first transceiver is configured to send the first vector and the second vector to a second processing device, wherein the second processing device is physically separated from the first processing device but communicatively coupled to it.

[0009] This disclosure provides a second processing apparatus, which includes a second memory, a second processor, and a second transceiver. The second transceiver is configured to receive a first vector and a second vector from a first processing apparatus as described above. The second processor is configured to calculate a third vector based on the first vector and the second vector. The second memory is configured to store the third vector. The second transceiver is further configured to send the third vector to the first processing apparatus.

[0010] This disclosure provides an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer-executable program, and when the processor executes the computer-executable program, the above-described apparatus is executed.

[0011] This disclosure provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-described apparatus.

[0012] According to another aspect of this disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the means provided in the foregoing aspects or various alternative implementations of the foregoing aspects.

[0013] This disclosure embodiment achieves an efficient distributed computing architecture by splitting and deploying the neural network computation process onto two physically separate physical devices. Different physical devices are used to perform different types of computational tasks, thereby reducing the computational pressure on a single device and improving the device's scalability and deployment capability on low- and mid-range devices.

[0014] In one optional aspect of the embodiments of this disclosure, at least one second processing device is physically separated from the first processing device and coupled to it via a communication interface. The first processing device processes only context-independent computations, while context-dependent computations are processed only in the second processing device, thereby decoupling context-independent and context-dependent computations. This split deployment reduces the storage and computing requirements of a single device and enhances the modular scalability of the device.

[0015] In one optional aspect of the embodiments of this disclosure, the first processing device only performs context-independent computational tasks and cannot access or deduce semantic relationships between data, thus preventing the acquisition or disclosure of user privacy information. The second processing device, which handles context-dependent computations, can be deployed only on the user's local device or a privacy device, and sensitive user data may not be uploaded to the cloud, reducing the risk of data leakage and thereby protecting user privacy.

[0016] In another optional aspect of the embodiments of this disclosure, the calculation of the first vector and the second vector depends only on some parameters of the neural network model and input information, without involving context-dependent vector or token association calculations. For example, when processing natural language tasks, the first processing device calculates independent feature vectors (such as word embedding vectors) for each input token based on the feedforward network part, embedding network part, or projection matrix part of the converter model, without considering the contextual relationships between tokens.

[0017] In another optional aspect of the embodiments of this disclosure, the calculation of the third vector is based solely on the first and second vectors, without involving model parameters, focusing instead on context-dependent computation. For example, in a multi-head attention mechanism, the second processing device calculates an attention score based on the first and second vectors to generate a context-dependent third vector. This approach deploys attention mechanism-dependent computations separately in a second processing device that is physically separated from the first processing device but can be coupled via a communication interface. Context-dependent computations are performed only in the second processing device, eliminating the need for the second processing device to store and read / write a large number of neural network model parameters, thus significantly improving computational efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. The accompanying drawings in the following description are merely exemplary embodiments of this disclosure.

[0019] Figure 1 This is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0020] Figure 2 A schematic diagram of a processing apparatus according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram of a first processing apparatus according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram of a second processing apparatus according to an embodiment of the present disclosure is shown.

[0023] Figure 5 A schematic diagram is shown illustrating the computation of a converter model that combines a multi-head attention mechanism with the processing device according to an embodiment of the present disclosure.

[0024] Figure 6 Further schematic diagrams are shown of the computation of a converter model that combines processing device processing with a multi-head attention mechanism according to embodiments of the present disclosure.

[0025] Figure 7 A schematic diagram is shown illustrating a processing apparatus according to an embodiment of the present disclosure deployed in a mobile phone and an artificial intelligence personal computer scenario.

[0026] Figure 8 A schematic diagram of a processing device deployed in a vehicle-mounted scenario according to an embodiment of the present disclosure is shown.

[0027] Figure 9 A schematic diagram of a processing apparatus deployed in a smart home scenario according to an embodiment of the present disclosure is shown.

[0028] Figure 10A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0029] Figure 11 A schematic diagram of the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0030] Figure 12 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of this disclosure more apparent, exemplary embodiments according to this disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this disclosure, and not all embodiments of this disclosure. It should be understood that this disclosure is not limited to the exemplary embodiments described herein.

[0032] In this specification and accompanying drawings, substantially the same or similar operations and elements are indicated by the same or similar reference numerals, and repeated descriptions of these operations and elements are omitted. Furthermore, in the description of this disclosure, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance or order.

[0033] To facilitate the description of this disclosure, the following concepts related to this disclosure are introduced.

[0034] Optionally, the models used in embodiments of this disclosure as described below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs, where neurons are arranged in different layers. Generally, a neural network model includes an input layer and an output layer, separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation useful for generating the output in the output layer. Nodes are fully connected to nodes in adjacent layers via edges, and there are no edges between nodes within each layer. Data received at nodes in the input layer of the neural network is propagated to nodes in the output layer via any of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and this disclosure does not limit this.

[0035] The solutions provided in this disclosure involve technologies such as artificial intelligence, machine learning, and semiconductors, and are specifically illustrated through the following embodiments.

[0036] First, refer to Figure 1 The present disclosure describes application scenarios of the processing apparatus and corresponding devices according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and a plurality of terminals 120 are schematically illustrated.

[0037] The neural network model disclosed in this embodiment can be integrated into various electronic devices, for example, Figure 1 The neural network model can be integrated into any electronic device in server 110 and multiple terminals 120. For example, the neural network model can be integrated into terminal 120. Terminal 120 can be a mobile phone, tablet, laptop, desktop computer, personal computer (PC), smart speaker, or smartwatch, but is not limited to these. Alternatively, the neural network model can also be integrated into server 110. Server 110 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminals and servers can be directly or indirectly connected via wired or wireless communication, and this disclosure does not impose any limitations.

[0038] It is understood that the apparatus for inference using the neural network model of this disclosure can be a terminal, a server, or a system composed of a terminal and a server. The data processing device of this disclosure can be executed on a terminal, on a server, or jointly by a terminal and a server.

[0039] The neural network model provided in this disclosure can also relate to artificial intelligence cloud services in the field of cloud technology. It is worth noting that both the terminal 110 and server 120 according to the embodiments of this disclosure adhere to data protection principles, respect users' data rights, and ensure user data security and privacy. The terminal 110 and server 120 according to the embodiments of this disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting user data, and obtain user consent. The terminal 110 and server 120 according to the embodiments of this disclosure will take reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The providers of the terminal 110 and server 120 according to the embodiments of this disclosure will regularly review and update user data, and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of this disclosure respect users' rights to data access, correction, deletion, withdrawal of consent, complaints, and claims, and provide convenient channels and procedures to enable users to effectively exercise these rights.

[0040] Furthermore, the process of data analysis using artificial intelligence technology in terminal 120 or server 110 is conducted based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to embodiments of this disclosure is relevant, necessary, and appropriate for the predictive purpose, and does not contain any personally identifiable or sensitive information. The neural network model according to embodiments of this disclosure employs appropriate techniques and organizational measures to protect the security and integrity of data, preventing unauthorized access, use, or disclosure.

[0041] The artificial intelligence-based neural network model according to embodiments of this disclosure will comply with relevant data protection regulations and ethical principles. This neural network model is trained on a large amount of anonymized and de-identified data, and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable, and do not cause misleading or discrimination. The artificial intelligence model is designed solely to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. Furthermore, the neural network model will be regularly reviewed and updated to adapt to changes in the data environment and legal regulations.

[0042] In recent years, the rapid development of artificial intelligence (AI) algorithms has propelled converter networks to become a core architecture in numerous AI applications, particularly in generative AI, where the algorithms involved in converter networks dominate. Converter networks are based on converter modules composed of attention mechanisms and feed-forward networks, forming complex model structures through cascading and combination. Taking traditional generative neural networks as an example, the converter module accounts for over 99% of the total network parameters, highlighting its central role in the model. In the converter architecture, multi-head attention (MHA) is a key component for achieving efficient computation, responsible for handling the contextual relationships between input information.

[0043] In terminal devices, traditional solutions primarily implement converter network computation in two ways. The first approach integrates a Neural Processing Unit (NPU) within the System-on-Chip (SoC), accelerating AI computation through dedicated hardware. For example, some smartphone SoCs use a built-in NPU for speech recognition model inference. The second approach involves attaching a coprocessor externally to the SoC, dedicated to performing the converter network's computational tasks. For example, some edge computing devices use external AI acceleration chips to handle image generation tasks. Both approaches rely on dedicated hardware to meet the high computational demands of the converter network.

[0044] Whether integrating an NPU within a SoC or using an external coprocessor, traditional solutions perform the converter network computation within a single physical device. The sheer volume of model parameters and computational demands, coupled with the frequent access to model parameters for different tasks, makes optimization difficult. Traditional solutions have extremely high requirements for memory bandwidth and computing power, significantly increasing system cost and power consumption. For example, running generative AI models on high-performance edge devices can consume power far exceeding the limits of low-power devices. Furthermore, further performance improvements will further increase costs, and because computation is concentrated in a single physical device, the scalability of the computing system is limited, making it difficult to increase computing power through modularization. On low- to mid-range terminal devices, limited storage and computing resources make it difficult to store and run large converter models, restricting the implementation of functions such as voice assistants or image generation.

[0045] In summary, traditional solutions perform converter network calculations within a single physical device, resulting in high costs, high power consumption, poor scalability, and difficulty in deploying complex models on low- to mid-range devices.

[0046] To this end, embodiments of this disclosure provide a processing device, including a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from the first processing unit and communicatively coupled thereto, wherein the first processing unit is configured to: calculate a first vector corresponding to the first input information based on at least a portion of model parameters of a neural network model and first input information; and calculate a second vector corresponding to the second input information based on at least a portion of model parameters of the neural network model and second input information; wherein the second processing unit is configured to: calculate a third vector based on the first vector and the second vector.

[0047] This disclosure embodiment achieves an efficient distributed computing architecture by splitting and deploying the neural network computation process onto two physically separate physical devices. Different physical devices are used to perform different types of computational tasks, thereby reducing the computational pressure on a single device and improving the device's scalability and deployment capability on low- and mid-range devices.

[0048] In one optional aspect of the embodiments of this disclosure, at least one second processing device is physically separated from the first processing device and coupled to it via a communication interface. The first processing device processes only context-independent computations, while context-dependent computations are processed only in the second processing device, thereby decoupling context-independent and context-dependent computations. This split deployment reduces the storage and computing requirements of a single device and enhances the modular scalability of the device.

[0049] In one optional aspect of the embodiments of this disclosure, the first processing device only performs context-independent computational tasks and cannot access or deduce semantic relationships between data, thus preventing the acquisition or disclosure of user privacy information. The second processing device, which handles context-dependent computations, can be deployed only on the user's local device or a privacy device, and sensitive user data may not be uploaded to the cloud, reducing the risk of data leakage and thereby protecting user privacy.

[0050] In another optional aspect of the embodiments of this disclosure, the calculation of the first vector and the second vector depends only on some parameters of the neural network model and input information, without involving context-dependent vector or token association calculations. For example, when processing natural language tasks, the first processing device calculates an independent feature vector (such as a word embedding vector) for each input token based on the feedforward network part, the embedding network part, or the projection matrix part of the converter model, without considering the contextual relationships between tokens. Here, although processing a natural language task is used as an example, this disclosure is not limited thereto.

[0051] In another optional aspect of the embodiments of this disclosure, the calculation of the third vector is based solely on the first and second vectors, without involving model parameters, and is context-dependent. For example, in a multi-head attention mechanism, the second processing device calculates an attention score based on the first and second vectors to generate a context-dependent third vector. This approach deploys attention mechanism-dependent calculations separately in a second processing device that is physically separated from the first processing device but can be coupled via a communication interface. Context-dependent calculations are performed only in the second processing device, eliminating the need for the second processing device to store and read / write a large number of neural network model parameters, thus significantly improving computational efficiency. Although a multi-head attention mechanism has been used as an example here, this disclosure is not limited thereto.

[0052] The following reference Figures 2 to 12 Some exemplary details of embodiments according to this disclosure are described.

[0053] Figure 2 A schematic diagram of a processing device 20 according to an embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, this disclosure provides a processing device 20, including a first processing unit 202 and at least one second processing unit 204. The at least one second processing unit 204 is physically separated from the first processing unit 202 and communicatively coupled to it.

[0055] Optionally, "physically separated" means that the first processing device 202 and the second processing device 204 are independent hardware entities located in different packages, or located in the same package but at different physical locations within that package. For example, the first processing device 202 may be deployed on a cloud server or edge computing node, while the second processing device 204 may be integrated into the SOC of a terminal device. Of course, this disclosure is not limited thereto.

[0056] Optionally, "communicatively coupled" means that the first processing device 202 and the second processing device 204 exchange data through a communication interface, such as through a wired connection (e.g., USB, PCIe bus) or a wireless network (e.g., 5G or Wi-Fi). Alternatively, in terms of communication method, the second processing device 204 and the first processing device 202 can achieve communicative coupling through a bus (e.g., USB, PCIe bus) or a network transmission method (e.g., Ethernet, 5G network). For example, in a smart home scenario, the first processing device 202 communicates with the second processing device 204 in the cloud via Wi-Fi; the first processing device 202 processes the parsing of voice commands, while the second processing device 204 processes the context analysis of the voice commands. In an autonomous driving system, the first processing device 202 communicates with the second processing device 204 at the vehicle edge node via a high-speed bus to complete real-time processing of sensor data. Of course, this disclosure is not limited to this.

[0057] Optionally, the at least one second processing device 204 includes at least two heterogeneous second processing devices 204. "Heterogeneous" means that these second processing devices 204 differ in hardware architecture, computing power, or application implementation. For example, one second processing device 204 may be based on a GPU, suitable for parallel computing, while another may be based on an FPGA, suitable for low-latency tasks. Both of the at least two heterogeneous second processing devices 204 are communicatively coupled to the first processing device 202 to ensure collaborative operation of distributed computing. Of course, this disclosure is not limited thereto.

[0058] Optionally, the first processing device 202 includes a first memory 2024, at least one first processor chip 2022, and a first transceiver 2026. The first processor chip 2022 can be implemented as a general-purpose processor (such as a CPU), a graphics processing unit (GPU), or a dedicated neural network accelerator (NPU). For example, in a smartphone, the first processor chip 2022 may be the NPU in an artificial intelligence chip. The first memory 2024 can be implemented as flash memory (such as NAND Flash), with a read speed higher than a write speed to meet the fast read requirements of the converter model parameters. For example, "flash memory" stores some parameters of the converter model, with read speeds reaching several GB / s, while write speeds are lower, suitable for storing static model parameters. The first transceiver 2026 is responsible for communicating with the second processing device 204, supporting wired or wireless data transmission. For example, in an IoT device, the first transceiver 2026 sends the calculation results to the cloud via a Wi-Fi module. Of course, this disclosure is not limited thereto.

[0059] Optionally, the first processing device 202 can be used as an attachment to an electronic processing device. For example, in a smartwatch, the first processing device 202 is an additional AI acceleration module that communicates with the main SOC via Bluetooth to process feature extraction of health data; in industrial control equipment, the first processing device 202 is an independent AI computing card that connects to the main control board via PCIe to process sensor signals. Of course, this disclosure is not limited thereto.

[0060] Optionally, the second processing device 204 includes a second memory 2044, a second processor 2042, and a second transceiver 2046. The second memory 2044 is RAM (such as DRAM or HBM), suitable for storing dynamically generated intermediate data, such as temporary vectors in multi-head attention mechanisms. The high bandwidth of RAM supports fast read and write operations, meeting the needs of context-sensitive computations. The second processor 2042 can be a GPU, FPGA, or a dedicated AI chip (such as a TPU), suitable for processing attention-based computations. The second transceiver 2046 supports high-speed communication with the first processing device 202, for example, receiving a first or second vector sent by the first processing device 202 via a 5G network, Ethernet, or bus.

[0061] Optionally, at least two of the at least one second processing device 204 may implement different applications. For example, one second processing device 204 may be configured to perform natural language processing tasks, such as machine translation. Another second processing device 204 may be configured to perform image processing tasks, such as image classification. Yet another second processing device 204 may be configured to perform speech recognition tasks by processing vectors of speech signals. Different second processing devices 204 may perform different tasks according to specific task requirements. Of course, this disclosure is not limited thereto.

[0062] Optionally, at least one second processing device 204 shares at least a portion of the model parameters of the neural network model in the first processing device 202. For example, in a distributed speech recognition system, the first processing device 202 stores the feedforward network parameters and projection matrix parameters of the converter model, and calculates a first vector or a second vector based on these parameters. Multiple second processing devices 204 obtain their own first or second vectors through network access / bus access to perform attention mechanism calculations. This parameter sharing mechanism reduces the storage requirements of the second processing devices 204 and improves the resource utilization of the system. Another example is an edge computing network, where multiple edge node second processing devices 204 share the model parameters of the cloud-based first processing device 202 to complete their respective specific image generation tasks.

[0063] Optionally, at least one second processing device 204 shares the computing or storage capabilities of the first processing device 202. For example, in a distributed speech recognition system, the first processing device 202 stores the feedforward network parameters and projection matrix parameters of the converter model, and calculates a first vector or a second vector based on these parameters. Multiple second processing devices 204 obtain their own first or second vectors through network access / bus access. This is equivalent to multiple second processing devices 204 sharing the computing and data storage capabilities of the first processing device 202, avoiding at least one second processing device 204 performing repeated data calls and calculations.

[0064] Therefore, the embodiments of this disclosure optimize the computational efficiency of the converter network through a physically separated distributed architecture, providing a low-cost, low-power, and highly scalable solution.

[0065] Figure 3 A schematic diagram of a first processing apparatus 202 according to an embodiment of the present disclosure is shown.

[0066] like Figure 3 As shown, the first processing device 202 can be configured to: calculate a first vector corresponding to the first input information based on at least a portion of the model parameters of the neural network model and the first input information; and calculate a second vector corresponding to the second input information based on at least a portion of the model parameters of the neural network model and the second input information. This disclosure is not limited thereto.

[0067] Optionally, at least a portion of the model parameters of a neural network model (e.g., a transformer model) refers to the parameters used to perform computations in the transformer model. Depending on the specific use case, these model parameters may be static parameters obtained after pre-training on a large-scale dataset, or dynamic parameters that are fine-tuned for a specific task or trained from scratch. Of course, this disclosure is not limited thereto.

[0068] Optionally, at least a portion of the model parameters of the neural network model include all or part of the weight parameters corresponding to the feedforward neural network model. This feedforward neural network can be used for tasks involving further transformation and nonlinear mapping of features processed by the attention mechanism. The weight parameters of the feedforward neural network mainly include the weight matrix and bias terms connecting the neurons in each layer, used to learn the mapping relationship between input features and output results. Of course, this disclosure is not limited thereto.

[0069] Optionally, at least a portion of the model parameters of the neural network model include: weight parameters corresponding to the query matrix, weight parameters corresponding to the key matrix, and weight parameters corresponding to the value matrix. Based on the weight parameters corresponding to the query matrix, key matrix, and value matrix, the input information can be linearly transformed to generate query vectors, key vectors, and value vectors, respectively, for subsequent calculation of attention scores between different vectors by the second processing device 204. Of course, this disclosure is not limited thereto.

[0070] Optionally, the first input information refers to the data unit input to the first processing device 202, such as the word token1. In a natural language processing scenario, token1 can be a word, such as "apple". Based on the weight parameters of the neural network model, the first processing device 202 maps "apple" into a high-dimensional vector, namely the first vector. The first vector is the numerical representation of the input information in a high-dimensional space, containing the semantic information of the word. For example, "apple" may be represented as a 512-dimensional floating-point vector, encoding its semantic features. Of course, this disclosure is not limited thereto.

[0071] Optionally, the second input information is similar; for example, the word token2 could be "banana". The second processing device 204, based on the same neural network model parameters, maps "banana" to a second vector, also a high-dimensional vector, encoding the semantic features of "banana". Of course, this disclosure is not limited to this.

[0072] Optionally, the process of calculating the first vector and the second vector is context-independent because the first processing device 202 performs mapping based solely on individual lexical units and model parameters, without relying on the contextual information of other lexical units. For example, regardless of whether "apple" appears in "princess eating apples" or "apple tree," the calculation result of its first vector is the same, relying only on the weight mapping of "apple" itself.

[0073] Optionally, as described above, the first processing device 202 may include a first memory 2024, a first processor chip 2022, and a first transceiver 2026. The first memory 2024 may be configured to store at least a portion of the model parameters of the neural network model, such as the weight parameters of the query matrix, key matrix, and value matrix. These parameters may be stored as floating-point arrays or tensors, occupying a specific address space or a variable address space in the semiconductor memory.

[0074] Optionally, the first processor chip 2022 can be configured to: obtain at least a portion of the model parameters of the neural network model from the first memory 2024, and calculate the first vector and the second vector. For example, the first processor chip 2022 can obtain the aforementioned model parameters from the first memory 2024, and calculate the first vector based on first input information (e.g., token1, representing a word in a sentence, such as "apple"), and calculate the second vector based on second input information (e.g., token2, representing another word, such as "banana").

[0075] Optionally, the first processor chip 2022 is further configured to compress the first vector and the second vector to obtain compressed first vector and compressed second vector. In neural network models based on converter structures, due to the large scale of model parameters, deep stacking of computing units, and high dimensionality of feature space, the generated intermediate representation vectors (including the first vector and the second vector) often exhibit sparsity. That is, a considerable portion of the dimensions of the first vector or the second vector are close to zero or do not change significantly, and the number of elements actually participating in effective computation is limited. Utilizing this sparsity characteristic, the first processor chip 2022 can compress the first vector and the second vector through sparse matrix compression, quantization encoding, bitmap representation, vector splitting, etc., to reduce the amount of data.

[0076] Optionally, the first transceiver 2026 is configured to send the first vector and the second vector to the second processing device 204. To improve the transmission efficiency of the first and second vectors in a multi-processor system or a multi-chip collaborative system, sending the first and second vectors to the second processing device 204 includes sending compressed versions of the first and second vectors. The compressed first and second vectors consume fewer transmission resources, which is beneficial for quickly exchanging intermediate results in a multi-chip parallel computing architecture, especially suitable for heterogeneous computing architectures of the processing device 20, where the first processing device 202 and the second processing device 204 perform collaborative computing via a high-speed bus or inter-chip interconnect. This disclosure is not limited thereto.

[0077] Optionally, the first processing device 202 is further configured to: calculate output data based on at least a portion of the model parameters of the neural network model and the third vector. This disclosure is not limited thereto.

[0078] Optionally, the first transceiver 2026 is further configured to receive a third vector (or a compressed third vector). The third vector originates from the second processing device 204 and contains information fused from the first vector and the second vector. The reception of the third vector can be implemented through an inter-chip communication interface or a shared cache to improve transmission efficiency.

[0079] Optionally, the first processor chip 2022 is further configured to: obtain at least a portion of the model parameters of the neural network model from the first memory 2024, and calculate output data based on at least a portion of the model parameters of the neural network model and the third vector.

[0080] The model parameters called by the first processor chip 2022 when calculating the output data may include some neural network parameters from the aforementioned feedforward neural network weight matrix. In one example of this disclosure, the first processor chip 2022 reads the above parameters and combines them with the third vector to perform matrix multiplication, nonlinear activation, residual connection, normalization, and other calculation operations to generate the output data required for the next stage. For example, the output data may be the output features of a certain layer in the encoder, the prediction result of a certain step in the decoder, or the joint representation result used in multimodal output. Of course, this disclosure is not limited thereto.

[0081] Optionally, during the training of the neural network model, the first processor chip 2022 is configured to: update at least a portion of the model parameters of the neural network model stored in the first memory 2024 based on the third vector; and calculate the first vector and the second vector based on the updated model parameters of at least a portion of the neural network model. For example, during backpropagation, the third vector may contain gradient information used to adjust the weight parameters of the query matrix, key matrix, and value matrix. The updated model parameters may then be used to recalculate the first and second vectors to improve the prediction accuracy of the neural network model.

[0082] Figure 4 A schematic diagram of a second processing device 204 according to an embodiment of the present disclosure is shown, wherein the second processing device 204 may be configured to calculate a third vector based on the first vector and the second vector.

[0083] like Figure 4As shown, the second processing device 204 includes, but is not limited to, a second transceiver 2046, a second processor 2042, and a second memory 2044. The second transceiver 2046 receives a first vector and a second vector from the first processing device 202, such as vector representations of "apple" and "banana". The second processor 2042 computes a third vector based on these vectors. The third vector is an output vector based on an attention mechanism, whose semantic information represents the relationship between the first and second vectors.

[0084] Optionally, the computation of the third vector is context-dependent, as it depends on the interaction between the first and second vectors. For example, the attention mechanism generates attention weights by computing the dot product of the first vector ("apple") and the second vector ("banana"), reflecting the contribution of each input to the output.

[0085] Optionally, the third vector may contain information fusion results across modules, layers, or modalities. For example, in the multi-head attention mechanism detailed later, multiple sub-vectors are weighted by attention weights and then merged to form the output vector; or in a cross-modal generation model, text features and image features are fused to construct a unified semantic representation for subsequent generation. For example, in the sentence "Apples are delicious, and bananas are sweet," the third vector may represent the fused semantic information of "apple" and "banana" in the context. The calculation process of the third vector will be detailed later, and will not be repeated here.

[0086] Optionally, the second transceiver 2046 of the second processing device 204 may receive the compressed first vector and the compressed second vector. After receiving the compressed first vector and the compressed second vector, the second processing device 204 may perform vector decompression, linear calculation, or nonlinear calculation to construct the fused third vector. Of course, this disclosure is not limited thereto.

[0087] Optionally, the second memory 2044 can be used to store the third vector for subsequent processing or transmission. The second transceiver 2046 is also configured to send the third vector to the first processing device 202. To reduce transmission overhead, the second processor 2042 compresses the third vector and stores it in the second memory 2044, and the second memory 2044 sends the compressed third vector through the second transceiver 2046. Of course, this disclosure is not limited thereto.

[0088] Figure 5 A schematic diagram is shown of a processing device 20 according to an embodiment of the present disclosure processing computations of a converter model combined with a multi-head attention mechanism (MHA).

[0089] Transformer models can be combined with multi-head attention mechanisms to capture the contextual relationships between different lexical units in an input sequence. A transformer model consists of multiple stacked transformer modules, each potentially including an MHA module and a feed-forward network (FFN) module. MHA enhances the model's ability to process information at different positions within the sequence by computing multiple attention heads in parallel. For example, when processing the input sequence "Apples are delicious, bananas are sweet," MHA can capture the semantic association between "apples" and "bananas," thereby improving the accuracy of the output data.

[0090] The input to MHA is the hidden state of each word. (e.g., the first vector or the second vector), where This refers to the hidden dimensions of the model, for example, d=512. Each vector is processed through three weight matrices (query matrices). , Value matrix Projecting the query vector (No. Subvectors of each vector in the query dimension, and key vectors (No. (subvectors of each vector along the key dimension) and value vector (No. (Subvectors of a vector along the value dimension). Query vector Key vector Sum value vector It was subsequently divided into A head of attention, for example Dimensions of each head For example, 512 / 8 = 64 dimensions.

[0091] The MHA calculation process is divided into four steps, which are completed in cooperation by the first processing unit 202 and the second processing unit 204 of the processing device 20, thereby improving the execution efficiency of the converter module.

[0092] Optionally, the first processing unit 202 performs the first step in the MHA calculation process, namely linear projection. This is based on the hidden state of the first input information (e.g., the word "apple"). , dimension The first processing device 202 can calculate the sub-vectors of the first vector corresponding to the first input information in the query dimension, key dimension and value dimension respectively in the following manner, and then combine the sub-vectors in these three dimensions into the first vector.

[0093] Specifically, the first processing unit 202 can calculate the first vector (assuming the first vector corresponds to the weight parameter of the query matrix) based on the weight parameter of the query matrix. The sub-vector of each term in the query dimension is calculated using the following formula: , , The dimension is Similarly, the first processing unit 202 can calculate the sub-vectors of the first vector along the key dimension based on the weight parameters corresponding to the key matrix, using the following formula: , , The dimension is Similarly, the first processing device 202 can calculate the sub-vectors of the first vector in the value dimension based on the weight parameters corresponding to the value matrix, and the calculation formula is as follows: , , The dimension is These calculations are context-free calculations because the first... Each word element corresponds to Relying solely on itself The weight matrix does not involve other lexical units.

[0094] Hidden state based on second input information (e.g., the word "banana") (That is, the second vector corresponds to the first) (each term), the first processing unit 202 can calculate the second vector in a similar manner. Specifically, based on the weight parameters corresponding to the query matrix, the first processing unit 202 can calculate the sub-vectors of the second vector in the query dimension; based on the weight parameters corresponding to the key matrix, the first processing unit 202 can calculate the sub-vectors of the second vector in the key dimension; based on the weight parameters corresponding to the value matrix, the first processing unit 202 can calculate the sub-vectors of the second vector in the value dimension. Similarly, since the same weight matrix is ​​used... , , To calculate the second vector and generate the corresponding... , , . , , The combination of these vectors is denoted as the second vector. Of course, this disclosure is not limited to this.

[0095] After completing the first step of the MHA calculation process, the first vector and the second vector are transmitted to the second processing device 204 for subsequent calculations in the second and third steps of the MHA. Optionally, after receiving the first vector and the second vector, the second processing device 204 may also perform dimensionality reduction on the first vector and the second vector before storing them. Of course, this disclosure is not limited thereto.

[0096] For example, the second processing device 204 may perform a dimensionality reduction operation on the subvectors of the first vector in the query dimension and the subvectors in the key dimension, and store the dimensionality-reduced subvectors of the first vector in the query dimension and the key dimension in the second memory 2044. Similarly, the second processing device 204 may perform a dimensionality reduction operation on the subvectors of the second vector in the query dimension and the subvectors in the key dimension, and store the dimensionality-reduced subvectors of the second vector in the query dimension and the key dimension in the second memory 2044. Of course, this disclosure is not limited thereto.

[0097] Optionally, the second memory 2044 forms a key-value cache by storing these dimensionality-reduced query vectors and key vectors. The principle of the key-value cache is that when an autoregressive model (such as a Transformer decoder) generates a sequence, the key and value vectors at each time step can be cached. When processing the next term, it is not necessary to recalculate the key and value vectors of all previous terms; they can be retrieved directly from the key-value cache. This significantly reduces computation, especially when processing long sequences. Of course, this disclosure is not limited to this.

[0098] For example, when generating a long sentence, the model needs to calculate the attention of each new word with all previous words. Without a key-value cache, the key and value vectors of all previous words need to be recalculated each time. With a key-value cache, these key and value vectors of previous words are stored, and the new word only needs to perform attention calculations with the cached key and value vectors, thus saving a significant amount of computational resources and time. Of course, this disclosure is not limited to this.

[0099] Optionally, the second processing unit 204 may then calculate the third vector based at least in part on the dimensionality-reduced subvectors of the first vector in the query dimension, the dimensionality-reduced subvectors of the first vector in the key dimension, the dimensionality-reduced subvectors of the second vector in the query dimension, and the dimensionality-reduced subvectors of the second vector in the key dimension. This disclosure is not limited thereto.

[0100] Next, we will introduce the second step in the MHA calculation process, which is performed by the second processing unit 204.

[0101] In the second step of the MHA calculation process, the second processing device 204 divides the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively. Of course, this disclosure is not limited thereto.

[0102] For example, if there exists Each attention head, then the query vector It will be divided into key vector It will be divided into value vector It will be divided into Each head processes a portion of the original first vector.

[0103] Similarly, the second processing unit 204 further divides the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively. Similarly, if there exists... Each attention head, then the query vector It will be divided into key vector It will be divided into value vector It will be divided into Each head processes a portion of the original second vector.

[0104] In the third step of the MHA calculation process, the second processing unit 204 performs attention calculation. The process of the second processing unit 204 performing attention calculation can be summarized as follows: based on multiple head vectors of the first vector in the query dimension, multiple head vectors of the second vector in the query dimension, multiple head vectors of the first vector in the key dimension, and multiple head vectors of the second vector in the key dimension, an attention weight set is calculated; and based on the attention weight set, a weighted sum is performed on the multiple head vectors of the first vector in the value dimension and the multiple head vectors of the second vector in the value dimension to obtain a third vector. Of course, this disclosure is not limited to this.

[0105] Optionally, the second processing device 204 traverses multiple head vectors of the second vector in the query dimension and calculates attention weights for each head vector of the second vector in the query dimension.

[0106] Specifically, regarding the second vector in the query dimension... Head vector, If the integer is an integer, the second processing unit 204 can calculate the second vector in the query dimension. The head vector is relative to the first vector in the key dimension. The first attention weight of the head vector. Then, the second processing unit 204 calculates the first attention weight of the second vector on the query dimension. The first head vector is relative to the second vector in the key dimension. The second attention weights for each head vector. This process is accomplished through dot product operations, scaling, and normalization, and is a context-dependent computation.

[0107] Specifically, the second vector in the query dimension... The head vector can be represented as The first vector in the key dimension The head vector is represented as The second vector in the key dimension The head vector is represented as .

[0108] First, calculate and dot product This dot product is called the first dot product. The result of the first dot product is a scalar, meaning it has a dimension of 1. Dividing this scalar by... Scaling is performed, where This defines the dimension for each attention head. The scaled result is then input into the Softmax function for normalization, yielding the first attention weights. .

[0109] Next, calculate the second vector in the query dimension. The first head vector is relative to the second vector in the key dimension. The second attention weight of the head vector. Similarly, the second vector in the query dimension... Head Vector Compared to its first in the key dimension Head Vector The dot product of these values ​​is called the second dot product. This second dot product also undergoes scaling and normalization to ultimately obtain the second attention weights. .

[0110] Then, the second processing device 204 calculates the first vector in the value dimension based on the first attention weight and the second attention weight. The first vector and the second vector in the value dimension are... The weighted sum of the first vectors is used as the third vector. Head vector.

[0111] Specifically, the first attention weight is applied to the first vector in the value dimension. Head Vector , and the second vector in the value dimension Head Vector The third vector is generated by weighted summation. Representation of each head vector.

[0112] Here, although the above only illustrates the example of the third vector possibly depending on the first and second vectors, in reality, a vector may depend on multiple other vectors as its context. If we assume that the vector depends on a total of *k* context vectors starting from the first vector (let t=1), then the attention output corresponding to the *k*th attention head can be expressed as: .

[0113] Similarly, the second processing unit 204 performs the same calculation on all head vectors, and finally concatenates all head vectors to form a complete third vector. The third vector reflects contextual relevance because its calculation depends on the key and value vectors of all words in the sequence. Of course, this disclosure is not limited thereto.

[0114] After the second processing unit 204 sends the third vector to the first processing unit 202, the first processing unit 202 executes the fourth step of MHA, which involves operations on the third vector and the model weight matrix to obtain the output vector. This process is context-independent computation, and the subsequent computation of the feedforward neural network also involves operations on vectors and weight matrices.

[0115] For example, this operation can concatenate and project all the head vectors of the third vector to generate a context-independent output vector. ,in, For the output vector, To output the projection matrix, Subsequently, the feedforward neural network performs additional matrix multiplications, which are also context-independent computations. These computations seamlessly integrate with the first step of the next transformer module, improving overall efficiency.

[0116] Figure 6 Further schematic diagrams are shown of a processing device 20 according to embodiments of the present disclosure processing computations of a converter model combined with a multi-head attention mechanism (MHA).

[0117] As previously mentioned, the MHA calculation process consists of four steps. In another embodiment of this disclosure, the MHA calculation can also be performed solely by the second processing device 204, thereby improving the execution efficiency of the converter module, while the FFN operation following the MHA is performed by the first processing device 202. That is, in this embodiment, the second processing device 204 may perform both context-dependent and context-independent calculations, while the first processing device 202 only performs context-independent calculations.

[0118] Specifically, the second processing device 204 can be configured to: calculate the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; calculate the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; divide the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; divide the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; and calculate the third vector based on each head vector of the first vector and each head vector of the second vector.

[0119] At this point, the first vector corresponding to the first input information is the hidden state of the first input information (e.g., the word "apple"). , dimension The second vector corresponding to the second input information is the hidden state of the second input information (e.g., the word "banana"). (That is, the second vector corresponds to the first) (each word element).

[0120] Optionally, the second processing unit 202 can perform the first step in the MHA calculation process, namely linear projection. The second processing unit 202 can employ a reference... Figure 5 The method described calculates the subvectors of the first vector corresponding to the first input information in the query dimension, key dimension, and value dimension, respectively, and the subvectors of the second vector corresponding to the second input information in the query dimension, key dimension, and value dimension, respectively. These calculations are context-free calculations.

[0121] Optionally, after obtaining the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension, the second processing device 204 may further reduce the dimensionality of these sub-vectors before storing them. Similarly, after obtaining the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension, the second processing device 204 may further reduce the dimensionality of these sub-vectors before storing them. Of course, this disclosure is not limited thereto.

[0122] Optionally, the second processing unit 204 may then calculate the third vector based at least in part on the dimensionality-reduced subvectors of the first vector in the query dimension, the dimensionality-reduced subvectors of the first vector in the key dimension, the dimensionality-reduced subvectors of the second vector in the query dimension, and the dimensionality-reduced subvectors of the second vector in the key dimension. This disclosure is not limited thereto.

[0123] Optionally, after completing the first step in the MHA calculation process, the second processing unit 204 continues to use the reference... Figure 5 The method described is used to perform the calculations for the second and third steps of the subsequent MHA.

[0124] Next, the second and third steps in the MHA calculation process are both executed by the second processing unit 204. Specifically, the second processing unit 204 can perform the following steps: traversing multiple head vectors of the second vector in the query dimension, for the i-th head vector of the second vector in the query dimension (i is an integer): calculating the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension; calculating the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension; based on the first and second attention weights, calculating the weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension, as the i-th head vector of the third vector; and calculating the third vector based on the output projection matrix and all the head vectors of the third vector. This calculation process is similar to... Figure 5 The similarities are not elaborated further here.

[0125] The second processing unit 204 can also execute the fourth step of MHA, namely, performing operations involving the third vector and the model weight matrix to obtain the output vector. This process is context-independent computation, including subsequent calculations of the feedforward neural network, which also involve vector and weight matrix operations. Of course, it can also be performed according to... Figure 5 In this manner, the fourth step of MHA is executed by the first processing unit 202.

[0126] For example, the second processing device 204 can calculate the third vector based on the output projection matrix and all the head vectors of the third vector. ,in, For the third vector, To output the projection matrix, .

[0127] Subsequently, the third vector The vector is sent to the first processing unit 202 to perform additional matrix multiplications using the feedforward neural network of the first processing unit to obtain an output vector. This output vector may also serve as the first or second vector of the next converter module. Of course, this disclosure is not limited thereto.

[0128] Figure 7 A schematic diagram is shown illustrating a processing apparatus according to an embodiment of the present disclosure deployed in a mobile phone and an artificial intelligence personal computer scenario.

[0129] In the context of mobile phones and AI-powered personal computers, the first processing device 202 is implemented as a mobile phone, a portable computer, a desktop computer, or a workstation. Alternatively, the first processing device is implemented as a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation.

[0130] The at least one second processing device 204 includes at least one of the following: smart earphones, smart glasses, and smartwatches. Alternatively, the at least one second processing device 204 includes a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation.

[0131] The computing and storage capabilities of mobile phone main system-on-chips, AI personal computer system-on-chips, headphone system-on-chips, glasses system-on-chips, and watch system-on-chips vary significantly. In the discrete architecture based on processing device 20, the requirements for the computing and storage capabilities of the second processing device 204 are not high, and context-sensitive computations can be well implemented on traditional system-on-chip architectures. Therefore, these traditional system-on-chips can all serve as the second processing device 204 to implement context-sensitive computation functions.

[0132] For context-free computation, the first processing device 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing device 202. Since the trainable weight matrix stored in the first processing device 202 is primarily read-based during inference, a flash memory-based memory can be used. In mobile phone and AI personal computer scenarios, the first processing device 202 can exist as a coprocessor of the main system-on-a-chip (SoC) of the mobile phone or AI personal computer, or exist as a separate domain within the main SoC of the mobile phone or AI personal computer. In this configuration, second processing devices 204, such as headphone SoCs, glasses SoCs, and watch SoCs, can interact with the first processing device 202 without the participation of the main SoC of the mobile phone or AI personal computer or its main domain, to achieve powerful AI functions.

[0133] For example, in an optional embodiment according to this disclosure, the smart earphones can serve as a second processing device 204, sharing the same first processing device 202 (e.g., a mobile phone main system-on-a-chip). Leveraging the powerful processing capabilities of neural networks, the smart earphones can achieve complex functions through the first processing device 202 without further expanding their computing or storage capabilities.

[0134] For example, in an optional embodiment according to this disclosure, the smart earphone system-on-a-chip, smart glasses system-on-a-chip, and smartwatch system-on-a-chip can be used as a second processing device 204 parallel to the main system-on-a-chip of the mobile phone, connected to the same first processing device 202 (which can be an accessory device of the mobile phone). Leveraging the powerful processing capabilities of the neural network in the first processing device 202, the complex calculations required by the smart earphone, smart glasses, and smartwatch can achieve powerful functions without the intervention of the main system-on-a-chip of the mobile phone.

[0135] For example, in an alternative embodiment according to this disclosure, the second processing device 204 may also be built into a mobile phone, portable computer, desktop computer or workstation, and interact with the first processing device 202 as a component.

[0136] Figure 8 A schematic diagram of a processing device deployed in a vehicle-mounted scenario according to an embodiment of the present disclosure is shown.

[0137] In the vehicle-mounted scenario, the first processing device 202 is implemented as a vehicle main control terminal computing device or a component in the vehicle main control terminal computing device.

[0138] At least one second processing device 204 includes at least one of the following: an advanced driver assistance device, an in-vehicle infotainment device, an autonomous driving device, a digital cockpit device, or a domain control device. Alternatively, at least one second processing device includes a component of a domain control device or a vehicle main control computing device.

[0139] Advanced driver assistance system / autonomous driving system-on-chips (SoCs), in-vehicle infotainment system-on-chips (SoCs), digital cockpit system-on-chips (SoCs), domain control system-on-chips (SoCs), and controller system-on-chips (SoCs) differ significantly in their functions, computing power, and storage capacity. In the context-dependent and context-independent computation separation architecture based on processing device 20, the computing and storage requirements for the second processing device 204 are not high, and context-dependent computations can be well implemented on traditional system-on-chip architectures. Therefore, these traditional system-on-chips can all serve as the second processing device 204 to implement context-dependent computation functions, with the actual functional and business division based on the real-time requirements of the in-vehicle scenario.

[0140] For the context-free computation portion, the first processing unit 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing unit 202. Since the trainable weight matrix stored in the first processing unit 202 is primarily read-based during inference, a flash memory-based memory can be used. In automotive scenarios, the first processing unit 202 can exist as a separate processor unit or as a coprocessor within a cockpit / driver / cockpit-driver integrated main system-on-a-chip, depending on the specific functional scenario (such as in-vehicle entertainment or driving scenarios). In this configuration, second processing units 204, such as advanced driver assistance system / autonomous driving system-on-a-chip, in-vehicle infotainment system-on-a-chip, and digital cockpit system-on-a-chip, can interact with the first processing unit 202 according to the scenario to achieve powerful artificial intelligence functions.

[0141] For example, in an optional embodiment according to this disclosure, the advanced driver assistance device can serve as a second processing unit 204, sharing the same first processing unit 202 (e.g., the vehicle's main control computing device) with the in-vehicle infotainment device. Leveraging the powerful processing capabilities of neural networks, the advanced driver assistance device can achieve complex functions through the first processing unit 202 without further expanding its computing or storage capabilities.

[0142] For example, in an optional embodiment according to this disclosure, the advanced driver assistance system / autonomous driving system-on-a-chip, the in-vehicle infotainment system-on-a-chip, and the digital cockpit system-on-a-chip can be used as a second processing device 204 parallel to the vehicle's main control computing device, connected to the same first processing device 202 (which can be an attachment to the vehicle). Leveraging the powerful neural network processing capabilities of the first processing device 202, the complex calculations required by the advanced driver assistance system, the in-vehicle infotainment device, and the digital cockpit device can achieve powerful functions without the intervention of the vehicle's main control computing device.

[0143] For example, in an alternative embodiment according to this disclosure, the second processing device 204 may also be integrated into a domain control device or a vehicle master control terminal computing device, and interact with the first processing device 202 as a component.

[0144] Figure 9 A schematic diagram of a processing apparatus deployed in a smart home scenario according to an embodiment of the present disclosure is shown.

[0145] In a smart home scenario, the first processing device 202 is implemented as a smart home control device or gateway device. Alternatively, the first processing device is implemented as a component within a smart home control device or gateway device.

[0146] At least one second processing device 204 includes at least one of the following: a smart speaker device, a smart voice device, a smart camera device, or a voice control device. Alternatively, the at least one second processing device includes a component of a smart home control device or a gateway device.

[0147] The computing and storage capabilities of smart speaker system-on-chips, smart voice system-on-chips, smart camera system-on-chips, voice control system-on-chips, and smart home appliance control system-on-chips are relatively weak. In the context-dependent and context-independent computation separation architecture based on processing device 20, the requirements for the computing and storage capabilities of the second processing device 204 are not high, and context-dependent computations can be well implemented on traditional system-on-chip architectures. Therefore, these traditional system-on-chips can all serve as the second processing device 204 to implement context-dependent computation functions.

[0148] For the context-free computation portion, the first processing device 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing device 202. Since the trainable weight matrix stored in the first processing device 202 is primarily read-based during inference, a flash memory-based memory can be used. In smart home scenarios, the first processing device 202 can exist as a standalone processor, forming an intelligent processing system with a smart home interconnection chip (e.g., combined with a wireless network router). In this configuration, second processing devices 204, such as smart speaker system-on-a-chip, smart voice system-on-a-chip, and smart camera system-on-a-chip, can interact with the first processing device 202 to achieve powerful artificial intelligence functions.

[0149] For example, in an optional embodiment according to this disclosure, the smart speaker device can serve as a second processing device 204, sharing the same first processing device 202 with the smart camera device (e.g., a smart home control device). Leveraging the powerful processing capabilities of neural networks, the smart speaker device can achieve complex functions through the first processing device 202 without further expanding its computing or storage capabilities.

[0150] For example, in an optional embodiment according to this disclosure, the smart speaker system-on-chip, the smart voice system-on-chip, and the smart camera system-on-chip can serve as a second processing device 204 parallel to the smart home control device, connected to the same first processing device 202 (which can be an attachment device to the smart home). Leveraging the powerful neural network processing capabilities of the first processing device 202, the complex calculations required by the smart speaker device, the smart voice device, and the smart camera device can achieve powerful functions without the intervention of the smart home control device.

[0151] For example, in an alternative embodiment according to this disclosure, the second processing device 204 may also be built into a smart home control device or a gateway device, and interact with the first processing device 202 as a component.

[0152] Furthermore, according to another aspect of this disclosure, a processing method is also provided, executed by a processing device 20, wherein the processing device includes a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from and communicatively coupled to the first processing unit, the processing method comprising: calculating a first vector corresponding to the first input information by the first processing unit based on at least a portion of model parameters of a neural network model and first input information; calculating a second vector corresponding to the second input information by the first processing unit based on at least a portion of model parameters of the neural network model and second input information; and calculating a third vector by the second processing unit based on the first vector and the second vector.

[0153] Furthermore, according to another aspect of this disclosure, a first processing apparatus is also provided, including a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store model parameters of at least a portion of the neural network model; the first processor chip is configured to: calculate a first vector corresponding to the first input information based on the model parameters of at least a portion of the neural network model and the first input information; and calculate a second vector corresponding to the second input information based on the model parameters of at least a portion of the neural network model and the second input information; the first transceiver is configured to: transmit the first vector and the second vector to a second processing apparatus, wherein the second processing apparatus is physically separated from the first processing apparatus and communicatively coupled thereto.

[0154] Furthermore, according to another aspect of this disclosure, a second processing apparatus is also provided, the second processing apparatus comprising: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector from the aforementioned first processing apparatus; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: transmit the third vector to the first processing apparatus.

[0155] According to another aspect of this disclosure, an electronic device is also provided for implementing means according to embodiments of this disclosure. Figure 10 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0156] like Figure 10 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memories 2020 store computer-readable code that, when executed by the one or more processors 2010, can perform the apparatus as described above.

[0157] The processor in this disclosure embodiment can be an integrated circuit chip with signal processing capabilities. The processor can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the various devices, operations, and logic block diagrams disclosed in this disclosure embodiment. The general-purpose processor can be a microprocessor or any conventional processor, and can be based on an x86 architecture or an ARM architecture.

[0158] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or devices described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0159] For example, the apparatus, electronic device, or processing device according to embodiments of this disclosure can also be used by means of... Figure 11 The architecture of the computing device 3000 shown is used for implementation. For example... Figure 11 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage devices in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the device provided in this disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 10 The architecture shown is merely exemplary and can be omitted as needed when implementing different devices. Figure 10 One or more components in the computing device shown.

[0160] According to another aspect of this disclosure, a computer-readable storage medium is also provided. Figure 12 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0161] like Figure 12As shown, the computer-readable storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the apparatus according to embodiments of the present disclosure described with reference to the above figures can be performed. The computer-readable storage medium in the embodiments of the present disclosure may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that the memory of the apparatus described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0162] This disclosure also provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform an apparatus according to an embodiment of this disclosure.

[0163] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0164] In general, the various exemplary embodiments of this disclosure can be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device. When aspects of embodiments of this disclosure are illustrated or described as block diagrams, flowcharts, or using some other graphical representation, it will be understood that the blocks, apparatuses, systems, techniques, or devices described herein can be implemented as non-limiting examples in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controllers or other computing devices, or some combination thereof.

[0165] The exemplary embodiments of this disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will understand that various modifications and combinations of these embodiments or their features can be made without departing from the principles and spirit of this disclosure, and such modifications should fall within the scope of this disclosure. For example, the following provides an overview of some aspects of this disclosure, which can be combined with any other aspects.

[0166] Aspect 1: A processing device is provided, including a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from and communicatively coupled to the first processing unit, wherein the first processing unit is configured to: calculate a first vector corresponding to the first input information based on at least a portion of model parameters of a neural network model and first input information; and calculate a second vector corresponding to the second input information based on at least a portion of model parameters of the neural network model and second input information; wherein the second processing unit is configured to: calculate a third vector based on the first vector and the second vector.

[0167] Aspect 2: In the processing device, the calculation of the first vector is a context-independent calculation, the calculation of the second vector is a context-independent calculation, and the calculation of the third vector is a context-dependent calculation.

[0168] Aspect 3: In the processing device, the first processing device includes a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store at least a portion of the model parameters of the neural network model; the first processor chip is configured to: retrieve at least a portion of the model parameters of the neural network model from the first memory, and calculate the first vector and the second vector; the first transceiver is configured to: send the first vector and the second vector to the second processing device.

[0169] Aspect 4: In the processing device, the second processing unit includes: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: transmit the third vector.

[0170] Aspect 5: In the processing device, at least two of the at least one second processing device implement different applications.

[0171] Aspect 6: In the processing device, at least a portion of the model parameters of the neural network model include at least all or part of the weight parameters corresponding to the feedforward neural network model.

[0172] Aspect 7: In the processing device, at least a portion of the model parameters of the neural network model include: weight parameters corresponding to the query matrix, weight parameters corresponding to the key matrix, and weight parameters corresponding to the value matrix, wherein the first vector is divided into sub-vectors of the first vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension, and the calculation of the first vector corresponding to the first input information includes: calculating the sub-vector of the first vector in the query dimension based on the weight parameters corresponding to the query matrix; calculating the sub-vector of the first vector in the key dimension based on the weight parameters corresponding to the key matrix; and calculating the sub-vector of the first vector in the value dimension based on the weight parameters corresponding to the value matrix.

[0173] Aspect 8: In the processing device, the second vector is divided into sub-vectors of the second vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension. The calculation of the second vector corresponding to the second input information includes: calculating the sub-vector of the second vector in the query dimension based on the weight parameters corresponding to the query matrix; calculating the sub-vector of the second vector in the key dimension based on the weight parameters corresponding to the key matrix; and calculating the sub-vector of the second vector in the value dimension based on the weight parameters corresponding to the value matrix.

[0174] Aspect 9: In the processing device, calculating the third vector based on the first vector and the second vector includes: dividing the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; dividing the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; and calculating the third vector based on each head vector of the first vector and each head vector of the second vector.

[0175] Aspect 10: In the processing device, the calculation of the third vector based on the first vector and the second vector further includes: performing a dimensionality reduction operation on the sub-vectors of the first vector in the query dimension and the sub-vectors in the key dimension, and storing the dimensionality-reduced sub-vectors of the first vector in the query dimension and the dimensionality-reduced sub-vectors in the key dimension in the second memory; performing a dimensionality reduction operation on the sub-vectors of the second vector in the query dimension and the sub-vectors in the key dimension, and storing the dimensionality-reduced sub-vectors of the second vector in the query dimension and the dimensionality-reduced sub-vectors in the key dimension in the second memory; and calculating the third vector based at least in part on the dimensionality-reduced sub-vectors of the first vector in the query dimension, the dimensionality-reduced sub-vectors of the first vector in the key dimension, the dimensionality-reduced sub-vectors of the second vector in the query dimension, and the dimensionality-reduced sub-vectors of the second vector in the key dimension.

[0176] Aspect 11: In the processing device, the calculation of the third vector based on the first vector and the second vector includes: calculating an attention weight set based on multiple head vectors of the first vector in the query dimension, multiple head vectors of the second vector in the query dimension, multiple head vectors of the first vector in the key dimension, and multiple head vectors of the second vector in the key dimension; and performing a weighted summation of the multiple head vectors of the first vector in the value dimension and the multiple head vectors of the second vector in the value dimension based on the attention weight set to obtain the third vector.

[0177] Aspect 12: In the processing device, calculating the third vector based on each head vector of the first vector and each head vector of the second vector includes: traversing multiple head vectors of the second vector in the query dimension; for the i-th head vector of the second vector in the query dimension, where i is an integer: calculating a first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension; calculating a second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension; based on the first attention weight and the second attention weight, calculating a weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension, as the i-th head vector of the third vector; and concatenating all head vectors of the third vector to form the third vector.

[0178] Aspect 13: In the processing device, the calculation of the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension includes: calculating the first dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the first vector in the key dimension; and scaling and normalizing the first dot product to determine the first attention weight.

[0179] Aspect 14: In the processing device, the calculation of the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension includes: calculating the second dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and scaling and normalizing the second dot product to determine the second attention weight.

[0180] Aspect 15: In the processing device, the first processing unit is further configured to: calculate output data based on at least a portion of the model parameters of the neural network model and the third vector.

[0181] Aspect 16: In the processing device, the first transceiver is further configured to: receive a third vector; the first processor chip is further configured to: obtain at least a portion of the model parameters of the neural network model from the first memory, and calculate output data based on at least a portion of the model parameters of the neural network model and the third vector.

[0182] Aspect 17: In the processing device, the at least one second processing device includes at least two heterogeneous second processing devices, and the at least two heterogeneous second processing devices are communicatively coupled to the first processing device.

[0183] Aspect 18: In the processing device, the second processing device is communicatively coupled to the first processing device via a bus, or the second processing device is communicatively coupled to the first processing device via a network transmission.

[0184] Aspect 19: In the processing device, the speed at which the first memory reads data is higher than the speed at which it writes data.

[0185] Aspect 20: In the processing device, the first memory is flash memory.

[0186] Aspect 21: In the processing device, the second memory is RAM.

[0187] Aspect 22: In the processing device, the at least one second processing device shares at least a portion of the model parameters of the neural network model in the first processing device.

[0188] Aspect 23: In the processing device, the at least one second processing device shares the computing power and storage power of the first processing device.

[0189] Aspect 24: In the processing apparatus, the first processing device is an attachment device of the electronic processing apparatus.

[0190] Aspect 25: In the processing device, the first processor chip is further configured to: compress the first vector and the second vector to obtain the compressed first vector and the compressed second vector; wherein, sending the first vector and the second vector to the second processing device includes: sending the compressed first vector and the compressed second vector to the second processing device.

[0191] Aspect 26: In the processing device, the second processor is further configured to: compress the third vector to obtain the compressed third vector; and wherein sending the third vector includes: sending the compressed third vector to the first processing device.

[0192] Aspect 27: In a processing device, wherein the first processing device is implemented as a mobile phone, a portable computer, a desktop computer, or a workstation; or the first processing device is implemented as a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation. The at least one second processing device includes at least one of the following: a smart headset, smart glasses, a smartwatch; or the at least one second processing device includes a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation.

[0193] Aspect 28: In the processing device, the first processing device is implemented as a vehicle master control terminal computing device or a component of a vehicle master control terminal computing device, and the at least one second processing device includes at least one of the following: a driver assistance terminal device, an in-vehicle infotainment device, an autonomous driving terminal device, a digital cockpit device, or a domain control device; or the at least one second processing device includes a domain control device or a component of a vehicle master control terminal computing device.

[0194] Aspect 29: In the processing device, the first processing device is implemented as a smart home control device or a gateway device; or the first processing device is implemented as a component in a smart home control device or a gateway device, wherein the at least one second processing device includes at least one of the following: a smart speaker device, a smart voice device, a smart camera device, a voice control device; or the at least one second processing device includes a component in a smart home control device or a gateway device.

[0195] Aspect 30: In the processing device, the first processor chip is configured to: update at least a portion of the model parameters of the neural network model stored in the first memory based on the third vector; and calculate the first vector and the second vector based on the updated model parameters of at least a portion of the neural network model.

[0196] Aspect 31: In the processing device, the second processing unit is configured to calculate a third vector based on the first vector and the second vector, including: calculating sub-vectors of the first vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; calculating sub-vectors of the second vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; dividing the sub-vectors of the first vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; dividing the sub-vectors of the second vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension into multiple head vectors of the second vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively; and calculating the third vector based on each head vector of the first vector and each head vector of the second vector.

[0197] Aspect 32: In the processing device, calculating the third vector based on each head vector of the first vector and each head vector of the second vector includes: traversing multiple head vectors of the second vector in the query dimension; for the i-th head vector of the second vector in the query dimension, where i is an integer: calculating a first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension; calculating a second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension; based on the first attention weight and the second attention weight, calculating a weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension, as the i-th head vector of the third vector; and calculating the third vector based on the output projection matrix and all head vectors of the third vector.

[0198] Aspect 33: A processing method is provided, executed by a processing device, wherein the processing device includes a first processing unit and at least one second processing unit, wherein the at least one second processing unit is physically separated from and communicatively coupled to the first processing unit, the processing method comprising: calculating a first vector corresponding to the first input information by the first processing unit based on at least a portion of model parameters of a neural network model and first input information; calculating a second vector corresponding to second input information by the first processing unit based on at least a portion of model parameters of the neural network model and second input information; and calculating a third vector by the second processing unit based on the first vector and the second vector.

[0199] Aspect 34: A first processing device is provided, including a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store model parameters of at least a portion of a neural network model; the first processor chip is configured to: calculate a first vector corresponding to the first input information based on the model parameters of at least a portion of the neural network model and the first input information; and calculate a second vector corresponding to the second input information based on the model parameters of at least a portion of the neural network model and the second input information; the first transceiver is configured to: transmit the first vector and the second vector to a second processing device, wherein the second processing device is physically separated from the first processing device and communicatively coupled to it.

[0200] Aspect 35: A second processing apparatus is provided, the second processing apparatus comprising: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive a first vector and a second vector from the first processing apparatus; the second processor is configured to: calculate a third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: transmit the third vector to the first processing apparatus.

Claims

1. A processing apparatus, comprising a first processing unit and a second processing unit at least two ends, wherein, The at least two second processing devices on each end are physically separated from and communicatively coupled to the first processing device. Each of the at least two second processing devices on each end interacts with the first processing device to share its computing and storage capabilities. Furthermore, the applications implemented by the at least two second processing devices on each end are different. The first processing device is configured as follows: Based on at least a portion of the model parameters of the neural network model and the first input information, perform context-independent computation to determine the first vector corresponding to the first input information; and Based on at least a portion of the model parameters of the neural network model and the second input information, a context-independent computation is performed to determine the second vector corresponding to the second input information; Each of the second processing devices on at least two ends is configured as follows: Access the first processing device to obtain the first vector and the second vector, which are specific to the application implemented on the second processing device. Based on the first vector and the second vector, perform context-dependent calculations to determine the third vector.

2. The processing apparatus as described in claim 1, wherein, The first processing device includes a first memory, at least one first processor chip, and a first transceiver. in, The first memory is configured to store at least a portion of the model parameters of the neural network model; The first processor chip is configured to: obtain at least a portion of the model parameters of the neural network model from the first memory, and calculate the first vector and the second vector; The first transceiver is configured to send the first vector and the second vector to the second processing device.

3. The processing apparatus as described in claim 1, wherein, The second processing device includes: a second memory, a second processor, and a second transceiver. in, The second transceiver is configured to receive the first vector and the second vector; The second processor is configured to: calculate the third vector based on the first vector and the second vector; The second memory is configured to store the third vector; The second transceiver is also configured to transmit the third vector.

4. The processing apparatus as described in claim 1, wherein, The model parameters of the neural network model at least include: all or part of the weight parameters corresponding to the feedforward neural network model.

5. The processing apparatus as described in claim 1, wherein, The neural network model includes at least a portion of its model parameters: weight parameters corresponding to the query matrix, weight parameters corresponding to the key matrix, and weight parameters corresponding to the value matrix. The first vector is divided into sub-vectors in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension. Determining the first vector corresponding to the first input information includes: Based on the weight parameters corresponding to the query matrix, calculate the sub-vectors of the first vector in the query dimension; Based on the weight parameters corresponding to the key matrix, calculate the sub-vectors of the first vector in the key dimension; Based on the weight parameters corresponding to the value matrix, calculate the sub-vectors of the first vector in the value dimension.

6. The processing apparatus as described in claim 5, wherein, The second vector is divided into sub-vectors of the second vector in the query dimension, sub-vectors in the key dimension, and sub-vectors in the value dimension. Determining the second vector corresponding to the second input information includes: Based on the weight parameters corresponding to the query matrix, calculate the sub-vectors of the second vector in the query dimension; Based on the weight parameters corresponding to the key matrix, calculate the sub-vectors of the second vector along the key dimension; Based on the weight parameters corresponding to the value matrix, calculate the sub-vectors of the second vector in the value dimension.

7. The processing apparatus as described in claim 6, wherein, The step of performing context-dependent calculations based on the first vector and the second vector to determine the third vector includes: The subvectors of the first vector in the query dimension, the subvectors in the key dimension, and the subvectors in the value dimension are respectively divided into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension. The subvectors of the second vector in the query dimension, the subvector in the key dimension, and the subvector in the value dimension are respectively divided into multiple head vectors of the second vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; and The third vector is calculated based on the head vectors of the first vector and the head vectors of the second vector.

8. The processing apparatus as claimed in claim 1, wherein, The second processing unit is configured to perform context-dependent computations based on the first vector and the second vector to determine the third vector, including: Based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix, calculate the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension. Based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix, calculate the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension; The subvectors of the first vector in the query dimension, the subvectors in the key dimension, and the subvectors in the value dimension are respectively divided into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension. The subvectors of the second vector in the query dimension, the subvector in the key dimension, and the subvector in the value dimension are respectively divided into multiple head vectors of the second vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; and The third vector is calculated based on the head vectors of the first vector and the head vectors of the second vector.

9. The processing apparatus according to any one of claims 6 to 8, wherein, The step of performing context-dependent calculations based on the first vector and the second vector to determine the third vector further includes: The subvectors of the first vector in the query dimension and the subvectors in the key dimension are subjected to dimensionality reduction operation, and the dimensionality-reduced subvectors of the first vector in the query dimension and the dimensionality-reduced subvectors in the key dimension are stored in the second memory. Perform dimensionality reduction on the subvectors of the second vector in the query dimension and the subvectors in the key dimension, and store the dimensionality-reduced subvectors of the second vector in the second memory; and The third vector is calculated at least in part based on the subvectors of the first vector after dimensionality reduction in the query dimension, the subvectors of the first vector after dimensionality reduction in the key dimension, the subvectors of the second vector after dimensionality reduction in the query dimension, and the subvectors of the second vector after dimensionality reduction in the key dimension.

10. The processing apparatus according to any one of claims 6 to 8, wherein, The step of performing context-dependent calculations based on the first vector and the second vector to determine the third vector includes: Based on the multiple head vectors of the first vector in the query dimension, the multiple head vectors of the second vector in the query dimension, the multiple head vectors of the first vector in the key dimension, and the multiple head vectors of the second vector in the key dimension, calculate the attention weight set; and Based on the attention weight set, a weighted sum is performed on the multiple head vectors of the first vector in the value dimension and the multiple head vectors of the second vector in the value dimension to obtain the third vector.

11. The processing apparatus as claimed in claim 7 or 8, wherein, Calculating the third vector based on the head vectors of the first vector and the head vectors of the second vector includes: Iterate through the multiple head vectors of the second vector along the query dimension. For the i-th head vector of the second vector along the query dimension, where i is an integer: Calculate the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension; Calculate the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension; Based on the first attention weight and the second attention weight, calculate the weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension, and use this sum as the i-th head vector of the third vector; and The third vector is formed by concatenating all the head vectors of the third vector.

12. The processing apparatus as claimed in claim 7 or 8, wherein, Calculating the third vector based on the head vectors of the first vector and the head vectors of the second vector includes: Iterate through the multiple head vectors of the second vector along the query dimension. For the i-th head vector of the second vector along the query dimension, where i is an integer: Calculate the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension; Calculate the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension; Based on the first attention weight and the second attention weight, calculate the weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension, and use this sum as the i-th head vector of the third vector; and The third vector is calculated based on the output projection matrix and all the head vectors of the third vector.

13. The processing apparatus of claim 11, wherein, The calculation of the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension includes: Calculate the first dot product between the i-th head vector of the second vector in the query dimension and the i-th head vector of the first vector in the key dimension; and The first dot product is scaled and normalized to determine the first attention weight.

14. The processing apparatus of claim 12, wherein, The calculation of the first attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the first vector in the key dimension includes: Calculate the first dot product between the i-th head vector of the second vector in the query dimension and the i-th head vector of the first vector in the key dimension; and The first dot product is scaled and normalized to determine the first attention weight.

15. The processing apparatus of claim 11, wherein, The calculation of the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension includes: Calculate the second dot product between the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and The second dot product is scaled and normalized to determine the second attention weights.

16. The processing apparatus of claim 12, wherein, The calculation of the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension includes: Calculate the second dot product between the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and The second dot product is scaled and normalized to determine the second attention weights.

17. The processing apparatus as claimed in claim 1, wherein, The first processing device is further configured to: The output data is calculated based on at least some of the model parameters of the neural network model and the third vector.

18. The processing apparatus as claimed in claim 2, wherein, The first transceiver is also configured to receive a third vector; The first processor chip is further configured to: obtain at least a portion of the model parameters of the neural network model from the first memory, and calculate output data based on at least a portion of the model parameters of the neural network model and the third vector.

19. The processing apparatus as claimed in claim 1, wherein, The second processing device on at least two ends includes at least two heterogeneous second processing devices, and both of the at least two heterogeneous second processing devices can be communicatively coupled to the first processing device.

20. The processing apparatus as claimed in claim 1, wherein, The second processing device is communicatively coupled to the first processing device via a bus, or the second processing device is communicatively coupled to the first processing device via a network transmission.

21. The processing apparatus as claimed in claim 2, wherein, The first memory has a higher read speed than a write speed.

22. The processing apparatus as claimed in claim 2, wherein, The first memory is flash memory.

23. The processing apparatus as claimed in claim 3, wherein, The second storage device is RAM.

24. The processing apparatus as claimed in claim 1, wherein, The second processing devices at at least two ends share at least a portion of the model parameters of the neural network model in the first processing device.

25. The processing apparatus as claimed in claim 1, wherein, The first processing device is an attachment device for an electronic processing device.

26. The processing apparatus as claimed in claim 2, wherein, The first processor chip is further configured to: compress the first vector and the second vector to obtain the compressed first vector and the compressed second vector; The step of sending the first vector and the second vector to the second processing device includes sending the compressed first vector and the compressed second vector to the second processing device.

27. The processing apparatus as claimed in claim 3, wherein, The second processor is further configured to: compress the third vector to obtain the compressed third vector; as well as Sending the third vector includes sending the compressed third vector to the first processing device.

28. The processing apparatus as claimed in claim 1, wherein, The first processing device is implemented as a mobile phone, a portable computer, a desktop computer, or a workstation; or the first processing device is implemented as a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation. The second processing device on at least two ends includes at least one of the following: smart earphones, smart glasses, smartwatches; or the second processing device on at least two ends includes a component of at least one of the following: mobile phone, portable computer, desktop computer, or workstation.

29. The processing apparatus as claimed in claim 1, wherein, The first processing device is implemented as a vehicle main control terminal computing device or a component of a vehicle main control terminal computing device. The second processing device on at least two ends includes at least one of the following: a driver assistance device, an in-vehicle infotainment device, an autonomous driving device, a digital cockpit device, or a domain control device; or the second processing device on at least two ends includes a component of a domain control device or a vehicle main control terminal computing device.

30. The processing apparatus as claimed in claim 1, wherein, The first processing device is implemented as a smart home control device or gateway device; or the first processing device is implemented as a component in a smart home control device or gateway device. The second processing device at the at least two ends includes at least one of the following: a smart speaker device, a smart voice device, a smart camera device, a voice control device; or the second processing device at the at least two ends includes a component of a smart home control device or a gateway device.

31. The processing apparatus as claimed in claim 2, wherein, The first processor chip is configured as follows: Based on the third vector, update at least a portion of the model parameters of the neural network model stored in the first memory; and The first vector and the second vector are calculated based on at least a portion of the model parameters of the updated neural network model.

32. A processing method performed by a processing device, wherein the processing device includes a first processing unit and a second processing unit at least two ends, wherein, The at least two second processing devices on the terminal sides are physically separated from the first processing device and communicatively coupled. The at least two second processing devices on the terminal sides interact with the first processing device to share the computing power and storage capacity of the first processing device. The applications implemented by the at least two second processing devices on the terminal sides are different. The processing method includes: based on at least a portion of the model parameters of the neural network model and the first input information, the first processing device performs context-independent calculations to determine the first vector corresponding to the first input information. Based on at least a portion of the model parameters of the neural network model and the second input information, the first processing device performs context-independent calculations to determine the second vector corresponding to the second input information; Each of the second processing devices at the at least two ends accesses the first processing device to obtain the first vector and the second vector for the application itself implemented by the second processing device; and Based on the first vector and the second vector, the second processing device performs context-dependent calculations to determine the third vector.

33. A first processing apparatus, comprising a first memory, at least one first processor chip, and a first transceiver. in, The second processing device on the endpoint is physically separated from the first processing device and communicatively coupled thereto, and at least two of the second processing devices on the endpoint interact with the first processing device to share the computing and storage capabilities of the first processing device, and the applications implemented by the at least two of the second processing devices on the endpoint are different; The first memory is configured to store at least a portion of the model parameters of the neural network model; The first processor chip is configured as follows: Based on at least a portion of the model parameters of the neural network model and the first input information, perform context-independent computation to determine the first vector corresponding to the first input information; as well as Based on at least a portion of the model parameters of the neural network model and the second input information, a context-independent computation is performed to determine the second vector corresponding to the second input information; The first transceiver is configured to send the first vector and the second vector to the second processing device, wherein the first vector and the second vector are only for the application implemented by the second processing device.

34. A second processing device on an end side, the second processing device comprising: A second memory, a second processor, and a second transceiver. in, The second transceiver is configured to receive, from the first processing device as described in claim 33, a first vector and a second vector that are only implemented for the application implemented by the second processing device; The second processor is configured to perform context-dependent computations based on the first vector and the second vector to determine a third vector; The second memory is configured to store the third vector; The second transceiver is also configured to send the third vector to the first processing device.

Images