Processing apparatus and processing method

Abstract

The invention relates to processing equipment, a processing method, a first processing device, a second processing device, electronic equipment and a computer readable storage medium. The processing equipment provided by the invention comprises a first processing device and at least one second processing device, the at least one second processing device and the first processing device are arranged in a physically separated manner and can be coupled in a communication manner, the first processing device is configured to calculate a first vector corresponding to first input information based on at least one part of model parameters of a neural network model and the first input information; calculating a second vector corresponding to second input information based on at least one part of model parameters of the neural network model and the second input information; wherein the second processing device is configured to calculate a third vector based on the first vector and the second vector.

Description

Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and chips, and more specifically to a processing device, a processing method, a first processing device, a second processing device, an electronic device, and a computer-readable storage medium. Background Art

[0002] Artificial intelligence (AI) technology has advanced rapidly in recent years, with AI algorithms based on transformer networks becoming a mainstream architecture. Transformers were first applied to natural language processing tasks and, thanks to their efficient self-attention mechanism and feed-forward neural network architecture, have rapidly gained widespread adoption across a wide range of tasks. Currently, numerous AI systems, particularly generative AI applications such as text generation, image generation, and code generation, utilize transformers as their core algorithms. These applications rely on stacking and combining multiple transformer modules to achieve deeper information modeling and complex task processing capabilities.

[0003] Generative AI, a key branch of current artificial intelligence, focuses on generating new content based on existing data. Compared to traditional discriminative models, generative models emphasize the creativity and coherence of content, placing higher demands on the implementation of the underlying algorithms.

[0004] Transformer models have become a key algorithmic framework in generative AI. However, these models often suffer from large parameter counts and high computational resource consumption. As the model scales, the number of transformer modules increases dramatically, significantly increasing inference time, making deployment difficult, and unsuitable for real-time applications. Summary of the Invention

[0005] Embodiments of the present disclosure provide a processing device, a processing method, a first processing apparatus, a second processing apparatus, an electronic device, and a computer-readable storage medium.

[0006] An embodiment of the present disclosure provides a processing device, comprising a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and communicatively coupled, wherein the first processing device is configured to: calculate a first vector corresponding to the first input information based on model parameters of at least a portion of a neural network model and first input information; and calculate a second vector corresponding to the second input information based on model parameters of at least a portion of the neural network model and second input information; wherein the second processing device is configured to: calculate a third vector based on the first vector and the second vector.

[0007] An embodiment of the present disclosure provides a processing method, which is executed by a processing device, wherein the processing device includes a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and is communicatively coupled, and the processing method includes: based on model parameters of at least a part of a neural network model and first input information, the first processing device calculates a first vector corresponding to the first input information; based on model parameters of at least a part of the neural network model and second input information, the first processing device calculates a second vector corresponding to the second input information; and based on the first vector and the second vector, the second processing device calculates a third vector.

[0008] An embodiment of the present disclosure provides a first processing device, comprising a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store model parameters of at least a portion of the neural network model; the first processor chip is configured to: calculate a first vector corresponding to the first input information based on the model parameters of at least a portion of the neural network model and first input information; and calculate a second vector corresponding to the second input information based on the model parameters of at least a portion of the neural network model and second input information; the first transceiver is configured to: send the first vector and the second vector to a second processing device, wherein the second processing device is physically separated from the first processing device and is communicatively coupled.

[0009] An embodiment of the present disclosure provides a second processing device, which includes: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector from the first processing device as described above; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: send the third vector to the first processing device.

[0010] An embodiment of the present disclosure discloses an electronic device, including: one or more processors; and one or more memories, wherein the memories store a computer executable program, and when the computer executable program is executed by the processor, the above-mentioned device is executed.

[0011] An embodiment of the present disclosure provides a computer-readable storage medium having computer instructions stored thereon. When the computer instructions are executed by a processor, the above-mentioned apparatus is implemented.

[0012] According to another aspect of the present disclosure, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the means provided in the various aspects or various optional implementations of the various aspects described above.

[0013] The disclosed embodiment realizes an efficient distributed computing architecture by splitting the computing process of a neural network and deploying it onto two physically separate physical devices. Different physical devices are used to perform different types of computing tasks, thereby reducing the computing pressure of a single device and improving the scalability of the device and its deployment capabilities on mid- and low-end devices.

[0014] In an optional aspect of the disclosed embodiments, at least one second processing device is physically separated from the first processing device and coupled via a communication interface. The first processing device only processes context-independent computations, while context-dependent computations are processed only in the second processing device, thereby decoupling context-independent computations from context-dependent computations. This split deployment reduces the storage and computing requirements of a single device and enhances the modular scalability of the device.

[0015] In an optional aspect of the disclosed embodiments, the first processing device performs only context-independent computing tasks and cannot access or infer semantic relationships between data, thereby preventing the acquisition or disclosure of user privacy information. The second processing device, which handles context-dependent computing, can be deployed only on the user's local device or a privacy device. This prevents user sensitive data from being uploaded to the cloud, reducing the risk of data leakage and protecting user privacy.

[0016] In another optional aspect of the disclosed embodiments, the calculation of the first vector and the second vector depends only on some parameters and input information of the neural network model, and does not involve the calculation of context-related vectors or token associations. For example, when processing a natural language task, the first processing device calculates an independent feature vector (such as a word embedding vector) for each input token based on the feedforward network portion, the embedding network portion, or the projection matrix portion of the converter model, without considering the contextual relationships between tokens.

[0017] In another optional aspect of the disclosed embodiment, the calculation of the third vector is based only on the first vector and the second vector, does not involve model parameters, and focuses on context-related calculations. For example, in a multi-head attention mechanism, the second processing device calculates the attention score based on the first vector and the second vector to generate a context-related third vector. This method deploys the attention mechanism-related calculations separately in a second processing device that is physically separated from the first processing device and can be coupled via a communication interface. The context-related calculations are only performed in the second processing device. The second processing device does not need to store, read and write a large amount of neural network model parameters, which significantly improves the computing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for describing the embodiments. The drawings described below are only exemplary embodiments of the present disclosure.

[0019] Figure 1 is an example schematic diagram illustrating a scenario according to an embodiment of the present disclosure.

[0020] Figure 2 A schematic diagram of a processing device according to an embodiment of the present disclosure is shown.

[0021] Figure 3 A schematic diagram of a first processing device according to an embodiment of the present disclosure is shown.

[0022] Figure 4 A schematic diagram showing a second processing device according to an embodiment of the present disclosure

[0023] Figure 5 A schematic diagram illustrating a processing device processing calculations of a converter model combined with a multi-head attention mechanism according to an embodiment of the present disclosure is shown.

[0024] Figure 6 Another schematic diagram shows a processing device processing calculations of a converter model combined with a multi-head attention mechanism according to an embodiment of the present disclosure.

[0025] Figure 7 A schematic diagram showing a processing device according to an embodiment of the present disclosure deployed in a mobile phone and an artificial intelligence personal computer scenario is shown.

[0026] Figure 8 A schematic diagram showing a processing device according to an embodiment of the present disclosure deployed in a vehicle-mounted scenario is shown.

[0027] Figure 9 A schematic diagram showing the deployment of a processing device in a smart home scenario according to an embodiment of the present disclosure is shown.

[0028] Figure 10A schematic diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0029] Figure 11 A schematic diagram illustrating the architecture of an exemplary computing device according to an embodiment of the present disclosure is shown.

[0030] Figure 12 A schematic diagram of a storage medium according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present disclosure more apparent, the following will describe in detail exemplary embodiments of the present disclosure with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, rather than all the embodiments of the present disclosure, and it should be understood that the present disclosure is not limited to the exemplary embodiments described herein.

[0032] In this specification and the accompanying drawings, substantially the same or similar operations and elements are represented by the same or similar reference numerals, and repeated descriptions of these operations and elements will be omitted. At the same time, in the description of the present disclosure, the terms "first", "second", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance or ranking.

[0033] To facilitate description of the present disclosure, concepts related to the present disclosure are introduced below.

[0034] Optionally, the models that can be used in the embodiments of the present disclosure below can all be artificial intelligence models, especially artificial intelligence-based neural network models. Typically, artificial intelligence-based neural network models are implemented as acyclic graphs in which neurons are arranged in different layers. Typically, a neural network model includes an input layer and an output layer, which are separated by at least one hidden layer. The hidden layer transforms the input received by the input layer into a representation that is useful for generating output in the output layer. The nodes are fully connected to the nodes in the adjacent layers via edges, and there are no edges between the nodes in each layer. The data received at the nodes of the input layer of the neural network is propagated to the nodes of the output layer via any one of the hidden layers, activation layers, pooling layers, convolutional layers, etc. The input and output of the neural network model can take various forms, and the present disclosure does not limit this.

[0035] The solutions provided by the embodiments of the present disclosure involve technologies such as artificial intelligence, machine learning, and semiconductors, and are specifically illustrated by the following embodiments.

[0036] First, refer to Figure 1 Describe application scenarios of the processing device and corresponding apparatus according to embodiments of the present disclosure. Figure 1 A schematic diagram of an application scenario 100 according to an embodiment of the present disclosure is shown, wherein a server 110 and multiple terminals 120 are schematically shown.

[0037] The neural network model involved in the embodiments of the present disclosure can be integrated into various electronic devices, for example, Figure 1 The server 110 and any electronic device in the multiple terminals 120. For example, the neural network model can be integrated in the terminal 120. The terminal 120 can be a mobile phone, a tablet computer, a laptop computer, a desktop computer, a personal computer (PC, Personal Computer), a smart speaker or a smart watch, etc., but is not limited to this. For another example, the neural network model can also be integrated in the server 110. The server 110 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDN, Content Delivery Network), and big data and artificial intelligence platforms. The terminal and the server can be directly or indirectly connected through wired or wireless communication, and the present disclosure does not limit this.

[0038] It is understood that the device that uses the neural network model of the embodiments of the present disclosure to perform reasoning can be a terminal, a server, or a system composed of a terminal and a server. The data processing device of the embodiments of the present disclosure can be executed on a terminal, a server, or a terminal and a server.

[0039] The neural network model provided by the embodiments of the present disclosure may also involve artificial intelligence cloud services in the field of cloud technology. It is worth noting that the terminal 110 and server 120 according to the embodiments of the present disclosure adhere to data protection principles, respect users' data rights, and safeguard their data security and privacy. The terminal 110 and server 120 according to the embodiments of the present disclosure will clearly inform users of the purpose, method, and scope of collecting, using, storing, transmitting, and deleting their data, and obtain their consent. The terminal 110 and server 120 according to the embodiments of the present disclosure will implement reasonable technical and management measures to prevent user data from being leaked, tampered with, damaged, or lost. The provider of the terminal 110 and server 120 according to the embodiments of the present disclosure will regularly review and update user data and promptly delete expired or useless data. Furthermore, cloud service providers using the embodiments of the present disclosure respect users' rights to access, correct, delete, withdraw consent, file complaints, and make claims, and provide convenient channels and procedures for users to effectively exercise these rights.

[0040] Furthermore, the data analysis process using artificial intelligence technology in the terminal 120 or server 110 is based on the principles of legality, rationality, and transparency. The data collected and processed by the artificial intelligence model according to the embodiments of the present disclosure is relevant, necessary, and appropriate for the prediction purpose and does not contain any personally identifiable information or sensitive information. The neural network model according to the embodiments of the present disclosure adopts appropriate technical and organizational measures to protect the security and integrity of the data and prevent unauthorized access, use, or disclosure of the data.

[0041] The artificial intelligence-based neural network model according to the embodiments of the present disclosure will comply with relevant data protection regulations and ethical principles. The neural network model is trained based on a large amount of anonymized and de-identified data and does not infringe on the privacy rights of any individual or group. The artificial intelligence model has also undergone rigorous testing and evaluation to ensure that its output results are accurate and reliable and do not cause misleading or discrimination. The artificial intelligence model is designed only to improve service quality and customer satisfaction and will not be used for any illegal or unethical purposes. In addition, the neural network model will be reviewed and updated regularly to adapt to changes in the data environment and legal regulations.

[0042] In recent years, the rapid development of artificial intelligence (AI) algorithms has driven converter networks to become the core architecture of many AI applications. In particular, algorithms involving converter networks have come to dominate the field of generative AI. Converter networks are based on converter modules consisting of attention mechanisms and feed-forward neural networks, forming complex model structures through cascading and combining them. For example, in traditional generative neural networks, the converter module accounts for over 99% of the total network parameters, highlighting its central role in the model. Within the converter architecture, the multi-head attention mechanism (MHA) is a key component for efficient computation, responsible for processing the contextual associations between input information.

[0043] Traditionally, converter network computations have been implemented in terminal devices through two main approaches. The first involves integrating a neural network engine (NPU) within the system-on-chip (SoC) to accelerate AI computations using dedicated hardware. For example, some smartphone SoCs utilize built-in NPUs to implement inference for speech recognition models. The second approach involves attaching a coprocessor external to the SoC specifically to perform converter network computations. For example, some edge computing devices utilize external AI accelerator chips to handle image generation tasks. Both approaches rely on dedicated hardware to meet the high computational demands of converter networks.

[0044] Whether it is an NPU integrated within the SOC or an external coprocessor, traditional solutions complete the calculation of the converter network within a single physical device. Due to the huge number of model parameters and the amount of calculation, and the need to frequently call model parameters for different computing tasks, optimization is difficult. Traditional solutions have extremely high requirements for memory bandwidth and computing power, which significantly increases system cost and power consumption. For example, when running a generative AI model on a high-performance edge device, the power consumption may far exceed the tolerance of low-power devices. In addition, continuing to improve performance will further increase costs, and because the calculation is concentrated on a single physical device, the scalability of the computing system is limited, and it is difficult to increase computing power in a modular way. On mid- and low-end terminal devices, due to limited storage and computing resources, it is difficult to store and run large converter models, which limits the implementation of functions such as voice assistants or image generation.

[0045] In summary, traditional solutions perform converter network calculations within a single physical device, resulting in high cost, high power consumption, poor scalability, and difficulty in deploying complex models on mid- and low-end devices.

[0046] To this end, an embodiment of the present disclosure provides a processing device, comprising a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and communicatively coupled, wherein the first processing device is configured to: calculate a first vector corresponding to the first input information based on model parameters of at least a portion of a neural network model and first input information; and calculate a second vector corresponding to the second input information based on model parameters of at least a portion of the neural network model and second input information; wherein the second processing device is configured to: calculate a third vector based on the first vector and the second vector.

[0047] The disclosed embodiment realizes an efficient distributed computing architecture by splitting the computing process of a neural network and deploying it onto two physically separate physical devices. Different physical devices are used to perform different types of computing tasks, thereby reducing the computing pressure of a single device and improving the scalability of the device and its deployment capabilities on mid- and low-end devices.

[0048] In an optional aspect of the disclosed embodiments, at least one second processing device is physically separated from the first processing device and coupled via a communication interface. The first processing device only processes context-independent computations, while context-dependent computations are processed only in the second processing device, thereby decoupling context-independent computations from context-dependent computations. This split deployment reduces the storage and computing requirements of a single device and enhances the modular scalability of the device.

[0049] In an optional aspect of the disclosed embodiments, the first processing device performs only context-independent computing tasks and cannot access or infer semantic relationships between data, thereby preventing the acquisition or disclosure of user privacy information. The second processing device, which handles context-dependent computing, can be deployed only on the user's local device or a privacy device. This prevents user sensitive data from being uploaded to the cloud, reducing the risk of data leakage and protecting user privacy.

[0050] In another optional aspect of an embodiment of the present disclosure, the calculation of the first vector and the second vector depends only on some parameters and input information of the neural network model, and does not involve context-related vector or token association calculations. For example, when processing a natural language task, the first processing device calculates an independent feature vector (such as a word embedding vector) for each input token based on the feedforward network portion, the embedding network portion, or the projection matrix portion of the converter model, without considering the contextual relationship between tokens. Although the processing of a natural language task is used as an example here, the present disclosure is not limited to this.

[0051] In another optional aspect of an embodiment of the present disclosure, the calculation of the third vector is based only on the first vector and the second vector, does not involve model parameters, and is specific to context-related calculations. For example, in a multi-head attention mechanism, the second processing device calculates the attention score based on the first vector and the second vector to generate a context-related third vector. This method deploys the attention mechanism-related calculations separately in a second processing device that is physically separated from the first processing device and can be coupled through a communication interface. The context-related calculations are only performed in the second processing device, and the second processing device does not need to store, read and write a huge amount of neural network model parameters, which significantly improves the computing efficiency. Although the multi-head attention mechanism is used as an example for explanation here, the present disclosure is not limited to this.

[0052] Refer to the following Figures 2 to 12 Some exemplary details of embodiments according to the present disclosure are described.

[0053] Figure 2 A schematic diagram of a processing device 20 according to an embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, an embodiment of the present disclosure provides a processing device 20, including a first processing device 202 and at least one second processing device 204. The at least one second processing device 204 is physically separated from the first processing device 202 and communicatively coupled.

[0055] Alternatively, "physically separate" means that the first processing device 202 and the second processing device 204 are independent hardware entities, located in different packages, or located in the same package but at different physical locations within the package. For example, the first processing device 202 may be deployed on a cloud server or edge computing node, while the second processing device 204 may be integrated into the SOC of a terminal device. However, the present disclosure is not limited to this.

[0056] Optionally, "communicatively coupled" refers to the first processing device 202 and the second processing device 204 exchanging data via a communication interface, such as data transmission via a wired connection (e.g., USB, PCIe bus) or a wireless network (e.g., 5G or Wi-Fi). Alternatively, in terms of communication, the second processing device 204 and the first processing device 202 may be communicatively coupled via a bus (e.g., USB, PCIe bus) or a network transmission method (e.g., Ethernet, 5G network). For example, in a smart home scenario, the first processing device 202 communicates with the second processing device 204 in the cloud via Wi-Fi, with the first processing device 202 handling voice command parsing and the second processing device 204 handling contextual analysis of the voice commands. In an autonomous driving system, the first processing device 202 communicates with the second processing device 204 in a vehicle-mounted edge node via a high-speed bus to complete real-time processing of sensor data. Of course, the present disclosure is not limited to this.

[0057] Optionally, the at least one second processing device 204 includes at least two heterogeneous second processing devices 204. "Heterogeneous" refers to differences in hardware architecture, computing capabilities, or implemented applications of these second processing devices 204. For example, one second processing device 204 may be implemented based on a GPU, suitable for parallel computing, while another may be based on an FPGA, suitable for low-latency tasks. The at least two heterogeneous second processing devices 204 are communicatively coupled to the first processing device 202 to ensure coordinated distributed computing. Of course, the present disclosure is not limited to this.

[0058] Optionally, the first processing device 202 includes a first memory 2024, at least one first processor chip 2022, and a first transceiver 2026. The first processor chip 2022 can be implemented as a general-purpose processor (such as a CPU), a graphics processing unit (GPU), or a dedicated neural network accelerator (NPU). For example, in a smartphone, the first processor chip 2022 may be the NPU in an artificial intelligence chip. The first memory 2024 can be implemented as flash memory (such as NAND flash), which has a higher read speed than a write speed to meet the requirement for fast access to converter model parameters. For example, flash memory stores some converter model parameters, with a read speed of up to several GB / s and a lower write speed, making it suitable for storing static model parameters. The first transceiver 2026 is responsible for communicating with the second processing device 204 and supports wired or wireless data transmission. For example, in an IoT device, the first transceiver 2026 transmits computation results to the cloud via a Wi-Fi module. Of course, the present disclosure is not limited to this.

[0059] Alternatively, the first processing device 202 can be an attached device to an electronic processing device. For example, in a smartwatch, the first processing device 202 is an additional AI acceleration module that communicates with the main SOC via Bluetooth to process feature extraction of health data. In industrial control equipment, the first processing device 202 is an independent AI computing card that connects to the main control board via PCIe to process sensor signals. Of course, the present disclosure is not limited to this.

[0060] Optionally, the second processing device 204 includes a second memory 2044, a second processor 2042, and a second transceiver 2046. The second memory 2044 is a memory (such as DRAM or HBM) suitable for storing dynamically generated intermediate data, such as temporary vectors in a multi-head attention mechanism. The high bandwidth of the memory supports fast read and write speeds, meeting the needs of context-sensitive computing. The second processor 2042 can be a GPU, FPGA, or a dedicated AI chip (such as a TPU), suitable for processing computations based on the attention mechanism. The second transceiver 2046 supports high-speed communication with the first processing device 202, for example, receiving the first vector or second vector sent by the first processing device 202 via a 5G network, Ethernet, or bus.

[0061] Optionally, at least two of the at least one second processing device 204 implement different applications. For example, a second processing device 204 may be configured to perform a natural language processing task, such as machine translation. Another second processing device 204 may be configured to perform an image processing task, such as image classification. Yet another second processing device 204 may be configured to perform a speech recognition task by processing a vector of a speech signal. Different second processing devices 204 may perform different tasks based on specific task requirements. Of course, the present disclosure is not limited to this.

[0062] Optionally, at least one second processing device 204 shares at least a portion of the model parameters of the neural network model in the first processing device 202. For example, in a distributed speech recognition system, the first processing device 202 stores the feedforward network parameters of the converter model and the parameters of the projection matrix, and calculates the first vector or the second vector based on these parameters. Multiple second processing devices 204 obtain their own specific first vectors or second vectors through network access / bus access to perform attention mechanism calculations. This parameter sharing mechanism reduces the storage requirements of the second processing device 204 and improves the resource utilization of the system. Another example is an edge computing network, where the second processing devices 204 of multiple edge nodes share the model parameters of the first processing device 202 in the cloud, and each completes its specific image generation task.

[0063] Optionally, at least one second processing device 204 shares the computing power or storage capacity of the first processing device 202. For example, in a distributed speech recognition system, the first processing device 202 stores the feedforward network parameters and projection matrix parameters of the converter model and calculates the first vector or the second vector based on these parameters. Multiple second processing devices 204 obtain their own specific first vectors or second vectors through network access / bus access. This is equivalent to multiple second processing devices 204 sharing the computing power and data storage capacity of the first processing device 202, avoiding at least one second processing device 204 from performing repeated data calls and calculations.

[0064] Therefore, the embodiments of the present disclosure optimize the computing efficiency of the converter network through a physically separated distributed architecture, and provide a low-cost, low-power and highly scalable solution.

[0065] Figure 3 FIG. 2 shows a schematic diagram of a first processing device 202 according to an embodiment of the present disclosure.

[0066] like Figure 3 As shown, the first processing device 202 may be configured to: calculate a first vector corresponding to the first input information based on at least a portion of the model parameters of the neural network model and the first input information; and calculate a second vector corresponding to the second input information based on at least a portion of the model parameters of the neural network model and the second input information. Of course, the present disclosure is not limited to this.

[0067] Optionally, at least a portion of the model parameters of a neural network model (e.g., a converter model) refers to parameters used to perform calculations in the converter model. Depending on the specific use case, these model parameters may be static parameters obtained through pre-training on a large-scale dataset, or dynamic parameters fine-tuned for a specific task or trained from scratch. However, this disclosure is not limited to this.

[0068] Optionally, at least a portion of the model parameters of the neural network model include at least all or part of the weight parameters corresponding to the feedforward neural network model. The feedforward neural network can be used to perform further transformation and nonlinear mapping on features processed by the attention mechanism. The weight parameters of the feedforward neural network mainly include the weight matrix and bias terms connecting the neurons in each layer, which are used to learn the mapping relationship between input features and output results. Of course, the present disclosure is not limited to this.

[0069] Optionally, at least a portion of the model parameters of the neural network model include: weight parameters corresponding to a query matrix (Query Matrix), weight parameters corresponding to a key matrix (Key Matrix), and weight parameters corresponding to a value matrix (Value Matrix). Based on the weight parameters corresponding to the query matrix, key matrix, and value matrix, a linear transformation can be performed on the input information to generate a query vector, a key vector, and a value vector, respectively, for subsequent calculation of attention scores between different vectors by the second processing device 204. Of course, the present disclosure is not limited to this.

[0070] Optionally, the first input information refers to a data unit input to the first processing device 202, such as a word token1. In a natural language processing scenario, token1 can be a word, such as "apple". The first processing device 202 maps "apple" to a high-dimensional vector, namely the first vector, based on the weight parameters of the neural network model. The first vector is a numerical representation of the input information in a high-dimensional space, which contains the semantic information of the word. For example, "apple" may be represented as a 512-dimensional floating-point vector that encodes its semantic features. Of course, the present disclosure is not limited to this.

[0071] Alternatively, the second input information may be similar, for example, word unit token 2 may be "banana." Based on the same neural network model parameters, the second processing device 204 maps "banana" into a second vector, also a high-dimensional vector, encoding the semantic features of "banana." This disclosure is not limited to this.

[0072] Optionally, the process of calculating the first vector and the second vector is context-independent, because the first processing device 202 performs mapping based only on a single word-gram and model parameters, and does not rely on contextual information of other word-grams. For example, whether "apple" appears in "princess eats apples" or "apple tree," the calculation result of the first vector is the same, relying only on the weight mapping of "apple" itself.

[0073] Optionally, as described above, the first processing device 202 may include a first memory 2024, a first processor chip 2022, and a first transceiver 2026. The first memory 2024 may be configured to store at least a portion of model parameters of the neural network model, such as weight parameters of a query matrix, a key matrix, and a value matrix. These parameters may be stored in the form of floating-point arrays or tensors, occupying a specific address space or a variable address space in the semiconductor memory.

[0074] Optionally, the first processor chip 2022 may be configured to obtain at least a portion of the model parameters of the neural network model from the first memory 2024, and calculate the first vector and the second vector. For example, the first processor chip 2022 may obtain the aforementioned model parameters from the first memory 2024, and calculate the first vector based on first input information (e.g., word unit token 1, representing a word in a sentence, such as "apple"), and calculate the second vector based on second input information (e.g., word unit token 2, representing another word, such as "banana").

[0075] Optionally, the first processor chip 2022 is further configured to compress the first vector and the second vector to obtain the compressed first vector and the compressed second vector. In a neural network model based on a converter structure, due to the large scale of model parameters, deep stacking of computing units, and high dimensionality of the feature space, the generated intermediate representation vectors (including the first vector and the second vector) often exhibit sparsity. That is, the values ​​of a considerable portion of the dimensions in the first vector or the second vector are close to zero or do not change significantly, and the number of elements actually participating in the effective calculation is limited. Utilizing this sparse characteristic, the first processor chip 2022 can compress the first vector and the second vector through sparse matrix compression, quantization coding, bitmap representation, vector splitting, etc. to reduce the amount of data.

[0076] Optionally, the first transceiver 2026 is configured to send the first vector and the second vector to the second processing device 204. To improve transmission efficiency of the first and second vectors in a multi-processor system or a multi-chip collaborative system, sending the first and second vectors to the second processing device 204 includes sending compressed first and second vectors to the second processing device 204. The compressed first and second vectors occupy fewer transmission resources, facilitating the rapid exchange of intermediate results in a multi-chip parallel computing structure. This is particularly applicable to processing devices 20 with heterogeneous computing architectures, where the first and second processing devices 202 and 204 perform collaborative computing via a high-speed bus or inter-chip interconnect. The present disclosure is not limited to this.

[0077] Optionally, the first processing device 202 is further configured to calculate output data based on at least a portion of the model parameters of the neural network model and the third vector. Of course, the present disclosure is not limited to this.

[0078] Optionally, at this time, the first transceiver 2026 is further configured to receive a third vector (or a compressed third vector). The third vector originates from the second processing device 204 and includes information resulting from the fusion of the first and second vectors. The third vector can be received via an inter-chip communication interface or a shared cache to improve transmission efficiency.

[0079] Optionally, the first processor chip 2022 is further configured to: obtain model parameters of at least a portion of the neural network model from the first memory 2024, and calculate output data based on the model parameters of at least a portion of the neural network model and the third vector.

[0080] Among them, the model parameters called by the first processor chip 2022 when calculating the output data may include some neural network parameters in the aforementioned feedforward neural network weight matrix. In one example of the present disclosure, after reading the above parameters, the first processor chip 2022 performs matrix multiplication, nonlinear activation, residual connection, normalization and other calculation operations in combination with the third vector to generate the output data required for the next stage. For example, the output data may be the output features of a certain layer in the encoder, the prediction results of a certain step in the decoder, or the joint representation results used in multimodal output. Of course, the present disclosure is not limited to this.

[0081] Optionally, during training of the neural network model, the first processor chip 2022 is configured to: update at least a portion of the model parameters of the neural network model stored in the first memory 2024 based on the third vector; and calculate the first vector and the second vector based on at least a portion of the updated model parameters of the neural network model. For example, during backpropagation, the third vector may contain gradient information used to adjust weight parameters of the query matrix, the key matrix, and the value matrix. The updated model parameters may then be used to recalculate the first vector and the second vector to improve the prediction accuracy of the neural network model.

[0082] Figure 4 A schematic diagram of a second processing device 204 according to an embodiment of the present disclosure is shown, wherein the second processing device 204 may be configured to calculate a third vector based on the first vector and the second vector.

[0083] like Figure 4As shown, the second processing device 204 includes, but is not limited to, a second transceiver 2046, a second processor 2042, and a second memory 2044. The second transceiver 2046 receives a first vector and a second vector, such as the vector representations of "apple" and "banana," from the first processing device 202. The second processor 2042 calculates a third vector based on these vectors. The third vector is an output vector based on the attention mechanism, and its semantic information represents the relationship between the first vector and the second vector.

[0084] Optionally, the computation of the third vector is contextual, as it depends on the interaction between the first and second vectors. For example, the attention mechanism computes the dot product between the first vector ("apple") and the second vector ("banana") to generate attention weights that reflect the contribution of each input to the output.

[0085] Optionally, the third vector may contain the result of cross-module, cross-layer or cross-modal information fusion. For example, in the multi-head attention mechanism described in detail later, multiple sub-vectors are weighted by attention weights and merged to form an output vector, or in a cross-modal generation model, text features and image features are fused to construct a unified semantic representation for subsequent generation. For example, in the sentence "Apples are delicious and bananas are sweet", the third vector may represent the fused semantic information of "apples" and "bananas" in the context. The calculation process of the third vector will be described in detail later and will not be repeated in this disclosure.

[0086] Optionally, the second transceiver 2046 of the second processing device 204 may receive the compressed first vector and the compressed second vector. After receiving the compressed first vector and the compressed second vector, the second processing device 204 may perform vector decompression, linear calculation, or nonlinear calculation to construct the fused third vector. Of course, the present disclosure is not limited to this.

[0087] Optionally, the second memory 2044 can be used to store the third vector for subsequent processing or transmission. The second transceiver 2046 is further configured to transmit the third vector to the first processing device 202. To reduce transmission overhead, the second processor 2042 compresses the third vector and stores it in the second memory 2044. The second memory 2044 transmits the compressed third vector via the second transceiver 2046. However, the present disclosure is not limited to this.

[0088] Figure 5 FIG2 is a schematic diagram showing a processing device 20 processing calculations of a transformer model combined with a multi-head attention mechanism (MHA) according to an embodiment of the present disclosure.

[0089] The transformer model can be combined with the multi-head attention mechanism to capture the contextual relationships between different tokens in the input sequence. The transformer model consists of multiple stacked transformer modules, each of which may include a Multi-Head Attention (MHA) module and a Feed Forward Network (FFN) module. MHA enhances the model's ability to process information at different positions in the sequence by computing multiple attention heads in parallel. For example, when processing the input sequence "Apples are delicious, bananas are sweet," MHA can capture the semantic connection between "apples" and "bananas," thereby improving the accuracy of the output data.

[0090] The input of MHA is the hidden state of each word (e.g., first vector or second vector), where is the hidden dimension of the model, for example d=512. The vector is passed through three weight matrices (query matrix 、 , value matrix ) is projected into the query vector (No. vectors in the query dimension), key vector (No. vectors along the key dimension) and the value vector (No. Query vector , key vector Sum vector It was then divided into An attention head, such as , the dimensions of each head , for example 512 / 8=64 dimensions.

[0091] The calculation process of MHA is divided into four steps, which are completed by the first processing device 202 and the second processing device 204 of the processing device 20 in collaboration, thereby improving the execution efficiency of the converter module.

[0092] Optionally, the first processing device 202 performs the first step in the calculation process of MHA, namely linear projection. Based on the hidden state of the first input information (such as the word "apple") , the dimension is The first processing device 202 can calculate the sub-vectors of the first vector corresponding to the first input information in the query dimension, key dimension and value dimension respectively in the following manner, and then combine the sub-vectors in these three dimensions into the first vector.

[0093] Specifically, the first processing device 202 may calculate the first vector (assuming that the first vector corresponds to the first vector) based on the weight parameter corresponding to the query matrix. The sub-vector of the query dimension is calculated as follows: , , The dimension is Similarly, the first processing means 202 may calculate the sub-vector of the first vector on the key dimension based on the weight parameter corresponding to the key matrix, and the calculation formula is: , , The dimension is Similarly, the first processing device 202 can calculate the sub-vector of the first vector in the value dimension based on the weight parameter corresponding to the value matrix, and the calculation formula is: , , The dimension is These calculations are context-free because The corresponding word Relying only on itself and weight matrix, without involving other tokens.

[0094] The hidden state based on the second input information (e.g., the word "banana") (That is, the second vector corresponds to the word units), the first processing device 202 can calculate the second vector in a similar manner. Specifically, based on the weight parameters corresponding to the query matrix, the first processing device 202 can calculate the sub-vector of the second vector on the query dimension; based on the weight parameters corresponding to the key matrix, the first processing device 202 can calculate the sub-vector of the second vector on the key dimension; based on the weight parameters corresponding to the value matrix, the first processing device 202 can calculate the sub-vector of the second vector on the value dimension. Similarly, due to the same weight matrix 、 、 To calculate the second vector, generate the corresponding 、 、 . 、 、 The combination of is recorded as the second vector. Of course, the present disclosure is not limited to this.

[0095] After completing the first step of the MHA calculation process, the first vector and the second vector are transmitted to the second processing device 204 for subsequent calculation of the second and third steps of the MHA. Optionally, after receiving the first and second vectors, the second processing device 204 may further perform dimensionality reduction on the first and second vectors and store them. However, the present disclosure is not limited to this.

[0096] For example, the second processing device 204 may perform a dimensionality reduction operation on the subvector of the first vector in the query dimension and the subvector of the first vector in the key dimension, and store the subvector of the first vector after the dimensionality reduction on the query dimension and the subvector of the first vector after the dimensionality reduction on the key dimension in the second memory 2044. Similarly, the second processing device 204 may perform a dimensionality reduction operation on the subvector of the second vector in the query dimension and the subvector of the second vector in the key dimension, and store the subvector of the second vector after the dimensionality reduction on the query dimension and the subvector of the second vector after the dimensionality reduction on the key dimension in the second memory 2044. Of course, the present disclosure is not limited to this.

[0097] Optionally, the second memory 2044 forms a key-value cache (KV Cache) by storing these query vectors and key vectors after dimensionality reduction. The principle of the key-value cache is that when an autoregressive model (such as a Transformer decoder) generates a sequence, the key and value vectors of each time step can be cached. When processing the next word, there is no need to recalculate the key and value vectors of all previous words, and they can be directly obtained from the key-value cache. This significantly reduces the amount of computation, especially when processing long sequences. Of course, the present disclosure is not limited to this.

[0098] For example, when generating a long sentence, each time a new word is generated, the model needs to calculate the attention of that word with all previous words. Without a key-value cache, the key and value vectors of all previous words would need to be recalculated each time. With a key-value cache, the key and value vectors of these previous words are stored, and the new word only needs to perform attention calculations with the key and value vectors in the cache, thus saving a lot of computing resources and time. Of course, this disclosure is not limited to this.

[0099] Afterwards, optionally, the second processing module 204 may calculate the third vector based at least in part on the subvector of the first vector after dimension reduction in the query dimension, the subvector of the first vector after dimension reduction in the key dimension, the subvector of the second vector after dimension reduction in the query dimension, and the subvector of the second vector after dimension reduction in the key dimension. Of course, the present disclosure is not limited to this.

[0100] Next, the second step in the calculation process of MHA is introduced, which is performed by the second processing device 204.

[0101] In the second step of the MHA calculation process, the second processing device 204 divides the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension, respectively. Of course, the present disclosure is not limited to this.

[0102] For example, if there is attention heads, then the query vector will be divided into , key vector will be divided into , value vector will be divided into Each header processes a portion of the original first vector.

[0103] Similarly, the second processing means 204 further divides the sub-vectors of the second vector on the query dimension, the sub-vectors on the key dimension, and the sub-vectors on the value dimension into multiple head vectors of the second vector on the query dimension, multiple head vectors on the key dimension, and multiple head vectors on the value dimension. Similarly, if there is attention heads, then the query vector will be divided into , key vector will be divided into , value vector will be divided into Each header processes a portion of the original second vector.

[0104] In the third step of the MHA calculation process, the second processing device 204 performs attention calculation. The process of the second processing device 204 performing attention calculation can be briefly described as: calculating an attention weight set based on the multiple head vectors of the first vector in the query dimension, the multiple head vectors of the second vector in the query dimension, the multiple head vectors of the first vector in the key dimension, and the multiple head vectors of the second vector in the key dimension; and based on the attention weight set, performing a weighted summation of the multiple head vectors of the first vector in the value dimension and the multiple head vectors of the second vector in the value dimension to obtain a third vector. Of course, the present disclosure is not limited to this.

[0105] Optionally, the second processing module 204 traverses multiple head vectors of the second vector on the query dimension, and calculates an attention weight for each head vector of the second vector on the query dimension.

[0106] Specifically, for the second vector on the query dimension Head vector, is an integer, the second processing device 204 can calculate the first The first vector is the first vector in the key dimension Then, the second processing device 204 calculates the first attention weight of the second vector on the query dimension. The first vector is the first vector relative to the second vector in the key dimension The second attention weight of each head vector is calculated by performing dot product operations, scaling, and normalization, which is a context-dependent calculation.

[0107] Specifically, the second vector is The head vector can be expressed as , the first vector is the first The head vector is represented as , the second vector is the first The head vector is represented as .

[0108] First, calculate and The dot product of , this dot product is called the first dot product. The result of the first dot product is a scalar, that is, the dimension is 1. Divide the scalar by Zoom in, where The dimension of each attention head. The scaled result is input into the Softmax function for normalization to obtain the first attention weight .

[0109] Next, calculate the second vector in the query dimension The first vector is the first vector relative to the second vector in the key dimension The second attention weight of the head vector. Similarly, the second vector in the query dimension Head vector Its first Head vector The dot product of is called the second dot product. The second dot product also undergoes scaling and normalization, and finally obtains the second attention weight .

[0110] Then, the second processing device 204 calculates the first vector in the value dimension based on the first attention weight and the second attention weight. The first vector and the second vector in the value dimension The weighted sum of the head vectors is used as the weighted sum of the head vectors of the third vector. Head vector.

[0111] Specifically, the first attention weight acts on the first vector in the value dimension Head vector , and the second vector in the value dimension Head vector , the third vector is generated by weighted summation in the Representation in the individual dimension.

[0112] Although the above example only uses the third vector as an example to illustrate that it may depend on the first and second vectors, in reality, a vector may depend on multiple other vectors as its context. If we assume that the vector depends on a total of 𝑛 context vectors starting from the first vector (let t = 1), in this case, the attention output corresponding to the 𝑖th attention head can be expressed as: .

[0113] Similarly, the second processing device 204 performs the same calculation on all the head vectors, and finally concatenates all the head vectors to form a complete third vector The third vector reflects the context relevance because its calculation depends on the key and value vectors of all word units in the sequence. Of course, the present disclosure is not limited to this.

[0114] After the second processing device 204 sends the third vector to the first processing device 202, the first processing device 202 performs the fourth step of MHA, i.e., performs operations involving the third vector and the model weight matrix to obtain an output vector. This process is a context-free computation, including the subsequent feedforward neural network calculations, which also involve operations on vectors and weight matrices.

[0115] For example, this operation can concatenate and project all the head vectors of the third vector to generate a context-independent output vector ,in, is the output vector, is the output projection matrix, The feedforward neural network then performs additional matrix multiplications, which are also context-free computations. These computations flow seamlessly into the first step of the next converter module, improving overall efficiency.

[0116] Figure 6 FIG2 is another schematic diagram showing the processing device 20 processing the calculation of the transformer model combined with the multi-head attention mechanism (MHA) according to an embodiment of the present disclosure.

[0117] As previously described, the MHA calculation process is divided into four steps. In another embodiment of the present disclosure, the MHA calculation can also be performed solely by the second processing device 204, thereby improving the execution efficiency of the converter module, while the FFN operation after the MHA is performed by the first processing device 202. That is, in this embodiment, the second processing device 204 may perform both context-dependent and context-independent calculations, while the first processing device 202 only performs context-independent calculations.

[0118] Specifically, the second processing device 204 can be configured to: calculate the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; calculate the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; split the sub-vectors of the first vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; split the sub-vectors of the second vector in the query dimension, the sub-vectors in the key dimension, and the sub-vectors in the value dimension into multiple head vectors of the second vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; and calculate the third vector based on each head vector of the first vector and each head vector of the second vector.

[0119] At this time, the first vector corresponding to the first input information is the hidden state of the first input information (for example, the word "apple") , the dimension is The second vector corresponding to the second input information is the hidden state of the second input information (for example, the word "banana") (That is, the second vector corresponds to the words).

[0120] Optionally, the second processing device 202 may perform the first step in the calculation process of MHA, namely linear projection. Figure 5 The subvectors of the first vector corresponding to the first input information along the query dimension, key dimension, and value dimension are calculated in the described manner, as well as the subvectors of the second vector corresponding to the second input information along the query dimension, key dimension, and value dimension. These calculations are context-free.

[0121] Optionally, after obtaining the subvectors of the first vector in the query dimension, the subvectors in the key dimension, and the subvectors in the value dimension, the second processing device 204 may further perform dimensionality reduction on these subvectors and store them. Similarly, after obtaining the subvectors of the second vector in the query dimension, the subvectors in the key dimension, and the subvectors in the value dimension, the second processing device 204 may further perform dimensionality reduction on these subvectors and store them. Of course, the present disclosure is not limited to this.

[0122] Afterwards, optionally, the second processing module 204 may calculate the third vector based at least in part on the subvector of the first vector after dimension reduction in the query dimension, the subvector of the first vector after dimension reduction in the key dimension, the subvector of the second vector after dimension reduction in the query dimension, and the subvector of the second vector after dimension reduction in the key dimension. Of course, the present disclosure is not limited to this.

[0123] Optionally, after completing the first step in the calculation process of MHA, the second processing device 204 continues to use the reference Figure 5 The second and third steps of the subsequent MHA calculations are performed in the described manner.

[0124] Next, the second step and the third step in the calculation process of MHA are both executed by the second processing device 204. Specifically, the second processing device 204 can perform a traversal of multiple head vectors of the second vector on the query dimension, and for the i-th head vector of the second vector on the query dimension, i is an integer: calculate the first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension; calculate the second attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the second vector on the key dimension; based on the first attention weight and the second attention weight, calculate the weighted sum of the i-th head vector of the first vector on the value dimension and the i-th head vector of the second vector on the value dimension as the i-th head vector of the third vector; and calculate the third vector based on the output projection matrix and all the head vectors of the third vector. This calculation process is the same as Figure 5 Similarly, this disclosure will not be repeated here.

[0125] The second processing device 204 can also perform the fourth step of MHA, that is, perform operations involving the third vector and the model weight matrix to obtain an output vector. This process belongs to context-free calculation, including the subsequent calculation of the feedforward neural network, which is also an operation of the vector and the weight matrix. Of course, it can also be performed according to Figure 5 In this manner, the first processing device 202 performs the fourth step of MHA.

[0126] For example, the second processing device 204 may calculate the third vector based on the output projection matrix and all the head vectors of the third vector. ,in, is the third vector, is the output projection matrix, .

[0127] Then, the third vector The output vector is sent to the first processing device 202 to perform additional matrix multiplication using the feedforward neural network of the first processing device to obtain an output vector. The output vector may also serve as the first vector or the second vector of the next converter module. Of course, the present disclosure is not limited to this.

[0128] Figure 7 A schematic diagram showing a processing device according to an embodiment of the present disclosure deployed in a mobile phone and an artificial intelligence personal computer scenario is shown.

[0129] In the scenario of a mobile phone and an artificial intelligence personal computer, the first processing device 202 is implemented as a mobile phone, a laptop computer, a desktop computer, or a workstation. Alternatively, the first processing device is implemented as a component in at least one of the following: a mobile phone, a laptop computer, a desktop computer, or a workstation.

[0130] The at least one second processing device 204 includes at least one of the following: a smart headset, smart glasses, or a smart watch. Alternatively, the at least one second processing device 204 includes a component of at least one of the following: a mobile phone, a portable computer, a desktop computer, or a workstation.

[0131] The computing and storage capabilities of mobile phone main SoCs, AI personal computer SoCs, headphone SoCs, eyewear SoCs, and watch SoCs vary significantly. In a separate architecture based on processing device 20, the computing and storage capabilities of second processing device 204 are not demanding, and context-sensitive computing can be readily implemented on traditional SoC architectures. Therefore, any of these traditional SoCs can function as second processing device 204 to implement context-sensitive computing capabilities.

[0132] For the context-free computation, the first processing device 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing device 202. Because the trainable weight matrices stored by the first processing device 202 are primarily read during inference, flash memory can be used. In mobile phone and AI PC scenarios, the first processing device 202 can function as a coprocessor for the main SoC of the mobile phone or AI PC, or as a separate domain within the main SoC of the mobile phone or AI PC. In this configuration, the second processing device 204, such as a headphone SoC, eyeglass SoC, or watch SoC, can interact with the first processing device 202 without the involvement of the main SoC of the mobile phone or AI PC or its primary domain, enabling powerful AI functionality.

[0133] For example, in an optional embodiment of the present disclosure, a smart headset can serve as the second processing device 204, sharing the same first processing device 202 (e.g., a mobile phone's main system-on-chip) with the smart glasses. Leveraging the powerful processing capabilities of neural networks, the smart headset can implement complex functions using the first processing device 202 without further expanding the smart headset's computing power or storage capacity.

[0134] For example, in an optional embodiment of the present disclosure, a smart headset SoC, smart glasses SoC, or smart watch SoC can function as a second processing device 204 alongside a mobile phone's main SoC, connected to the same first processing device 202 (which can be an attached device to the mobile phone). Leveraging the powerful processing capabilities of the neural network in first processing device 202, the complex computations required by smart headsets, smart glasses, and smart watches can be performed without the need for the mobile phone's main SoC, enabling powerful functionality.

[0135] For another example, in an optional embodiment according to the present disclosure, the second processing device 204 may also be built into a mobile phone, a portable computer, a desktop computer, or a workstation, and interact with the first processing device 202 in the form of a component thereof.

[0136] Figure 8 A schematic diagram showing a processing device according to an embodiment of the present disclosure deployed in a vehicle-mounted scenario is shown.

[0137] In a vehicle-mounted scenario, the first processing device 202 is implemented as a vehicle master computing device or a component in the vehicle master computing device.

[0138] The at least one second processing device 204 includes at least one of the following: an advanced driver assistance device, an in-vehicle infotainment device, an autonomous driving device, a digital cockpit device, or a domain control device. Alternatively, the at least one second processing device includes a component in a domain control device or a vehicle master computing device.

[0139] Advanced driver assistance systems / autonomous driving SoCs, in-vehicle infotainment SoCs, digital cockpit SoCs, domain control system SoCs, and controller SoCs have significant differences in functionality, computing power, and storage capacity. In an architecture that separates context-dependent and context-independent computing based on processing device 20, the computing and storage requirements for second processing device 204 are relatively low, and context-dependent computing can be readily implemented on traditional SoC architectures. Therefore, any of these traditional SoCs can be used as second processing device 204 to implement context-dependent computing, with actual functional and business divisions based on the real-time requirements of the vehicle scenario.

[0140] For the context-free computation, the first processing device 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing device 202. Since the trainable weight matrices stored by the first processing device 202 are primarily read during inference, flash memory can be used. In an in-vehicle scenario, the first processing device 202 can exist as a standalone processor unit, depending on the functional scenario (such as in-vehicle entertainment or driving), or as a coprocessor for the main cabin / driver / cabin-driver system-on-chip. In this configuration, the second processing device 204, such as an advanced driver assistance system / autonomous driving system-on-chip, an in-vehicle infotainment system-on-chip, or a digital cockpit system-on-chip, can interact with the first processing device 202 based on the scenario, realizing powerful artificial intelligence capabilities.

[0141] For example, in an optional embodiment of the present disclosure, an advanced driver assistance device can serve as the second processing unit 204, sharing the same first processing unit 202 (e.g., the vehicle's main control computing device) with the in-vehicle infotainment device. Leveraging the powerful processing capabilities of neural networks, the advanced driver assistance device can implement complex functions using the first processing unit 202 without further expanding its computing or storage capabilities.

[0142] For another example, in an optional embodiment of the present disclosure, an advanced driver assistance system / autonomous driving system-on-chip, an in-vehicle infotainment system-on-chip, and a digital cockpit system-on-chip can function as a second processing device 204, parallel to the vehicle's main computing device, connected to the same first processing device 202 (which can be a vehicle-attached device). Leveraging the powerful neural network processing capabilities of first processing device 202, the complex computations required by advanced driver assistance devices, in-vehicle infotainment devices, and digital cockpit devices can be performed without the involvement of the vehicle's main computing device, achieving powerful functionality.

[0143] For another example, in an optional embodiment according to the present disclosure, the second processing device 204 may also be built into the domain control device or the vehicle master computing device, and interact with the first processing device 202 in the form of its component.

[0144] Figure 9 A schematic diagram showing the deployment of a processing device in a smart home scenario according to an embodiment of the present disclosure is shown.

[0145] In a smart home scenario, the first processing device 202 is implemented as a smart home control device or a gateway device, or as a component in the smart home control device or the gateway device.

[0146] The at least one second processing device 204 includes at least one of the following: a smart speaker device, a smart voice device, a smart camera device, or a voice control device, or the at least one second processing device includes a component in a smart home control device or a gateway device.

[0147] Smart speaker SoCs, smart voice SoCs, smart camera SoCs, voice control systems, and smart home appliance control systems have relatively weak computing and storage capabilities. In an architecture based on the separation of context-dependent and non-context-independent computing in processing device 20, the computing and storage capabilities of second processing device 204 are not demanding, and context-dependent computing can be readily implemented on traditional SoC architectures. Therefore, these traditional SoCs can all serve as second processing device 204 to implement context-dependent computing capabilities.

[0148] For the context-free computation, the first processing device 202 needs to store a large amount of weight data and possess powerful computing capabilities. This requirement can be met by implementing a separate processor for the first processing device 202. Since the trainable weight matrices stored by the first processing device 202 are primarily read during inference, flash memory can be used. In a smart home scenario, the first processing device 202 can exist as a standalone processor, forming an intelligent processing system with a smart home interconnect chip (for example, integrated with a wireless network router). In this configuration, a second processing device 204, such as a smart speaker system-on-chip, an intelligent voice system-on-chip, or a smart camera system-on-chip, can interact with the first processing device 202 to implement powerful artificial intelligence capabilities.

[0149] For example, in an optional embodiment of the present disclosure, a smart speaker device can serve as the second processing device 204, sharing the same first processing device 202 with a smart camera device (e.g., a smart home control device). Leveraging the powerful processing capabilities of neural networks, the smart speaker device can implement complex functions using the first processing device 202 without further expanding its computing power or storage capacity.

[0150] For another example, in an optional embodiment of the present disclosure, a smart speaker SoC, smart voice SoC, and smart camera SoC can function as a second processing device 204 alongside a smart home control device, connected to the same first processing device 202 (which can be an attached device to the smart home). Leveraging the powerful neural network processing capabilities of first processing device 202, the complex computations required by smart speaker, smart voice, and smart camera devices can be performed without the involvement of the smart home control device, achieving powerful functionality.

[0151] For another example, in an optional embodiment according to the present disclosure, the second processing device 204 may also be built into a smart home control device or a gateway device, and interact with the first processing device 202 in the form of its component.

[0152] In addition, according to another aspect of the present disclosure, a processing method is provided, which is executed by a processing device 20, wherein the processing device includes a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and is communicatively coupled, and the processing method includes: based on the model parameters of at least a part of the neural network model and the first input information, the first processing device calculates the first vector corresponding to the first input information; based on the model parameters of at least a part of the neural network model and the second input information, the first processing device calculates the second vector corresponding to the second input information; and based on the first vector and the second vector, the second processing device calculates the third vector.

[0153] In addition, according to another aspect of the present disclosure, a first processing device is provided, including a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store model parameters of at least a part of the neural network model; the first processor chip is configured to: calculate a first vector corresponding to the first input information based on the model parameters of at least a part of the neural network model and first input information; and calculate a second vector corresponding to the second input information based on the model parameters of at least a part of the neural network model and second input information; the first transceiver is configured to: send the first vector and the second vector to a second processing device, wherein the second processing device is physically separated from the first processing device and is communicatively coupled.

[0154] In addition, according to another aspect of the present disclosure, a second processing device is provided, which includes: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector from the above-mentioned first processing device; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: send the third vector to the first processing device.

[0155] According to yet another aspect of the present disclosure, an electronic device is provided for implementing the apparatus according to the embodiments of the present disclosure. Figure 10 A schematic diagram of an electronic device 2000 according to an embodiment of the present disclosure is shown.

[0156] like Figure 10 As shown, the electronic device 2000 may include one or more processors 2010 and one or more memories 2020. The memory 2020 stores computer-readable codes, which, when executed by the one or more processors 2010, may execute the above-described apparatus.

[0157] The processor in the embodiments of the present disclosure may be an integrated circuit chip having signal processing capabilities. The processor may be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The various devices, operations, and logic block diagrams disclosed in the embodiments of the present disclosure may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor, and may be an X86 architecture or an ARM architecture.

[0158] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or apparatus described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0159] For example, the device or electronic device or processing device according to the embodiment of the present disclosure may also be used by means of Figure 11 The architecture of the computing device 3000 shown in FIG. Figure 11 As shown, the computing device 3000 may include a bus 3010, one or more CPUs 3020, a read-only memory (ROM) 3030, a random access memory (RAM) 3040, a communication port 3050 connected to a network, an input / output component 3060, a hard disk 3070, etc. The storage device in the computing device 3000, such as the ROM 3030 or the hard disk 3070, may store various data or files used for processing and / or communication of the apparatus provided by the present disclosure, as well as program instructions executed by the CPU. The computing device 3000 may also include a user interface 3080. Of course, Figure 10 The architecture shown is only exemplary and can be omitted according to actual needs when implementing different devices. Figure 10 One or more components of a computing device are shown.

[0160] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided. Figure 12 A schematic diagram of a storage medium 4000 according to the present disclosure is shown.

[0161] like Figure 12As shown, the computer-readable storage medium 4020 stores computer-readable instructions 4010. When the computer-readable instructions 4010 are executed by a processor, the apparatus according to the embodiments of the present disclosure described with reference to the above figures can be executed. The computer-readable storage medium in the embodiments of the present disclosure can be volatile memory or non-volatile memory, or can include both volatile and non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM). It should be noted that memory of the apparatus described herein is intended to comprise, but not be limited to, these and any other suitable types of memory. It should be noted that memory of the apparatus described herein is intended to comprise, but not be limited to, these and any other suitable types of memory.

[0162] The present disclosure also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the apparatus according to the present disclosure.

[0163] It should be noted that the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the systems, devices and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of the code, and the module, program segment, or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0164] In general, various example embodiments of the present disclosure may be implemented in hardware or dedicated circuitry, software, firmware, logic, or any combination thereof. Certain aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device. When various aspects of the disclosed embodiments are illustrated or described as block diagrams, flow charts, or using some other graphical representation, it will be understood that the blocks, devices, systems, techniques, or apparatus described herein may be implemented, as non-limiting examples, in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or a controller or other computing device, or some combination thereof.

[0165] The exemplary embodiments of the present disclosure described in detail above are merely illustrative and not restrictive. Those skilled in the art will appreciate that various modifications and combinations may be made to these embodiments or their features without departing from the principles and spirit of the present disclosure, and such modifications should fall within the scope of the present disclosure. For example, an overview of some aspects of the present disclosure is provided below, and these aspects may be combined with any other aspects.

[0166] Aspect 1: A processing device is provided, comprising a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and communicatively coupled, wherein the first processing device is configured to: calculate a first vector corresponding to the first input information based on model parameters of at least a part of the neural network model and first input information; and calculate a second vector corresponding to the second input information based on model parameters of at least a part of the neural network model and second input information; wherein the second processing device is configured to: calculate a third vector based on the first vector and the second vector.

[0167] Aspect 2: In the processing device, the calculation of the first vector is a context-independent calculation, the calculation of the second vector is a context-independent calculation, and the calculation of the third vector is a context-dependent calculation.

[0168] Aspect 3: In the processing device, the first processing device includes a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store at least a part of the model parameters of at least a part of the neural network model; the first processor chip is configured to: obtain at least a part of the model parameters of the neural network model from the first memory, and calculate the first vector and the second vector; the first transceiver is configured to: send the first vector and the second vector to the second processing device.

[0169] Aspect 4: In a processing device, the second processing device includes: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: send the third vector.

[0170] Aspect 5: In the processing device, at least two of the at least one second processing device implement different applications.

[0171] Aspect 6: In the processing device, the model parameters of at least a portion of the neural network model include at least: all or part of the weight parameters corresponding to the feedforward neural network model.

[0172] Aspect 7: In the processing device, the model parameters of at least a part of the neural network model include: weight parameters corresponding to the query matrix, weight parameters corresponding to the key matrix, and weight parameters corresponding to the value matrix, wherein the first vector is divided into sub-vectors of the first vector on the query dimension, sub-vectors on the key dimension, and sub-vectors on the value dimension, and the calculation of the first vector corresponding to the first input information includes: calculating the sub-vector of the first vector on the query dimension based on the weight parameters corresponding to the query matrix; calculating the sub-vector of the first vector on the key dimension based on the weight parameters corresponding to the key matrix; and calculating the sub-vector of the first vector on the value dimension based on the weight parameters corresponding to the value matrix.

[0173] Aspect 8: In the processing device, the second vector is divided into a sub-vector of the second vector on the query dimension, a sub-vector on the key dimension, and a sub-vector on the value dimension. The calculation of the second vector corresponding to the second input information includes: calculating the sub-vector of the second vector on the query dimension based on the weight parameters corresponding to the query matrix; calculating the sub-vector of the second vector on the key dimension based on the weight parameters corresponding to the key matrix; and calculating the sub-vector of the second vector on the value dimension based on the weight parameters corresponding to the value matrix.

[0174] Aspect 9: In the processing device, the calculation of the third vector based on the first vector and the second vector includes: splitting the sub-vectors of the first vector on the query dimension, the sub-vectors on the key dimension, and the sub-vectors on the value dimension into multiple head vectors of the first vector on the query dimension, multiple head vectors on the key dimension, and multiple head vectors on the value dimension, respectively; splitting the sub-vectors of the second vector on the query dimension, the sub-vectors on the key dimension, and the sub-vectors on the value dimension into multiple head vectors of the second vector on the query dimension, multiple head vectors on the key dimension, and multiple head vectors on the value dimension, respectively; and calculating the third vector based on the respective head vectors of the first vector and the respective head vectors of the second vector.

[0175] Aspect 10: In the processing device, the calculation of the third vector based on the first vector and the second vector also includes: performing a dimensionality reduction operation on the subvector of the first vector on the query dimension and the subvector on the key dimension, and storing the subvector of the first vector after the dimensionality reduction on the query dimension and the subvector on the key dimension in the second memory; performing a dimensionality reduction operation on the subvector of the second vector on the query dimension and the subvector on the key dimension, and storing the subvector of the second vector after the dimensionality reduction on the query dimension and the subvector on the key dimension in the second memory; and calculating the third vector at least partially based on the subvector of the first vector after the dimensionality reduction on the query dimension, the subvector of the first vector after the dimensionality reduction on the key dimension, the subvector of the second vector after the dimensionality reduction on the query dimension, and the subvector of the second vector after the dimensionality reduction on the key dimension.

[0176] Aspect 11: In the processing device, the calculation of the third vector based on the first vector and the second vector includes: calculating an attention weight set based on multiple head vectors of the first vector on the query dimension, multiple head vectors of the second vector on the query dimension, multiple head vectors of the first vector on the key dimension, and multiple head vectors of the second vector on the key dimension; and based on the attention weight set, performing weighted summation of the multiple head vectors of the first vector on the value dimension and the multiple head vectors of the second vector on the value dimension to obtain the third vector.

[0177] Aspect 12: In a processing device, based on the head vectors of the first vector and the head vectors of the second vector, calculating the third vector includes: traversing multiple head vectors of the second vector on the query dimension, and for the i-th head vector of the second vector on the query dimension, i is an integer: calculating the first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension; calculating the second attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the second vector on the key dimension; based on the first attention weight and the second attention weight, calculating the weighted sum of the i-th head vector of the first vector on the value dimension and the i-th head vector of the second vector on the value dimension as the i-th head vector of the third vector; and splicing all the head vectors of the third vector into the third vector.

[0178] Aspect 13: In the processing device, the calculation of the first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension includes: calculating the first dot product of the i-th head vector of the second vector on the query dimension and the i-th head vector of the first vector on the key dimension; and scaling and normalizing the first dot product to determine the first attention weight.

[0179] Aspect 14: In the processing device, the calculation of the second attention weight of the i-th head vector of the second vector in the query dimension relative to the i-th head vector of the second vector in the key dimension includes: calculating the second dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and scaling and normalizing the second dot product to determine the second attention weight.

[0180] Aspect 15: In the processing device, the first processing means is further configured to: calculate output data based on model parameters of at least a portion of the neural network model and the third vector.

[0181] Aspect 16: In the processing device, the first transceiver is further configured to: receive a third vector; the first processor chip is further configured to: obtain model parameters of at least a part of the neural network model from the first memory, and calculate output data based on the model parameters of at least a part of the neural network model and the third vector.

[0182] Aspect 17: In the processing apparatus, the at least one second processing device includes at least two heterogeneous second processing devices, and the at least two heterogeneous second processing devices are communicatively coupled to the first processing device.

[0183] Aspect 18: In the processing apparatus, the second processing device is communicably coupled to the first processing device via a bus, or the second processing device is communicably coupled to the first processing device via a network transmission.

[0184] Aspect 19: In the processing device, a data reading speed of the first memory is higher than a data writing speed.

[0185] Aspect 20: In the processing device, the first memory is a flash memory.

[0186] Aspect 21: In the processing device, the second memory is a memory.

[0187] Aspect 22: In the processing apparatus, the at least one second processing device shares model parameters of at least a portion of the neural network model in the first processing device.

[0188] Aspect 23: In the processing apparatus, the at least one second processing device shares computing power and storage capacity of the first processing device.

[0189] Aspect 24: In a processing device, the first processing device is an attachment device of the electronic processing device.

[0190] Aspect 25: In the processing device, the first processor chip is further configured to: compress the first vector and the second vector to obtain the compressed first vector and the compressed second vector; wherein, sending the first vector and the second vector to the second processing device includes: sending the compressed first vector and the compressed second vector to the second processing device.

[0191] Aspect 26: In the processing device, the second processor is further configured to: compress the third vector to obtain the compressed third vector; and wherein the sending the third vector includes: sending the compressed third vector to the first processing device.

[0192] Aspect 27: In a processing device, wherein the first processing device is implemented as a mobile phone, a laptop computer, a desktop computer, or a workstation; or the first processing device is implemented as a component of at least one of the following: a mobile phone, a laptop computer, a desktop computer, or a workstation. The at least one second processing device includes at least one of the following: a smart headset, smart glasses, or a smart watch; or the at least one second processing device includes a component of at least one of the following: a mobile phone, a laptop computer, a desktop computer, or a workstation.

[0193] Aspect 28: In the processing device, the first processing device is implemented as a vehicle master-end computing device or a component in the vehicle master-end computing device, and the at least one second processing device includes at least one of the following: a driving assistance end-side device, an in-vehicle infotainment device, an autonomous driving end-side device, a digital cockpit device, or a domain control device; or the at least one second processing device includes a domain control device or a component in the vehicle master-end computing device.

[0194] Aspect 29: In the processing device, the first processing device is implemented as a smart home control device or a gateway device; or the first processing device is implemented as a component in a smart home control device or a gateway device, and the at least one second processing device includes at least one of the following items: a smart speaker device, a smart voice device, a smart camera device, a voice control device; or the at least one second processing device includes a component in a smart home control device or a gateway device.

[0195] Aspect 30: In the processing device, the first processor chip is configured to: update the model parameters of at least a part of the neural network model stored in the first memory based on the third vector; and calculate the first vector and the second vector based on the updated model parameters of at least a part of the neural network model.

[0196] Aspect 31: In the processing device, the second processing device is configured to calculate the third vector based on the first vector and the second vector, including: calculating the sub-vector of the first vector in the query dimension, the sub-vector in the key dimension, and the sub-vector in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; calculating the sub-vector of the second vector in the query dimension, the sub-vector in the key dimension, and the sub-vector in the value dimension based on the weight parameters corresponding to the query matrix, the weight parameters corresponding to the key matrix, and the weight parameters corresponding to the value matrix; dividing the sub-vector of the first vector in the query dimension, the sub-vector in the key dimension, and the sub-vector in the value dimension into multiple head vectors of the first vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; dividing the sub-vector of the second vector in the query dimension, the sub-vector in the key dimension, and the sub-vector in the value dimension into multiple head vectors of the second vector in the query dimension, multiple head vectors in the key dimension, and multiple head vectors in the value dimension; and calculating the third vector based on each head vector of the first vector and each head vector of the second vector.

[0197] Aspect 32: In a processing device, based on the head vectors of the first vector and the head vectors of the second vector, calculating the third vector includes: traversing multiple head vectors of the second vector on the query dimension, for the i-th head vector of the second vector on the query dimension, i is an integer: calculating the first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension; calculating the second attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the second vector on the key dimension; based on the first attention weight and the second attention weight, calculating the weighted sum of the i-th head vector of the first vector on the value dimension and the i-th head vector of the second vector on the value dimension as the i-th head vector of the third vector; and calculating the third vector based on the output projection matrix and all the head vectors of the third vector.

[0198] Aspect 33: A processing method is provided, which is performed by a processing device, wherein the processing device includes a first processing device and at least one second processing device, wherein the at least one second processing device is physically separated from the first processing device and is communicatively coupled, and the processing method includes: based on the model parameters of at least a part of the neural network model and the first input information, the first processing device calculates the first vector corresponding to the first input information; based on the model parameters of at least a part of the neural network model and the second input information, the first processing device calculates the second vector corresponding to the second input information; and based on the first vector and the second vector, the second processing device calculates the third vector.

[0199] Aspect 34: A first processing device is provided, comprising a first memory, at least one first processor chip, and a first transceiver, wherein the first memory is configured to: store model parameters of at least a part of the neural network model; the first processor chip is configured to: calculate a first vector corresponding to the first input information based on the model parameters of at least a part of the neural network model and first input information; and calculate a second vector corresponding to the second input information based on the model parameters of at least a part of the neural network model and second input information; the first transceiver is configured to: send the first vector and the second vector to a second processing device, wherein the second processing device is physically separated from the first processing device and is communicatively coupled.

[0200] Aspect 35: A second processing device is provided, comprising: a second memory, a second processor, and a second transceiver, wherein the second transceiver is configured to: receive the first vector and the second vector from the first processing device; the second processor is configured to: calculate the third vector based on the first vector and the second vector; the second memory is configured to: store the third vector; and the second transceiver is further configured to: send the third vector to the first processing device.

Claims

1. A processing device comprising a first processing device and at least one second processing device, wherein: The at least one second processing device is physically separated from the first processing device and is communicatively coupled to the first processing device, Wherein, the first processing device is configured to: Calculating a first vector corresponding to the first input information based on at least a portion of model parameters of the neural network model and the first input information; and Calculating a second vector corresponding to the second input information based on at least a portion of model parameters of the neural network model and the second input information; Wherein, the second processing device is configured as follows: Based on the first vector and the second vector, a third vector is calculated.

2. The processing device according to claim 1, wherein The calculation of the first vector is a context-free calculation. The computation of the second vector is context-free, and The calculation of the third vector is a context-sensitive calculation.

3. The processing device according to claim 1, wherein The first processing device includes a first memory, at least one first processor chip, and a first transceiver, in, The first memory is configured to: store at least a portion of model parameters of at least a portion of the neural network model; The first processor chip is configured to: obtain model parameters of at least a portion of the neural network model from the first memory, and calculate the first vector and the second vector; The first transceiver is configured to send the first vector and the second vector to the second processing device.

4. The processing device according to claim 1, wherein The second processing device includes: a second memory, a second processor, and a second transceiver, in, The second transceiver is configured to: receive the first vector and the second vector; The second processor is configured to: calculate the third vector based on the first vector and the second vector; The second memory is configured to: store the third vector; The second transceiver is further configured to: transmit the third vector.

5. The processing device according to claim 1, wherein At least two of the at least one second processing device implement different applications.

6. The processing device according to claim 1, wherein At least a portion of the model parameters of the neural network model include at least: all or part of the weight parameters corresponding to the feedforward neural network model.

7. The processing device according to claim 1, wherein At least a portion of the model parameters of the neural network model include: weight parameters corresponding to the query matrix, weight parameters corresponding to the key matrix, and weight parameters corresponding to the value matrix. The first vector is divided into a sub-vector of the first vector on the query dimension, a sub-vector on the key dimension, and a sub-vector on the value dimension. Calculating the first vector corresponding to the first input information includes: Calculating a subvector of the first vector in the query dimension based on a weight parameter corresponding to the query matrix; Calculating a subvector of the first vector on the key dimension based on a weight parameter corresponding to the key matrix; Based on the weight parameters corresponding to the value matrix, a sub-vector of the first vector in the value dimension is calculated.

8. The processing device according to claim 7, wherein The second vector is divided into a sub-vector of the second vector on the query dimension, a sub-vector on the key dimension, and a sub-vector on the value dimension. Calculating the second vector corresponding to the second input information includes: Calculating a subvector of the second vector in the query dimension based on a weight parameter corresponding to the query matrix; Calculating a subvector of the second vector on the key dimension based on a weight parameter corresponding to the key matrix; Based on the weight parameters corresponding to the value matrix, a subvector of the second vector in the value dimension is calculated.

9. The processing device according to claim 8, wherein The calculating a third vector based on the first vector and the second vector includes: Splitting the subvector of the first vector on the query dimension, the subvector on the key dimension, and the subvector on the value dimension into multiple head vectors of the first vector on the query dimension, multiple head vectors on the key dimension, and multiple head vectors on the value dimension, respectively; Splitting the subvector of the second vector on the query dimension, the subvector on the key dimension, and the subvector on the value dimension into a plurality of head vectors of the second vector on the query dimension, a plurality of head vectors on the key dimension, and a plurality of head vectors on the value dimension, respectively; and The third vector is calculated based on the respective head vectors of the first vector and the respective head vectors of the second vector.

10. The processing device according to claim 1, wherein The second processing device is configured to calculate a third vector based on the first vector and the second vector, comprising: Calculating a subvector of the first vector in the query dimension, a subvector in the key dimension, and a subvector in the value dimension based on a weight parameter corresponding to the query matrix, a weight parameter corresponding to the key matrix, and a weight parameter corresponding to the value matrix; Calculating a subvector of the second vector in the query dimension, a subvector in the key dimension, and a subvector in the value dimension based on a weight parameter corresponding to the query matrix, a weight parameter corresponding to the key matrix, and a weight parameter corresponding to the value matrix; Splitting the subvector of the first vector on the query dimension, the subvector on the key dimension, and the subvector on the value dimension into multiple head vectors of the first vector on the query dimension, multiple head vectors on the key dimension, and multiple head vectors on the value dimension, respectively; Splitting the subvector of the second vector on the query dimension, the subvector on the key dimension, and the subvector on the value dimension into a plurality of head vectors of the second vector on the query dimension, a plurality of head vectors on the key dimension, and a plurality of head vectors on the value dimension, respectively; and The third vector is calculated based on the respective head vectors of the first vector and the respective head vectors of the second vector.

11. The processing device according to any one of claims 8 to 10, wherein The calculating a third vector based on the first vector and the second vector further includes: performing a dimensionality reduction operation on a subvector of the first vector on the query dimension and a subvector of the first vector on the key dimension, and storing the subvector of the first vector on the query dimension and the subvector of the first vector on the key dimension after the dimensionality reduction in the second memory; Performing a dimensionality reduction operation on a subvector of the second vector on the query dimension and a subvector of the second vector on the key dimension, and storing the subvector of the second vector on the query dimension and the subvector of the second vector on the key dimension after the dimensionality reduction in the second memory; and The third vector is calculated based at least in part on a reduced subvector of the first vector in the query dimension, a reduced subvector of the first vector in the key dimension, a reduced subvector of the second vector in the query dimension, and a reduced subvector of the second vector in the key dimension.

12. The processing apparatus according to any one of claims 8 to 10, wherein: The calculating a third vector based on the first vector and the second vector includes: Calculating a set of attention weights based on a plurality of head vectors of the first vector in the query dimension, a plurality of head vectors of the second vector in the query dimension, a plurality of head vectors of the first vector in the key dimension, and a plurality of head vectors of the second vector in the key dimension; and Based on the attention weight set, a weighted sum is performed on multiple head vectors of the first vector in the value dimension and multiple head vectors of the second vector in the value dimension to obtain a third vector.

13. The processing device according to claim 9 or 10, wherein: Calculating the third vector based on each head vector of the first vector and each head vector of the second vector includes: Traverse multiple head vectors of the second vector in the query dimension, for the i-th head vector of the second vector in the query dimension, where i is an integer: Calculating a first attention weight of an i-th head vector of the second vector in the query dimension relative to an i-th head vector of the first vector in the key dimension; calculating a second attention weight of an i-th head vector of the second vector in the query dimension relative to an i-th head vector of the second vector in the key dimension; Based on the first attention weight and the second attention weight, calculating a weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension as the i-th head vector of the third vector; and All head vectors of the third vector are concatenated into the third vector.

14. The processing device according to claim 9 or 10, wherein: Calculating the third vector based on each head vector of the first vector and each head vector of the second vector includes: Traverse multiple head vectors of the second vector in the query dimension, for the i-th head vector of the second vector in the query dimension, where i is an integer: Calculating a first attention weight of an i-th head vector of the second vector in the query dimension relative to an i-th head vector of the first vector in the key dimension; calculating a second attention weight of an i-th head vector of the second vector in the query dimension relative to an i-th head vector of the second vector in the key dimension; Based on the first attention weight and the second attention weight, calculating a weighted sum of the i-th head vector of the first vector in the value dimension and the i-th head vector of the second vector in the value dimension as the i-th head vector of the third vector; and A third vector is calculated based on the output projection matrix and all head vectors of the third vector.

15. The processing device of claim 13, wherein: The calculating a first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension includes: Calculating a first dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the first vector in the key dimension; and The first dot product is scaled and normalized to determine the first attention weight.

16. The processing device of claim 14, wherein: The calculating a first attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the first vector on the key dimension includes: Calculating a first dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the first vector in the key dimension; and The first dot product is scaled and normalized to determine the first attention weight.

17. The processing device of claim 13, wherein: The calculating a second attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the second vector on the key dimension includes: Calculating a second dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and The second dot product is scaled and normalized to determine the second attention weight.

18. The processing device of claim 14, wherein: The calculating a second attention weight of the i-th head vector of the second vector on the query dimension relative to the i-th head vector of the second vector on the key dimension includes: Calculating a second dot product of the i-th head vector of the second vector in the query dimension and the i-th head vector of the second vector in the key dimension; and The second dot product is scaled and normalized to determine the second attention weight.

19. The processing device of claim 1, wherein: The first processing device is further configured to: Output data is calculated based on the model parameters of at least a portion of the neural network model and the third vector.

20. The processing apparatus of claim 3, wherein: The first transceiver is further configured to: receive a third vector; The first processor chip is also configured to: obtain model parameters of at least a portion of the neural network model from the first memory, and calculate output data based on the model parameters of at least a portion of the neural network model and the third vector.

21. The processing device of claim 1, wherein: The at least one second processing device includes at least two heterogeneous second processing devices, and the at least two heterogeneous second processing devices are communicatively coupled to the first processing device.

22. The processing apparatus of claim 1, wherein: The second processing device is communicably coupled to the first processing device via a bus, or the second processing device is communicably coupled to the first processing device via network transmission.

23. The processing device of claim 3, wherein: The first memory has a data reading speed higher than a data writing speed.

24. The processing apparatus of claim 3, wherein: The first memory is a flash memory.

25. The processing apparatus of claim 4, wherein: The second storage is a memory.

26. The processing apparatus of claim 1, wherein: The at least one second processing device shares at least a portion of model parameters of the neural network model in the first processing device.

27. The processing apparatus of claim 1, wherein: The at least one second processing device shares computing power and storage capacity of the first processing device.

28. The processing apparatus of claim 1, wherein: The first processing device is an attachment device of the electronic processing equipment.

29. The processing apparatus of claim 3, wherein: The first processor chip is further configured to: compress the first vector and the second vector to obtain a compressed first vector and a compressed second vector; The sending of the first vector and the second vector to the second processing device includes sending the compressed first vector and the compressed second vector to the second processing device.

30. The processing apparatus of claim 4, wherein: The second processor is further configured to: compress the third vector to obtain the compressed third vector; as well as The sending of the third vector includes: sending the compressed third vector to the first processing device.

31. The processing apparatus of claim 1, wherein: The first processing device is implemented as a mobile phone, a portable computer, a desktop computer, or a workstation; or the first processing device is implemented as a component of at least one of the following: a component in a mobile phone, a portable computer, a desktop computer, or a workstation, The at least one second processing device includes at least one of the following: smart headphones, smart glasses, smart watches; or the at least one second processing device includes a component of at least one of the following: a mobile phone, a portable computer, a desktop computer or a workstation.

32. The processing apparatus of claim 1, wherein: The first processing device is implemented as a vehicle master computing device or a component in the vehicle master computing device. The at least one second processing device includes at least one of the following: a driving assistance end-side device, an in-vehicle infotainment device, an autonomous driving end-side device, a digital cockpit device, or a domain control device; or the at least one second processing device includes a component in a domain control device or a vehicle master end computing device.

33. The processing apparatus of claim 1, wherein: The first processing device is implemented as a smart home control device or a gateway device; or the first processing device is implemented as a component in a smart home control device or a gateway device, The at least one second processing device includes at least one of the following: a smart speaker device, a smart voice device, a smart camera device, a voice control device; or the at least one second processing device includes a component in a smart home control device or a gateway device.

34. The processing apparatus of claim 3, wherein: The first processor chip is configured as follows: Based on the third vector, updating model parameters of at least a portion of the neural network model stored in the first memory; and The first vector and the second vector are calculated based on at least a portion of the model parameters of the updated neural network model.

35. A processing method performed by a processing device, wherein the processing device comprises a first processing device and at least one second processing device, wherein: The at least one second processing device is physically separated from the first processing device and communicatively coupled to the first processing device, and the processing method includes: Based on at least a portion of model parameters of the neural network model and first input information, the first processing device calculates a first vector corresponding to the first input information; Based on at least a portion of the model parameters of the neural network model and the second input information, the first processing device calculates a second vector corresponding to the second input information; and A third vector is calculated by the second processing device based on the first vector and the second vector.

36. A first processing device comprising a first memory, at least one first processor chip, and a first transceiver, in, The first memory is configured to: store model parameters of at least a portion of the neural network model; The first processor chip is configured as follows: Calculating a first vector corresponding to the first input information based on at least a portion of model parameters of the neural network model and the first input information; as well as Calculating a second vector corresponding to the second input information based on at least a portion of model parameters of the neural network model and the second input information; The first transceiver is configured to: send the first vector and the second vector to the second processing device, The second processing device is physically separated from the first processing device and is communicatively coupled to the first processing device.

37. A second processing device, comprising: a second memory, a second processor, and a second transceiver, in, The second transceiver is configured to: receive the first vector and the second vector from the first processing device according to claim 36; The second processor is configured to: calculate the third vector based on the first vector and the second vector; The second memory is configured to: store the third vector; The second transceiver is further configured to send the third vector to the first processing device.

Images