Data processing method, device and equipment

Abstract

The invention provides a data processing method, device and equipment, relates to the technical field of communication, and is used for protecting the safety of model information in an operation period under the condition of not using hardware isolation technologies such as TEE and the like. The method is applied to the data acceleration processing device, the data acceleration processing device comprises an acceleration processor and a memory, and the memory comprises a non-protected storage area and a protected storage area. The method comprises the steps that the memory receives model information sent by the universal processing device and stores the model information to the non-protection storage area, the universal processing device can access the non-protection storage area, and the model information comprises a model weight in a ciphertext form; and the acceleration processor decrypts the model weight in the ciphertext form and stores the decrypted model weight in the plaintext form to the protected storage area, and the universal processing device cannot access the protected storage area.

Description

Technical Field

[0001] The present application relates to the field of communication technology, and in particular to a data processing method, device and equipment. Background Art

[0002] With the popularization and application of artificial intelligence (AI) in various industries, the protection of AI models is becoming more and more important. The protection of AI models can mainly include the protection of storage, transmission and operation stages. Among them, how to achieve protection in the operation stage has always been a difficult problem in the industry.

[0003] At present, in order to protect the security of AI models, electronic devices with confidential computing capabilities are usually used. Taking the electronic device including a central processing unit (CPU) and a graphics processing unit (GPU), and the CPU and GPU are connected through a peripheral component interconnect express (PCIe) bus as an example, the CPU is required to support the trusted execution environment (TEE) capability, the GPU to support multi-instance isolation capability, and the PCIe bus to support encryption and decryption transmission capabilities, so as to achieve a hardware isolation environment across the CPU and GPU and ensure the security of the AI ​​model running therein.

[0004] However, this solution has high requirements on the hardware of electronic devices, such as requiring the hardware of electronic devices to support isolation capabilities and encryption and decryption transmission capabilities. It is difficult to apply to related products already on the market, and thus its use is limited. Summary of the invention

[0005] The present application provides a data processing method, apparatus and device for protecting the security of model information during runtime without using hardware isolation technologies such as TEE.

[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0007] In a first aspect, a data processing method is provided, which is applied to a data acceleration processing device (i.e., a device with a data acceleration processing function, such as a GPU or an NPU, etc.), wherein the data acceleration processing device includes an acceleration processor and a memory, wherein the memory includes a non-protected storage area and a protected storage area, and the data acceleration processing device can communicate with a general processing device, and the method includes: the memory receives model information sent by the general processing device, and stores the model information in the non-protected storage area, and the general processing device can access the non-protected storage area, and the model information includes model weights in ciphertext form; the acceleration processor decrypts the model weights in ciphertext form, and stores the decrypted model weights in plaintext form in the protected storage area, and the general processing device cannot access the protected storage area.

[0008] In the above technical solution, the general processing device can send model information to the data acceleration processing device, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the corresponding computing tasks according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related technology, the embodiments of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0009] In a possible implementation of the first aspect, the specific process of the acceleration processor decrypting the model weight in ciphertext form can be performed by a corresponding accelerator or operator in the acceleration processor. For example, the acceleration processor may include a decryption operator AICPU, which can be used to decrypt the model weight in ciphertext form. In addition, a TEE environment is also provided in the data acceleration processing device, and the process of the acceleration processor decrypting the model weight in ciphertext form can be performed in the TEE environment.

[0010] In a possible implementation of the first aspect, the data acceleration processing device further includes a persistent memory, the persistent memory stores a decryption key; the acceleration processor decrypts the ciphertext model weight, including: the acceleration processor obtains the decryption key from the persistent memory, and decrypts the ciphertext model weight according to the decryption key. In the above possible implementation, the security of the model weight is ensured by storing the decryption key in the persistent memory and decrypting the ciphertext model weight according to the decryption key.

[0011] In a possible implementation of the first aspect, the data acceleration processing device further includes a persistent memory, the persistent memory stores a signature verification public key, and the model information further includes signature information; the method further includes: the acceleration processor obtains the signature verification public key from the persistent memory, and determines that the signature information of the model information is successfully verified according to the signature verification public key. In the above possible implementation, the integrity of the model information can be guaranteed by signing and verifying the model information.

[0012] In a possible implementation of the first aspect, the decryption key and / or the signature verification public key are encrypted and stored, and the data acceleration processing device also includes a one-time programmable memory, which stores a hardware unique key HUK; the method also includes: the acceleration processor obtains the HUK from the one-time programmable memory, and encrypts and decrypts the decryption key and / or the signature verification public key according to the derived key of the HUK. In the above possible implementation, by encrypting and storing the decryption key and / or the signature verification public key, the confidentiality of the decryption key and / or the signature verification public key can be guaranteed, thereby ensuring the security of the model weight.

[0013] In a possible implementation manner of the first aspect, the model weight and signature information in ciphertext form in the model information may be generated in a development environment.

[0014] In a possible implementation of the first aspect, the model weight is stored continuously in a protected storage area; the method further includes: the acceleration processor sends the base address of the model weight in plain text in the protected storage area to the general processing device, and the base address is used to determine the logical address of the model weight; the acceleration processor receives the logical address of the model weight sent by the general processing device, and obtains the model weight in plain text from the protected storage area according to the logical address. In the above possible implementation, the acceleration processor can send the base address of the model weight in plain text in the protected storage area to the general processing device, so that the general processing device determines the logical address according to the base address and the address offset value of the model weight, and sends it to the acceleration processor, so that the general processing device does not need to read the model weight when scheduling the data acceleration processing device to perform a computing task, but only needs to send the logical address of the model weight, thereby ensuring the security of the model weight.

[0015] In a possible implementation of the first aspect, the method further includes: the acceleration processor receives the address offset value of the model weight sent by the general processing device; the acceleration processor determines the logical address of the model weight according to the base address of the model weight in plain text in the protected storage area and the address offset value. In the above possible implementation, when the general processing device schedules the data acceleration processing device to perform a computing task, it does not need to read the model weight, but only needs to send the address offset value of the model weight, thereby ensuring the security of the model weight.

[0016] In a possible implementation of the first aspect, the method further includes: the acceleration processor receives memory configuration information, and configures the non-protected storage area and / or the protected storage area in the memory according to the memory configuration information. In the above possible implementation, the security of the model weight can be ensured by configuring a protected storage area in the memory of the data acceleration processing device that is inaccessible to the general processing device.

[0017] In a second aspect, a data processing method is provided, which is applied to a general processing device, the general processing device communicates with a data acceleration processing device through a bus, the memory of the data acceleration processing device includes an unprotected storage area accessible to the general processing device, and a protected storage area inaccessible to the general processing device, the method comprising: obtaining model information, the model information including model weights in ciphertext form; sending the model information to the acceleration processor so that the memory of the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor decrypts the ciphertext model weights and stores the model weights in plaintext in the protected storage area.

[0018] In the above technical solution, the general processing device can transmit model information to the data acceleration processing device, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the calculation task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related technology, the embodiments of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0019] In a possible implementation of the second aspect, the model information further includes signature information. In the above possible implementation, the integrity of the model information can be ensured by signing and verifying the signature of the model information.

[0020] In a possible implementation of the second aspect, the method further includes: the general processing device receives the base address of the model weight in plain text form in the protected storage area sent by the data acceleration processing device; the general processing device determines the logical address of the model weight according to the base address and the address offset value of the model weight; the general processing device sends the logical address of the model weight to the data acceleration processing device. In the above possible implementation, the general processing device can receive the base address of the model weight in plain text form in the protected storage area sent by the data acceleration processing device, determine the logical address according to the base address and the address offset value of the model weight, and send it to the data acceleration processing device, so that the general processing device does not need to read the model weight when scheduling the data acceleration processing device to perform a computing task, but only needs to read the logical address of the model weight, thereby ensuring the security of the model weight.

[0021] In a possible implementation of the second aspect, the method further includes: the general processing device sends the address offset value of the model weight to the data acceleration processing device, so that the data acceleration processing device determines the logical address of the model weight according to the base address of the model weight in plain text in the protected storage area and the address offset value. In the above possible implementation, when scheduling the data acceleration processing device to perform a computing task, the general processing device does not need to read the model weight, but only needs to send the address offset value of the model weight, thereby ensuring the security of the model weight.

[0022] In a possible implementation of the second aspect, the method further includes: sending memory configuration information to the data acceleration processing device through the management channel, the memory configuration information being used to configure the non-protected storage area and / or the protected storage area. In the above possible implementation, the security of the model weights can be ensured by configuring a protected storage area in the memory of the data acceleration processing device that is inaccessible to the general processing device.

[0023] According to a third aspect, a data processing method is provided, which is applied to a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area and a protected storage area, the unprotected storage area includes model information, the model information includes model weights in ciphertext form, the protected storage area includes model weights in plaintext form after the ciphertext form is decrypted, the method includes: the acceleration processor obtains a target task sent by a general processing device based on the model information; the acceleration processor obtains a first logical address of the target weight, and reads the target weight from the protected storage area according to the first logical address, the target weight is a weight in the model weight required to be used when the target task is executed, and the general processing device cannot access the protected storage area; the acceleration processor executes the target task according to the target weight.

[0024] In the above technical solution, the data acceleration processing device stores the model information including the model weights in ciphertext form in a non-protected storage area, and stores the corresponding decrypted model weights in plaintext form in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the model weights in plaintext form. In this way, when the data acceleration processing device executes the corresponding target task according to the model weights in plaintext form, the security of the model weights during execution can be ensured.

[0025] In a possible implementation of the third aspect, the acceleration processor obtains the logical address of the target weight, including: the acceleration processor determines the base address of the model weight in plain text in the protected storage area; the acceleration processor sends the base address to the general processing device, and the base address is used to determine the first logical address of the target weight; the acceleration processor receives the first logical address of the target weight sent by the general processing device. In the above possible implementation, when the data acceleration processing device executes the corresponding target task, it does not need to read the target weight, but only needs to send the first logical address of the target weight, thereby ensuring the security of the model weight.

[0026] In a fourth aspect, a data acceleration processing device is provided, which includes a processing unit and a storage unit, the storage unit including a non-protected storage area and a protected storage area; the storage unit is used to receive model information sent by a general processing device, and store the model information in the non-protected storage area, and the general processing device can access the non-protected storage area, the model information includes model weights in ciphertext form; the processing unit is used to decrypt the model weights in ciphertext form, and store the decrypted model weights in plaintext form in the protected storage area, and the general processing device cannot access the protected storage area.

[0027] In a possible implementation of the fourth aspect, the data acceleration processing device also includes a persistent memory, which stores a decryption key; the processing unit is also used to: obtain the decryption key from the persistent memory, and decrypt the model weight in ciphertext form according to the decryption key.

[0028] In a possible implementation of the fourth aspect, the data acceleration processing device also includes a persistent memory, which stores a signature verification public key, and the model information also includes signature information; the processing unit is also used to: obtain the signature verification public key from the persistent memory, and determine whether the signature information verification of the model information is successful based on the signature verification public key.

[0029] In a possible implementation of the fourth aspect, the decryption key and / or the signature verification public key is stored in an encrypted manner, and the data acceleration processing device also includes a one-time programmable memory, which stores a hardware unique key HUK; the processing unit is also used to: obtain the HUK from the one-time programmable memory, and encrypt and decrypt the decryption key and / or the signature verification public key according to the derived key of the HUK.

[0030] In a possible implementation of the fourth aspect, the model weight is stored continuously in the protected storage area; the processing unit is also used to: send the base address of the model weight in plain text in the protected storage area to the general processing device, and the base address is used to determine the logical address of the model weight; receive the logical address of the model weight sent by the general processing device, and obtain the model weight in plain text from the protected storage area according to the logical address.

[0031] In a possible implementation manner of the fourth aspect, the processing unit is further used to: receive memory configuration information, and configure the unprotected storage area and / or the protected storage area according to the memory configuration information.

[0032] In a fifth aspect, a general processing device is provided, which is used to communicate with a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area accessible to the general processing device, and a protected storage area inaccessible to the general processing device, the general processing device includes: a processing unit, used to obtain model information, the model information includes model weights in ciphertext form; a sending unit, used to send the model information to the data acceleration processing device, so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor decrypts the ciphertext model weights and stores them in plaintext in the protected storage area.

[0033] In a possible implementation manner of the fifth aspect, the model information also includes signature information, and the signature information is used to verify the model information.

[0034] In a possible implementation of the fifth aspect, the model weight is stored continuously in the protected storage area, and the general processing device also includes a receiving unit; the receiving unit is used to receive the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; the processing unit is also used to determine the logical address of the model weight based on the base address and the address offset value of the model weight; the sending unit is also used to send the logical address of the model weight to the data acceleration processing device.

[0035] In a possible implementation manner of the fifth aspect, the sending unit is further used to: send memory configuration information to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

[0036] In a sixth aspect, a data acceleration processing device is provided, which includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; the storage unit is used to store model information in the non-protected storage area, and the model information includes model weights in ciphertext form, and the general processing device cannot access the protected storage area; the storage unit is also used to store model weights in plaintext corresponding to the model weights in ciphertext form in the protected storage area, and the general processing device cannot access the protected storage area; the processing unit is used to obtain a target task sent by the general processing device based on the model information; the processing unit is also used to obtain a first logical address of the target weight, and read the target weight from the protected storage area according to the first logical address, the target weight being the weight in the model weight that needs to be used when the target task is executed; the processing unit is also used to execute the target task according to the target weight.

[0037] In a possible implementation of the sixth aspect, the data acceleration processing device also includes a sending unit and a receiving unit; the processing unit is also used to determine the base address of the model weight in plain text in the protected storage area; the sending unit is also used to send the base address of the target weight to the general processing device, and the base address is used to determine the first logical address of the target weight; the receiving unit is used to receive the first logical address of the target weight sent by the general processing device.

[0038] In the seventh aspect, a data acceleration processing device is provided, which includes an acceleration processor and a memory, in which instructions are stored. When the acceleration processor runs the instructions, the device executes the data processing method provided by the first aspect or any possible implementation of the first aspect, or executes the data processing method provided by the third aspect or any possible implementation of the third aspect.

[0039] In an eighth aspect, a general processing device is provided, which includes a processor and a memory, wherein instructions are stored in the memory, and when the processor runs the instructions, the device executes the data processing method provided in the second aspect or any possible implementation of the second aspect.

[0040] Another aspect of the present application provides an electronic device, which includes: a data acceleration processing device as provided in any of the above aspects, and a general processing device as provided in any of the above aspects.

[0041] In another aspect of the present application, a terminal device is provided, the terminal device comprising: a data acceleration processing device as provided in any of the above aspects, and a general processing device as provided in any of the above aspects. Exemplarily, the terminal device may be a mobile phone, a tablet, a computer, a camera, a wearable device, or a vehicle-mounted device.

[0042] According to another aspect of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the data processing method provided by the first aspect or any possible implementation of the first aspect.

[0043] According to another aspect of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the data processing method provided in the second aspect or any possible implementation of the second aspect.

[0044] According to another aspect of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the data processing method provided by the third aspect or any possible implementation of the third aspect.

[0045] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method provided in the first aspect or any possible implementation of the first aspect.

[0046] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method provided in the second aspect or any possible implementation of the second aspect.

[0047] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method provided in the third aspect or any possible implementation of the third aspect.

[0048] It can be understood that the beneficial effects that can be achieved by any of the data acceleration processing devices, general processing devices, electronic devices, terminal devices, computer-readable storage media and computer program products provided above can correspond to the beneficial effects in the data processing method provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A schematic diagram for implementing runtime protection of an AI model;

[0050] Figure 2 A schematic diagram of another method for implementing runtime protection of an AI model;

[0051] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0052] Figure 4 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application;

[0053] Figure 5 A flowchart of a data processing method provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of a model information provided in an embodiment of the present application;

[0055] Figure 7 A flowchart of another data processing method provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of a model weight in plain text provided in an embodiment of the present application;

[0057] Fig. 9 A schematic diagram of a model weight in ciphertext form provided in an embodiment of the present application;

[0058] Fig.10A schematic diagram of data processing provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of a configuration data acceleration processing device provided in an embodiment of the present application;

[0060] Fig.12 A schematic diagram of another data processing provided in an embodiment of the present application;

[0061] Fig.13 A flowchart of another data processing method provided in an embodiment of the present application;

[0062] Fig.14 A schematic diagram of the structure of a data acceleration processing device provided in an embodiment of the present application;

[0063] Fig.15 A schematic diagram of the structure of another data acceleration processing device provided in an embodiment of the present application;

[0064] Fig.16 A schematic diagram of the structure of a general processing device provided in an embodiment of the present application;

[0065] Fig.17 A schematic diagram of the structure of another general processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. In the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c or a, b and c, where a, b and c can be single or multiple.

[0067] The embodiments of the present application use words such as "first" and "second" to distinguish objects with similar names, functions or effects. Those skilled in the art will understand that words such as "first" and "second" do not limit the quantity and execution order. The term "coupled" is used to indicate electrical connection, including direct connection through wires or connection terminals or indirect connection through other devices. Therefore, "coupled" should be regarded as a broad electronic communication connection.

[0068] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in this application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0069] Before introducing the embodiments of the present application, the relevant background technology involved in the present application is first introduced and explained.

[0070] With the popularization and application of artificial intelligence (AI) in all walks of life, the protection of AI models is becoming more and more important. The protection of AI models can mainly include protection in three stages: storage, transmission and operation. Among them, how to achieve protection in the operation stage has always been a difficult problem in the industry. In related technologies, the following two methods are usually used to achieve protection in the operation stage of AI models, which can also be called protection during the operation period.

[0071] The first is to use electronic devices with confidential computing capabilities. Figure 1As shown, taking an electronic device including a central processing unit (CPU) and a graphics processing unit (GPU), and the CPU and GPU are connected via a peripheral component interconnect express (PCIe) bus as an example, the CPU is required to support a trusted execution environment (TEE) capability, the GPU supports multi-instance isolation capability, and the PCIe bus supports encryption and decryption transmission capability. Specifically, each TEE of the CPU is configured with a confidential virtual machine (VM), and the confidential VM is provided with a GPU driver, which is encrypted and transmitted via the PCIe bus. The GPU includes a secure GPU instance isolated by a firewall, and each secure GPU instance is correspondingly configured with a PCIe virtual function (VF), which can be used to support the secure GPU instance for transmission via the PCIe bus. Therefore, in the electronic device, a confidential VM in the CPU, a secure GPU instance in the GPU, and the corresponding PCIe VF can form a hardware isolation environment (or called a secure environment) across the CPU and GPU, thereby ensuring the security of the AI ​​model running therein.

[0072] In the above solution, the CPU can be used to process the input of AI applications, load model files, etc.; the GPU can be used to perform corresponding AI computing tasks; and the PCIe bus can be used to transmit interactive data streams between the CPU and the GPU. That is, this solution covers three parts of data flow design. In the first part, TEE isolation capability is used on the CPU, so that non-secure environments cannot access AI applications and AI models. By connecting to a remote attestation server or local attestation, it can be ensured that the operating status of the GPU environment connected to the TEE meets the requirements, such as the correct firmware version. In the second part, the operating status on the GPU can be measured and can be perceived by the CPU; multi-instance isolation provides hardware isolation capabilities to ensure that AI computing tasks in secure environments and AI computing tasks in non-secure environments are isolated and do not affect each other. In the third part, the PCIe bus uses encrypted transmission to ensure the confidentiality and integrity of the data flow between the CPU and the GPU.

[0073] However, this solution has high requirements on the hardware of electronic devices. For example, it requires the hardware of electronic devices to support isolation capabilities and encryption and decryption transmission capabilities. It is difficult to apply to related products already on the market (for example, electronic devices or chips). In other words, it is impossible to achieve the runtime protection capability of the AI ​​model only through software enhancement, and thus its use is limited.

[0074] The second method is to abstract the units of model-related operations in TEE and combine them with the existing computing and storage resources in TEE to implement parameter protection of AI models during operation. Figure 2 As shown, the electronic device includes a trusted execution environment TEE and a non-trusted execution environment non-TEE. The TEE includes a configuration manager, a model processing unit, a model management unit, an attestation manager, a secure storage and a computation processing unit. The non-TEE includes an AI service, an application and an accelerator. Among them, the model management unit can be used to communicate with the configuration manager, the model processing unit, the attestation manager and the secure storage, and the model processing unit can be used to communicate with the AI ​​service, and to schedule the computation processing unit and the accelerator to perform corresponding calculations. The AI ​​model in the above electronic device can be configured after development and training.

[0075] The above solution makes significant changes to the software stack and does not protect the relevant information of the AI ​​model sent between the model processing unit and the accelerator.

[0076] Based on this, an embodiment of the present application provides a data processing method, which can be used to protect the security of AI models and prevent theft during offline reasoning without using hardware isolation technologies such as TEE. The method can be applied to electronic devices, including but not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MID), wearable devices (such as smart watches and smart bracelets), vehicle-mounted devices (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electric meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, or wireless terminals in smart homes, flight equipment (such as intelligent robots, hot air balloons, drones, airplanes), etc. The above electronic devices can also be called terminal devices.

[0077] The specific structure of the electronic device is described below.

[0078] Figure 3 The schematic diagram of the structure of an electronic device provided in an embodiment of the present application includes: a general processing device 100, a data acceleration processing device 200 and a bus 300, wherein the general processing device 100 and the data acceleration processing device 200 are coupled via the bus 300. The general processing device 100 may refer to a device having general data processing, execution of instructions or commands, and task delivery capabilities, and the general processing device 100 may also be referred to as a host, a general server, or a master device, etc. The general processing device 100. The data acceleration processing device 200 may refer to a device having data acceleration processing capabilities, and the data acceleration processing device 200 may also be referred to as a device, a dedicated processing device, or a slave device, etc.

[0079] The general processing device 100 may include one or more general processors, which may include but are not limited to: a central processing unit CPU, a digital signal processor (DSP), a microcontroller or a microprocessor. The general processor may be used to execute various functions of the electronic device and process data, etc., so as to monitor the electronic device as a whole, such as for processing the operating system, user interface and application program of the electronic device. In some embodiments, the general processing device 100 may also include a memory (such as a memory and an external memory, etc.), which may be used to store data, software programs and modules related to the general processing device 100, such as the software program may include an inference framework. For example, the memory may include a program storage area and a data storage area; wherein the program storage area may store software programs, including instructions formed in code, including but not limited to an operating system, an application program required for at least one function, such as a sound playback function, an image playback function, etc.; the data storage area may store data created according to the use of the electronic device, such as audio data, image data, text data, etc. Figure 3 In the description, the general processing device 100 includes a CPU and a first memory as an example. The first memory can be DRAM, DDR or other memory.

[0080] The data acceleration processing device 200 may include one or more acceleration processors, which may include but are not limited to: graphics processing unit (GPU), neural network processing unit (NPU), DSP, CPU, application specific integrated circuit (ASIC), complex programmable logic device (CPLD), etc. The acceleration processor may include one or more accelerators (e.g., vector calculation unit, matrix calculation unit and / or scalar calculation unit, etc.), and one or more operators (e.g., decryption operator AICPU, or other custom operators, etc.), and the acceleration processor may be used to perform corresponding calculations (e.g., vector calculations, matrix calculations and / or scalar calculations, etc.) under the call of the general processing device 100. In some embodiments, the data acceleration processing device 200 may also include a memory, which may be used to store data, software programs and modules related to the data acceleration processing device 200, such as the software program may include firmware. In one example, the memory in the data acceleration processing device 200 may include a second memory, which may include an unprotected storage area accessible to the general processing device 100 and a protected storage area inaccessible to the general processing device 100. Furthermore, the memory may also include a persistent memory and a one-time programmable memory (efuse), etc. For example, the persistent memory may include a flash memory, which may be used to store the decryption key and / or the signature verification public key described below. The one-time programmable memory refers to a memory that is only allowed to be written once (or is called a memory that is only allowed to be programmed once). In an embodiment of the present application or otherwise, the one-time programmable memory may be used to store the hardware unique key (HUK) described below. Figure 3 The data acceleration processing device 200 including the NPU and the second memory is taken as an example for explanation.

[0081] It is understandable that the memory (e.g., the first memory and the second memory) in the embodiments of the present application may refer to the internal memory of the corresponding device, which can be used to temporarily store data during the operation of the processor of the corresponding device (e.g., the general processor and the accelerated processor), and has a high access efficiency and a small capacity. The stored data will usually be lost after power failure. The above-mentioned persistent memory and one-time programmable memory may refer to external memory, which can be used for long-term storage of data, has a relatively low access efficiency and a large capacity, and the stored data will not be lost after power failure. The persistent memory may be a memory that supports multiple writes, and the one-time programmable memory refers to a memory that allows only one write.

[0082] The bus 300 may be used to transmit information between the general processing device 100 and the data acceleration processing device 200, so as to realize communication between the general processing device 100 and the data acceleration processing device 200. In some embodiments, the bus 300 may be a PCIe bus, or an extended industry standard architecture (EISA) bus, etc.

[0083] Although not shown, the general processing device 100 may also include other components such as sensor components (e.g., acceleration sensors, gyroscope sensors, pressure sensors and / or temperature sensors, etc.), multimedia components (e.g., display panels and cameras, etc.), input and output devices (e.g., mice and keyboards, etc.), communication modules (e.g., Bluetooth modules and WiFi modules) and power supply components, which are not described in detail in the embodiments of the present application.

[0084] Furthermore, the electronic device processing includes the above-mentioned hardware resources, and may also include a software architecture running on the above-mentioned hardware resources (such as a processor and a memory, etc.). In some embodiments of the present application, the software architecture of the general processing device 100 in the electronic device may include an application layer (such as an AI application), an application framework layer, a function library layer, and a kernel layer. In one example, Figure 4 As shown, the software architecture may include an inference framework and a device management driver; the inference framework may be located in the above-mentioned application framework layer, and the inference framework may be used to provide various APIs used by application programs for access; the device management driver may be located in the above-mentioned kernel layer, and the device management driver may be used to configure the second memory in the data acceleration processing device 200, such as for configuring the above-mentioned protected storage area and non-protected storage area. In other embodiments of the present application, such as Figure 4 As shown, the data acceleration processing device 200 may also include a corresponding software architecture, for example, the software architecture may include firmware running on an acceleration processor.

[0085] It is understandable that the above Figure 3 and Figure 4 The hardware resources and software architecture of the electronic device shown are only exemplary. In actual applications, the electronic device may also include components or software that are more than or different from those shown in the diagram. The above examples do not constitute a limitation to the embodiments of the present application.

[0086] Figure 5 A flow chart of a data processing method provided in an embodiment of the present application, the method comprises the following steps. The data processing can be applied to the electronic device provided above, the electronic device comprising a general processing device and a data acceleration processing device. Hereinafter, the general processing device comprises a general processor (e.g., a CPU) and a first memory, and the data acceleration processing device comprises an acceleration processor (e.g., an NPU) and a second memory as an example for explanation.

[0087] S301: The general processing device obtains model information, where the model information includes a model weight in a ciphertext form.

[0088] Among them, the model information may be model information of an AI model. The model information may be stored in the first memory of the general processing device. The model weight (weight) in the ciphertext form in the model weight may refer to the encrypted model weight. Optionally, the model information may also include multiple computing tasks. The model weight includes the weights that the multiple computing tasks need to use during the calculation process. The multiple computing tasks may be in plain text in the model information, and the multiple computing tasks may specifically be computing tasks that need to be scheduled for execution by a data acceleration processing device, for example, the multiple computing tasks may be AI computing tasks. Optionally, the model information may also include other parameters in addition to the model weight, such as the input parameters and output parameters required by the model, etc. The other parameters may also be in ciphertext form. The embodiment of the present application takes the model weight in ciphertext form as an example for explanation.

[0089] In a possible embodiment, the model information is stored in the first memory of the general processing device, and the general processor of the general processing device can obtain the model information from the first memory. Furthermore, the general processor of the general processing device can also parse and convert the model information so that the converted model information can be recognized and processed by the data acceleration processing device.

[0090] Optionally, the model weights may be stored continuously in the first memory and may be arranged in the order of calculation in the multiple computing tasks. The model weights may include multiple weights, and the arrangement order of the multiple weights in the first memory is consistent with the calculation order of the multiple weights in the multiple computing tasks.

[0091] In addition, the model information may also include plaintext computational graph information, which may include multiple computational nodes and the computational order corresponding to the multiple computational nodes. The multiple weights of the model weight may include the weights used by the multiple computational nodes during the computation process. The multiple computational tasks may also be referred to as multiple computational tasks corresponding to the computational graph information. Therefore, the arrangement order of the model weights in the first memory may also be consistent with the computational order of the model weights in the computational graph information.

[0092] Optionally, the model information may also include signature information, which may be used to verify other information in the model information (e.g., computational graph information, model weights, and computational tasks) other than the signature information to ensure the integrity of the model information. For example, the signature information may be obtained by calculating other information in the model information other than the signature information using a preset algorithm.

[0093] In a possible embodiment, the encrypted model weight and signature information in the model information may be generated in a development environment. For example, Figure 6 As shown, when model information is developed in a development environment, the model weights in plain text in the model information can be encrypted to obtain model information containing the model weights in ciphertext form; then the encrypted model information is signed to obtain model information containing the signature information and the model weights in ciphertext form.

[0094] It is understandable that the above-mentioned encryption of the model weight in the model information in the development environment and signing of the encrypted model information are only exemplary. In actual applications, the model weight may not be encrypted and signed in the development environment, but encrypted and signed in the subsequent configuration process. The above example does not limit the embodiments of the present application.

[0095] Furthermore, before acquiring the model information, the general processing device may also load the model information into the first memory. In a possible embodiment, the general processing device may include an external memory, and the model information may be configured (or stored) in the external memory after development, so that the general processor of the general processing device may load the model information from the external memory into the first memory.

[0096] Optionally, if the model weights are not encrypted and the encrypted signature is not performed in the development environment, the model weights in the model information may be encrypted and the encrypted model information may be signed during the process of configuring the model information in the external memory.

[0097] S302: The general processing device sends the model information to the data acceleration processing device.

[0098] In a possible embodiment, the general processing device can transmit the model information to the data acceleration processing device via a bus. Exemplarily, the bus can be a PCIe bus. When the CPU of the general processing device obtains the model information, the CPU of the general processing device can send the model information to the data acceleration processing device via the PCIe bus, or directly store the model information in the second memory of the data acceleration processing device via the bus, specifically, it can be stored in the non-protected storage area of ​​the second memory. The general processing device can access the non-protected storage area. For example, the CPU of the general processing device can copy the model information from the general processing device to the non-protected storage area in the second memory of the data acceleration processing device.

[0099] S303: The second memory of the data acceleration processing device receives the model information and stores the model information in a non-protected storage area of ​​the second memory.

[0100] The second memory may include an unprotected storage area accessible to the general processing device, which may also be referred to as an unprotected storage area operable by the general processing device, that is, the general processing device may read and write in the unprotected storage area. In a possible embodiment, when the second memory of the data acceleration processing device receives the model information through the bus, the model information may be stored in the unprotected storage area; or, the acceleration processor of the data acceleration processing device may copy the model information from the general processing device through direct memory access (DMA) and store it in the unprotected storage area.

[0101] In addition, the model information stored in the unprotected storage area may include model weights in ciphertext. Further, the model information stored in the unprotected storage area may also include at least one of a plurality of computing tasks and graph computing information, and the plurality of computing tasks and the graph computing information may be in plaintext.

[0102] Optionally, the model information may also include signature information. When the second memory of the data acceleration processing device stores the model information in the non-protected storage area, the data acceleration processing device may also use the signature verification public key to verify the signature of the model information. In a possible embodiment, the data acceleration processing device also includes a persistent memory, in which the signature verification public key is stored. The acceleration processor of the data acceleration processing device verifies the signature information of the model information, which may specifically include: the acceleration processor obtains the signature verification public key from the persistent memory, and determines that the signature information of the model information is successfully verified based on the signature verification public key.

[0103] Optionally, the signature verification public key may be encrypted and stored in the persistent memory, so that the general processing device may first decrypt the signature verification public key after obtaining the signature verification public key. The key used to decrypt the signature verification public key may be a derived key of the hardware unique key HUK. The HUK may be stored in the one-time programmable memory.

[0104] In one possible example, the general processing device may decrypt the encrypted signature verification public key according to the derived key of the HUK to obtain the decrypted signature verification public key. In another possible example, when the signature verification public key is configured in the persistent memory of the data acceleration processing device, the signature verification public key may be encrypted using the derived key of the HUK, and then the encrypted signature verification public key may be stored in the persistent memory.

[0105] S304: The acceleration processor of the data acceleration processing device decrypts the model weight in the ciphertext form, and stores the decrypted model weight in the plaintext form to the protected storage area of ​​the second memory. The general processing device cannot access the protected storage area.

[0106] The second memory may also include a protected storage area that is inaccessible to the general purpose processing device, which may also be referred to as a protected storage area that is inoperable to the general purpose processing device, that is, the general purpose processing device cannot read or write in the protected storage area.

[0107] In addition, the model weights in the model information are in ciphertext form, and the acceleration processor of the data acceleration processing device can decrypt the ciphertext model weights stored in the non-protected storage area to obtain the model weights in plaintext. In order to protect the security of the model weights in plaintext, the acceleration processor of the data acceleration processing device can also store the model weights in plaintext to a protected storage area that is inaccessible to the general processing device.

[0108] Optionally, the specific process of the acceleration processor of the data acceleration processing device decrypting the model weight in ciphertext form can be performed by a corresponding accelerator or operator. For example, the data acceleration processing device may include a decryption operator AICPU, which can be used to decrypt the model weight in ciphertext form. In addition, a TEE environment is also provided in the data acceleration processing device, and the process of the data acceleration processing device decrypting the model weight in ciphertext form can be performed in the TEE environment.

[0109] Furthermore, in a possible embodiment, the data acceleration processing device also includes a persistent memory, for example, the persistent memory may include a flash memory, and a decryption key is stored in the persistent memory. Accordingly, the acceleration processor of the data acceleration processing device decrypts the model weight in ciphertext form, which may specifically include: the acceleration processor obtains the decryption key from the persistent memory, and decrypts the model weight in ciphertext form according to the decryption key to obtain the model weight in plaintext form. The decryption key may be configured in the persistent memory, for example, in the process of configuring the model information in the external memory of the general processing device, the decryption key may be configured in the persistent memory of the data acceleration processing device.

[0110] Optionally, the decryption key may be encrypted and stored in the persistent memory, so that the acceleration processor of the data acceleration processing device may first decrypt the decryption key after obtaining the decryption key. The key used to decrypt the decryption key may be a derived key of the HUK. The data acceleration processing device further includes a one-time programmable memory, and the HUK may be stored in the one-time programmable memory.

[0111] In one possible example, the acceleration processor of the data acceleration processing device may decrypt the encrypted decryption key according to the derived key of the HUK to obtain the decrypted decryption key. In another possible example, when the decryption key is configured in the persistent memory of the data acceleration processing device, the decryption key may be encrypted using the derived key of the HUK, and then the encrypted decryption key may be stored in the persistent memory.

[0112] After the acceleration processor of the data acceleration processing device stores the decrypted plaintext model weights in the protected storage area of ​​the second memory, Figure 7 As shown, the method may further include S305.

[0113] S305: The acceleration processor of the data acceleration processing device executes multiple computing tasks corresponding to the model information according to the model weight in plain text.

[0114] When the acceleration processor of the data acceleration processing device decrypts and obtains the model weight in plain text, the acceleration processor can execute the multiple computing tasks according to the model weight. Optionally, after completing the multiple computing tasks, the data acceleration processing device can also return the computing results corresponding to the multiple computing tasks to the general processing device.

[0115] In an embodiment of the present application, the general processing device can transmit model information to the data acceleration processing device through a bus, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the calculation task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related art, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0116] Further, such as Figure 7 As shown, before S305, the method further includes: S306 to S309.

[0117] S306: The acceleration processor of the data acceleration processing device sends the base address of the model weight in plain text form in the protected storage area to the general processing device.

[0118] Among them, the model weight in plain text form can be stored continuously in the protected storage area, and can be arranged according to the calculation order in the multiple computing tasks. For example, the model weight may include multiple weights, and the arrangement order of the multiple weights in the protected storage area is consistent with the calculation order of the multiple weights in the multiple computing tasks. The base address can also be called the first address, specifically refers to the logical address of the first weight among the multiple weights included in the model weight. The logical address can refer to the address used by a processor (such as an accelerated processor) to find the data when accessing the data.

[0119] For example, the graph computing information in the model information includes multiple computing nodes and the computing order corresponding to the multiple computing nodes, and the model weight includes multiple weights and the multiple weights are the weights used by the multiple computing nodes in the computing process. Then, the arrangement order of the model weights in plain text in the protected storage area can be as follows: Figure 8 shown.

[0120] In a possible embodiment, after storing the model weights in plain text to the protected storage area, the acceleration processor of the data acceleration processing device can determine the base address of the model weights in plain text in the protected storage area according to the position of the first weight in the model weights in the protected storage area, and send the base address to the general processing device through the bus.

[0121] S307: The general processing device receives the base address of the model weight in plain text form in the protected storage area sent by the data acceleration processing device, and determines the logical address of the model weight according to the base address and the address offset value of the model weight.

[0122] Among them, the model weights in the encrypted form in the general processing device are also stored continuously and arranged according to the calculation order in the multiple computing tasks. For example, the model weight may include multiple weights, and the arrangement order of the multiple weights in the general processing device is consistent with the calculation order of the multiple weights in the computing task. The address offset value of the model weight may include the address offset value of any one of the multiple weights. Taking the i-th weight among the multiple weights as an example, the address offset value of the i-th weight may refer to the offset value between the i-th logical address of the i-th weight and the base address corresponding to the first weight, and i is an integer in the value range.

[0123] For example, the graph computing information in the model information includes a plurality of computing nodes and the computing order corresponding to the plurality of computing nodes, and the model weight includes a plurality of weights and the plurality of weights are the weights used by the plurality of computing nodes in the computing process. Then, the arrangement order of the ciphertext model weights in the general processing device can be as follows: Fig. 9 shown.

[0124] When the model weights in ciphertext form are continuously stored in the first memory of the general processing device, the general processing device can determine the address offset value of any one of the model weights according to the arrangement order of the model weights in ciphertext form when they are stored. The address offset value of a weight can be the offset value between the logical address of the weight and the base address corresponding to the first weight in the arrangement order of the model weights.

[0125] In a possible embodiment, when the general processing device receives the base address of the model weight in plain text in the protected storage area, the general processing device can determine the logical address of the weight in the protected storage area based on the base address and the address offset value of any one of the model weights.

[0126] S308: The general processing device sends the logical address of the model weight to the data acceleration processing device.

[0127] When the general processing device determines the logical address of each weight in the model weight, the general processing device can send the logical address of each weight in the model weight to the data acceleration processing device through the bus. In a possible embodiment, the general processing device can send the logical address of each weight in the model weight to be used when executing each computing task to the data acceleration processing device in sequence according to the order in which the data acceleration processing device executes the above-mentioned multiple computing tasks.

[0128] S309: The acceleration processor of the data acceleration processing device receives the logical address of the model weight, and obtains the model weight in plain text from the protected storage area according to the logical address.

[0129] When the acceleration processor of the data acceleration processing device receives the logical address of each weight in the model weight, the acceleration processor of the data acceleration processing device can obtain the corresponding model weight in plain text from the protected storage area according to the logical address when executing the multiple computing tasks, and execute the multiple computing tasks according to the model weight in plain text.

[0130] In another possible embodiment, the acceleration processor of the data acceleration processing device may also determine the logical address of the model weight in the following manner: the general processing device sends the address offset value of the model weight to the data acceleration processing device through the bus; the acceleration processor of the data acceleration processing device receives the address offset value of the model weight through the bus, and determines the logical address of the model weight according to the base address of the model weight in plain text in the protected storage area and the address offset value.

[0131] Furthermore, the non-protected storage area and the protected storage area may be pre-configured by those skilled in the art. Optionally, the protected storage area may be a statically divided storage area with continuous physical addresses, and the capacity of the protected storage area is configurable.

[0132] Exemplarily, a non-protected storage area accessible to the general processing device is pre-configured in the second memory, in which case the storage areas other than the non-protected storage area in the second memory may be the protected memory; or, a protected storage area inaccessible to the general processing device is pre-configured in the second memory, in which case the storage areas other than the protected storage area in the second memory may be the non-protected memory; or, both the non-protected storage area and the protected storage area are configured in the second memory at the same time.

[0133] In a possible embodiment, the bus includes a management channel. Before the general processing device obtains the model information, the general processing device can also send memory configuration information to the data acceleration processing device through the management channel. The memory configuration information is used to configure the non-protected storage area and / or the protected storage area. In this way, the acceleration processor of the data acceleration processing device can receive the memory configuration information through the management channel, and configure the non-protected storage area and / or the protected storage area in the second memory according to the memory configuration information. The following is an explanation of the relevant content of configuring the protected storage area.

[0134] When configuring the unprotected storage area and the protected storage area, the second memory of the data acceleration processing device may be cleaned first, that is, the data in the second memory may be cleaned to prevent the residual data in the second memory from being leaked.

[0135] In addition, the size of the protected storage area is configured through the management channel, and a storage area with continuous physical addresses is statically divided as the protected storage area.

[0136] Secondly, after configuring the protected storage area, address mapping is performed, and the address of the protected storage area does not participate in the address mapping corresponding to the bus. For example, if the bus is a PCIe bus, the page table corresponding to the PCIe bus (also referred to as a PCIe page table) does not include the address mapping of the protected storage area, so that the general processing device cannot access the protected storage area through the bus. When mapping the page table of the acceleration processor in the data acceleration processing device, the address of the protected storage area and the address of the non-protected storage area can both participate in the mapping, that is, the page table corresponding to the acceleration processor in the data acceleration processing device includes both the address mapping of the protected storage area and the address mapping of the non-protected storage area, thereby ensuring that the acceleration processor of the data acceleration processing device can access the non-protected storage area and the protected storage area.

[0137] In addition, when configuring the non-protected storage area and the protected storage area, if there are other running services in the electronic device, the device can be reset to forcibly interrupt the execution of the service.

[0138] In the above embodiment, the technical solution of the embodiment of the present application is described from the perspective of the interaction between the general processing device and the data acceleration processing device. In practical applications, the steps corresponding to the general processing device can be implemented by the processor in the general processing device and the inference framework running on the processor, and the steps corresponding to the data acceleration processing device can be implemented by the acceleration processor in the data acceleration processing device and the firmware running on the acceleration processor. For ease of understanding, the following will be described respectively. Figures 10 to 12Taking the structure of the electronic device shown as an example, the technical solution of the embodiment of the present application is illustrated. Figures 10 to 12 The processor in the general processing device includes a CPU as an example for explanation.

[0139] In one example, Fig.10 As shown, the general processing device includes a CPU and a first memory, the data acceleration processing device includes an acceleration processor and a second memory, the second memory includes a protected storage area and an unprotected storage area, and the acceleration processor includes multiple accelerators and a decryption operator AICPU. Accordingly, the method may include: S11. The CPU of the general processing device copies the model information to the non-protected storage area of ​​the data acceleration processing device through the bus, and the model information includes calculation graph information (plain text), model weight (ciphertext) and calculation task (plain text); S12. The acceleration processor of the data acceleration processing device schedules the decryption operator AICPU to decrypt the model weight, and stores the decrypted model weight (plain text) to the protected storage area in the memory of the data acceleration processing device; S13. The acceleration processor of the data acceleration processing device sends the base address of the model weight in the protected storage area to the general processing device through the bus; S14. When receiving the base address, the CPU of the general processing device determines the address offset value of the model weight; S15. The CPU of the general processing device determines the logical address of the model weight according to the base address and the address offset value, and sends the logical address to the data acceleration processing device through the bus; S16. The acceleration processor of the data acceleration processing device obtains the model weight from the protected memory according to the logical address, and executes the calculation task according to the model weight.

[0140] In another example, combining Fig.10 ,like Fig.11 As shown, the general processing device is coupled to the data acceleration processing device through a PCIe bus, the general processing device also includes an inference architecture and a device management driver, and the data acceleration processing device also includes a flash memory, a one-time programmable memory, and firmware. Accordingly, the method may also include: S21. Configuring the data acceleration processing device through the device management driver, the configuration may include configuring a decryption key and a signature verification public key in the flash memory, and encrypting and protecting the decryption key and the signature verification public key using a derived key of the HUK in the one-time programmable memory; S22. Configuring a protected storage area in the memory of the data acceleration processing device through the device management driver, wherein the PCIe page table only maps the address of the non-protected storage area, and the page table of the acceleration processor in the data acceleration processing device simultaneously maps the address of the non-protected storage area and the address of the protected storage area.

[0141] In yet another example, combining Fig.11 ,like Fig.12As shown, the method may also include: S31. The inference architecture in the general processing device loads the model information, and copies the model information to the non-protected storage area of ​​the data acceleration processing device through the bus; S32. The firmware in the data acceleration processing device obtains the signature verification public key from the flash memory, and verifies the signature information of the model information; S33. After the signature verification is successful, the firmware in the data acceleration processing device obtains the decryption key from the flash memory to decrypt the model weights in ciphertext form stored in the non-protected storage area, and stores the decrypted model weights in plaintext form to the protected storage area; S34. The firmware in the data acceleration processing device loads the model weights in plaintext and executes the calculation task.

[0142] In the above example, the firmware in the data acceleration processing device provides the following functions in order to configure the inference framework to protect the model information during runtime: a. Model information signature verification: protects the integrity of the model information and prevents the model information from being tampered with to implant malicious operators, etc.; b. Model weight decryption: protects the confidentiality of the model weight; c. Protected storage area: isolates a storage area in the second memory of the data acceleration processing device, which can only be accessed by the acceleration processor of the data acceleration processing device and cannot be accessed by the general processing device through the bus; in addition, storing the decrypted plaintext model weights in the protected storage area in the memory can improve the reading efficiency of the acceleration processor; d. Key management: encrypts and manages the signature verification public key and the decryption key to ensure the confidentiality of the key.

[0143] In an embodiment of the present application, the general processing device can transmit model information to the data acceleration processing device through a bus, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the calculation task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related art, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0144] Based on this, the embodiment of the present application also provides another data processing method, which can be applied to the data acceleration processing device described above, such as Fig.13 As shown, the data processing method includes the following steps.

[0145] S320: The acceleration processor of the data acceleration processing device obtains the target task sent by the general processing device based on the model information.

[0146] The target task may be any one of the multiple computing tasks corresponding to the model information. Optionally, the target task may include a computing type and operand information. For example, the computing type may be one of an addition operation, a subtraction operation, an XOR operation, an OR operation, and an AND operation. The operand information may be used to indicate storage information of input data corresponding to the target task, and the storage information may be used to obtain the input data.

[0147] S321: The data acceleration processing device obtains a first logical address of the target weight, and reads the target weight from the protected storage area according to the first logical address. The general processing device cannot access the protected storage area.

[0148] In a possible embodiment, the acceleration processor obtains the first logical address of the target weight, which may include: the acceleration processor determines the base address of the model weight in plain text in the protected storage area; the acceleration processor sends the base address to the general processing device, and the base address is used to determine the first logical address of the target weight; the acceleration processor receives the first logical address of the target weight sent by the general processing device.

[0149] The target weight is the weight in the model weight that needs to be used when the target task is executed, that is, the model weight includes the target weight. The first logical address of the target weight specifically refers to the logical address of the target weight in the protected storage area.

[0150] S322: The acceleration processor of the data acceleration processing device executes the target task according to the target weight.

[0151] In a possible embodiment, the acceleration processor of the data acceleration processing device can also obtain the corresponding operand according to the operand information included in the target task; the acceleration processor can execute the target task according to the obtained target weight, operand and the calculation type included in the target task. Afterwards, the acceleration processor can also send the execution result corresponding to the target task to the general processing device.

[0152] It is understandable that the above Figures 5 to 12 All relevant contents of each step in the corresponding method embodiment can be cited to Fig.13 In the description of the corresponding method embodiment, the embodiments of the present application will not be repeated here.

[0153] In an embodiment of the present application, the data acceleration processing device can store the model weight in plain text obtained by decrypting the model weight in ciphertext form in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the model weight in plain text. In this way, when the data acceleration processing device executes the corresponding target task according to the model weight in plain text, the security of the model weight during execution can be ensured.

[0154] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between the general processing device and the data acceleration processing device. It is understandable that, in order to realize the above functions, the general processing device and the data acceleration processing device include hardware structures and / or software modules corresponding to the execution of each function. It should be easily appreciated by those skilled in the art that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present application.

[0155] The embodiment of the present application can divide the functional modules of the general processing device and the data acceleration processing device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above functional modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. The following is an example of dividing each functional module according to the corresponding function:

[0156] In the case of an integrated unit, Fig.14 A possible structural schematic diagram of the data acceleration processing device involved in the above embodiment is shown. The data acceleration processing device may include: a storage unit 401 and a processing unit 402. In a possible embodiment, the storage unit 401 can be used to support the data acceleration processing device to execute S303 in the above method embodiment; the processing unit 402 can be used to support the device to execute S304, S305, and S309 in the above method embodiment. Further, the data acceleration processing device also includes a sending unit 403; wherein the sending unit 403 can be used to support the data acceleration processing device to execute S306 in the above method embodiment. In a possible embodiment, the processing unit 402 can be used to support the data acceleration processing device to execute S320 to S322 in the above method embodiment.

[0157] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0158] On the basis of hardware implementation, the processing unit 402 may be an accelerated processor, the sending unit 403 may be a transmitter, the transmitter and the receiver may be integrated into a transceiver, and the transceiver may also be called a communication interface.

[0159] Fig.15 This is a possible structural diagram of a data acceleration processing device involved in an embodiment of the present application, and the data acceleration processing device may include: a memory 411 and an acceleration processor 412. Among them, the memory 411 is used to store the program code and data of the device, and the acceleration processor 412 is used to control the actions of the data acceleration processing device in the above method embodiment. For example, the acceleration processor 412 is used to support the device to execute S304, S305 and / or S309 in the above method embodiment, or execute S320 to S322 in the above method embodiment, and / or other processes for the technology described herein. Optionally, the data acceleration processing device may also include a communication interface 413, and the communication interface 413 is used to support the device to execute the steps of communicating with the general processing device in the above method embodiment.

[0160] Among them, the acceleration processor 412 can be a GPU, NPU, DSP, CPU, a dedicated integrated circuit, a processing chip, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various logic blocks, modules and circuits described in conjunction with the disclosure of the embodiments of the present application. The acceleration processor 412 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0161] The memory 411 may be a volatile memory or a non-volatile memory, etc. Optionally, the memory 411 may be integrated into the acceleration processor 412. In one example, the memory 411 may include a memory, a persistent memory, and a one-time programmable memory, etc. For example, the persistent memory may include a flash memory, which may be used to store a decryption key and / or a signature verification public key described below, etc., and the one-time programmable memory may be used to store a hardware unique key HUK described below.

[0162] In the case of an integrated unit, Fig.16A possible structural schematic diagram of the general processing device involved in the above embodiment is shown. The device may include: a processing unit 501 and a sending unit 502. In a possible embodiment, the processing unit 501 is used to support the device to perform S301 in the above method embodiment, and the step of determining the logical address in S307; the sending unit 502 can be used to support the device to perform S302 and S308 in the above method embodiment. Further, the general processing device also includes a receiving unit 503; wherein the receiving unit 503 is used to support the general processing device to perform the step of receiving the base address in S307 in the above method embodiment.

[0163] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module, and will not be repeated here.

[0164] Based on the hardware implementation, the processing unit 501 may be a processor, the sending unit 502 may be a transmitter, the receiving unit 503 may be a receiver, the receiver and the transmitter may be integrated into a transceiver, and the transceiver may also be called a communication interface.

[0165] Fig.17 This is a possible structural diagram of a general processing device involved in an embodiment of the present application, and the device includes: a memory 511 and a processor 512. Among them, the memory 511 is used to store the program code and data of the device, and the processor 512 is used to control the actions of the general processing device in the above method embodiment. For example, the processor 512 is used to execute the steps of determining the logical address in S301 and S307 in the above method embodiment, and / or other processes for the technology described herein. Optionally, the device may also include a communication interface 513, and the communication interface 513 is used to support the device to execute the steps of communicating with the data acceleration processing device in the above method embodiment.

[0166] Among them, the processor 512 can be a central processing unit, a general processor, a digital signal processor, an application-specific integrated circuit, a processing chip, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof. It can implement or execute various logic boxes, modules and circuits described in conjunction with the disclosure of the embodiments of the present application. The processor 512 can also be a combination that implements a computing function, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The communication interface 513 can be a transceiver, a transceiver circuit or a transceiver interface, and the like. The memory 511 can be a volatile memory or a non-volatile memory, and the like.

[0167] For example, the communication interface 513, the processor 512, and the memory 511 are connected to each other via a bus 514; the bus 514 may be a PCI bus or an EISA bus, etc. The bus 514 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0168] Optionally, the memory 511 may be included in the processor 512 .

[0169] In another aspect of the present application, an electronic device is provided, which includes any one of the data acceleration processing devices provided above and any one of the general processing devices provided above. The data acceleration processing device is used to execute the steps of the data acceleration processing device in the above method embodiment; the general processing device is used to execute the steps of the general processing device in the above method embodiment.

[0170] In another aspect of the present application, a terminal device is provided, which includes any one of the data acceleration processing devices provided above and any one of the general processing devices provided above. The data acceleration processing device is used to execute the steps of the data acceleration processing device in the above method embodiment; the general processing device is used to execute the steps of the general processing device in the above method embodiment.

[0171] Exemplarily, the terminal device may be a mobile phone, a tablet, a computer, a camera, a wearable device, or a vehicle-mounted device.

[0172] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware or a combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, a network device or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any medium that a computer can access or a data storage device such as a server or a data center that includes one or more media integrations. The medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., an optical disk), or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0173] In another aspect of the present application, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes one or more steps of the data acceleration processing device in the above method embodiment.

[0174] In another aspect of the present application, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes one or more steps of the general processing device in the data processing method provided by the above method embodiment.

[0175] In another aspect of the present application, a computer program product including instructions is provided, which, when executed on a computer, enables the computer to execute one or more steps of the data acceleration processing device in the above method embodiment.

[0176] In another aspect of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute one or more steps of the general processing device in the above method embodiment.

[0177] Finally, it should be noted that the above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A data processing method, characterized in that: Applied in a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area and a protected storage area, the method includes: The memory receives model information sent by the general processing device, and stores the model information in the unprotected storage area, the general processing device can access the unprotected storage area, and the model information includes model weights in ciphertext form; The acceleration processor decrypts the model weights in ciphertext form, and stores the decrypted model weights in plaintext form in the protected storage area, and the general processing device cannot access the protected storage area.

2. The method according to claim 1, characterized in that The data acceleration processing device also includes a persistent memory, wherein the persistent memory stores a decryption key; The acceleration processor decrypts the ciphertext model weights, including: The acceleration processor obtains the decryption key from the persistent memory, and decrypts the ciphertext model weights according to the decryption key.

3. The method according to claim 1 or 2, characterized in that: The data acceleration processing device further includes a persistent memory, wherein the persistent memory stores a signature verification public key, and the model information further includes signature information; the method further includes: The acceleration processor obtains the signature verification public key from the persistent memory, and determines, based on the signature verification public key, that the signature information of the model information has been successfully verified.

4. The method according to claim 2 or 3, characterized in that: The decryption key and / or the signature verification public key are stored in encrypted form, the data acceleration processing device further includes a one-time programmable memory, and the one-time programmable memory stores a hardware unique key HUK; the method further includes: The acceleration processor obtains the HUK from the one-time programmable memory, and encrypts and decrypts the decryption key and / or the signature verification public key according to a derived key of the HUK.

5. The method according to any one of claims 1 to 4, characterized in that: The model weights are stored continuously in the protected storage area, and the method further comprises: The acceleration processor sends the base address of the model weight in plain text in the protected storage area to the general processing device, where the base address is used to determine the logical address of the model weight; The acceleration processor receives the logical address of the model weight sent by the general processing device, and obtains the model weight in plain text from the protected storage area according to the logical address.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The acceleration processor receives memory configuration information, and configures the unprotected storage area and / or the protected storage area according to the memory configuration information.

7. A data processing method, characterized in that: Applied in a general processing device, the general processing device is used to communicate with a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device, the method includes: Obtaining model information, wherein the model information includes a model weight in ciphertext form; The model information is sent to the data acceleration processing device so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor stores the model weight in plain text after decrypting the ciphertext model weight in the protected storage area.

8. The method according to claim 7, characterized in that The model information also includes signature information, and the signature information is used to verify the model information.

9. The method according to claim 7 or 8, characterized in that: The model weights are stored continuously in the protected storage area, and the method further comprises: Receiving the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; Determining a logical address of the model weight according to the base address and the address offset value of the model weight; The logical address of the model weight is sent to the data acceleration processing device.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: Memory configuration information is sent to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

11. A data processing method, characterized in that: The method is applied to a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area and a protected storage area, the unprotected storage area includes model information, the model information includes a model weight in ciphertext form, and the protected storage area includes a model weight in plaintext form after the ciphertext model weight is decrypted. The method includes: The acceleration processor obtains a target task sent by the general processing device based on the model information; The acceleration processor obtains a first logical address of a target weight, and reads the target weight from the protected storage area according to the first logical address, wherein the target weight is a weight in the model weight required to be used when the target task is executed, and the general processing device cannot access the protected storage area; The acceleration processor executes the target task according to the target weight.

12. The method according to claim 11, characterized in that The acceleration processor obtains the logical address of the target weight, including: The acceleration processor determines a base address of the model weight in plain text in the protected storage area; The acceleration processor sends the base address to the general processing device, where the base address is used to determine the first logical address of the target weight; The acceleration processor receives the first logical address of the target weight sent by the general processing device.

13. A data acceleration processing device, characterized in that: The data acceleration processing device includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; The storage unit is used to receive the model information sent by the general processing device and store the model information in the unprotected storage area, the general processing device can access the unprotected storage area, and the model information includes the model weight in ciphertext form; The processing unit is used to decrypt the model weights in ciphertext form and store the decrypted model weights in plaintext form in the protected storage area. The general processing device cannot access the protected storage area.

14. The data acceleration processing device according to claim 13, characterized in that: The data acceleration processing device also includes a persistent memory, wherein the persistent memory stores a decryption key; The processing unit is further configured to obtain the decryption key from the persistent memory, and decrypt the ciphertext model weight according to the decryption key.

15. The data acceleration processing device according to claim 13 or 14, characterized in that: The data acceleration processing device also includes a persistent memory, the persistent memory stores a signature verification public key, and the model information also includes signature information; The processing unit is further configured to obtain the signature verification public key from the persistent storage, and determine, based on the signature verification public key, whether the signature information of the model information has been successfully verified.

16. The data acceleration processing device according to claim 14 or 15, characterized in that: The decryption key and / or the signature verification public key are stored in encrypted form, and the data acceleration processing device further includes a one-time programmable memory, which stores a hardware unique key HUK; The processing unit is further configured to obtain the HUK from the one-time programmable memory, and encrypt and decrypt the decryption key and / or the signature verification public key according to a derived key of the HUK.

17. The data acceleration processing device according to any one of claims 13 to 16, characterized in that: The model weights are stored continuously in the protected storage area; The processing unit is further used to send the base address of the model weight in plain text in the protected storage area to the general processing device, where the base address is used to determine the logical address of the model weight; The processing unit is further used to receive the logical address of the model weight sent by the general processing device, and obtain the model weight in plain text from the protected storage area according to the logical address.

18. The data acceleration processing device according to any one of claims 13 to 17, characterized in that: The processing unit is further used to receive memory configuration information and configure the unprotected storage area and / or the protected storage area according to the memory configuration information.

19. A general processing device, characterized in that: The general processing device is used to communicate with the data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes a non-protected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device, and the general processing device includes: A processing unit, configured to obtain model information, wherein the model information includes a model weight in a ciphertext form; A sending unit is used to send the model information to the data acceleration processing device, so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor stores the model weight in plain text after decrypting the ciphertext model weight in the protected storage area.

20. The universal processing device according to claim 19, characterized in that: The model information also includes signature information, and the signature information is used to verify the model information.

21. The universal processing device according to claim 19 or 20, characterized in that: The model weights are stored continuously in the protected storage area, and the general processing device further comprises a receiving unit; The receiving unit is used to receive the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; The processing unit is further used to determine the logical address of the model weight according to the base address and the address offset value of the model weight; The sending unit is also used to send the logical address of the model weight to the data acceleration processing device.

22. The universal processing device according to any one of claims 19 to 21, characterized in that: The sending unit is further used to send memory configuration information to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

23. A data acceleration processing device, characterized in that: The data acceleration processing device includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; The storage unit is used to store model information in the unprotected storage area, wherein the model information includes model weights in ciphertext form, and the protected storage area is not accessible to a general processing device; The storage unit is further used to store the model weight in plain text corresponding to the model weight in cipher text in the protected storage area, and the general processing device cannot access the protected storage area; The processing unit is used to obtain the target task sent by the general processing device based on the model information; The processing unit is further used to obtain a first logical address of a target weight, and read the target weight from the protected storage area according to the first logical address, wherein the target weight is a weight in the model weight required to be used when the target task is executed, and the general processing device cannot access the protected storage area; The processing unit is further configured to execute the target task according to the target weight.

24. The data acceleration processing device according to claim 23, characterized in that: The data acceleration processing device also includes a sending unit and a receiving unit; The processing unit is further used to determine a base address of the model weight in plain text in the protected storage area; The sending unit is further used to send the base address to the general processing device, where the base address is used to determine the first logical address of the target weight; The receiving unit is used to receive the first logical address of the target weight sent by the general processing device.

25. A data acceleration processing device, characterized in that: The data acceleration processing device includes an acceleration processor and a memory, wherein instructions are stored in the memory. When the acceleration processor runs the instructions, the device executes the data processing method as described in any one of claims 1 to 6, or executes the data processing method as described in any one of claims 11 to 12.

26. A general processing device, characterized in that: The general processing device includes a processor and a memory, wherein instructions are stored in the memory. When the processor executes the instructions, the device executes the data processing method according to any one of claims 7 to 10.

27. An electronic device, characterized in that: The electronic device comprises: a data acceleration processing device as described in any one of claims 13-18, any one of claims 23-24, or claim 25, and a general processing device as described in any one of claims 19-22 or claim 26.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed by a device, the device executes the data processing method according to any one of claims 1 to 12.

Images