Data processing method and apparatus, and device

Abstract

The present application relates to the technical field of communications, and provides a data processing method and apparatus, and a device, used for protecting the security of model information in an operating period without using hardware isolation technologies such as TEE. The method is applied to a data acceleration processing apparatus, the data acceleration processing apparatus comprises an acceleration processor and a memory, and the memory comprises a non-protected storage area and a protected storage area. The method comprises: a memory receives model information sent by a general-purpose processing apparatus, and stores the model information into a non-protected storage area, wherein the general-purpose processing apparatus can access the non-protected storage area, and the model information comprises a model weight in a ciphertext form; an acceleration processor decrypts the model weight in the ciphertext form, and stores the decrypted model weight in a plaintext form into a protected storage area, wherein the general-purpose processing apparatus cannot access the protected storage area.

Description

Data processing method, device and equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on July 27, 2023, with application number 202310935441.X and application name “A data processing method, device and equipment”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of communication technology, and in particular to a data processing method, apparatus, and device. Background Art

[0003] With the widespread application of artificial intelligence (AI) across various industries, the protection of AI models is becoming increasingly important. AI model protection primarily encompasses three stages: storage, transmission, and operation. Protecting AI models during the operation phase has always been a challenge for the industry.

[0004] Currently, electronic devices with confidential computing capabilities are often used to protect the security of AI models. For example, an electronic device includes a central processing unit (CPU) and a graphics processing unit (GPU), and the CPU and GPU are connected via a peripheral component interconnect express (PCIe) bus. The CPU is required to support a trusted execution environment (TEE), the GPU to support multi-instance isolation, and the PCIe bus to support encryption and decryption transmission capabilities, thereby achieving a hardware isolation environment across the CPU and GPU and ensuring the security of the AI ​​models running within it.

[0005] However, this solution has high requirements on the hardware of electronic devices, such as requiring the hardware of electronic devices to support isolation capabilities and encryption and decryption transmission capabilities. It is difficult to apply to related products already on the market, and thus its use is limited.

[0006] Summary of the Invention

[0007] The present application provides a data processing method, apparatus, and device for protecting the security of model information during runtime without using hardware isolation technologies such as TEE.

[0008] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0009] In a first aspect, a data processing method is provided, which is applied to a data acceleration processing device (i.e., a device with a data acceleration processing function, such as a GPU or an NPU, etc.), the data acceleration processing device including an acceleration processor and a memory, the memory including an unprotected storage area and a protected storage area, the data acceleration processing device can communicate with a general processing device, the method comprising: the memory receiving model information sent by the general processing device, and storing the model information in the unprotected storage area, the general processing device can access the unprotected storage area, the model information including model weights in ciphertext form; the acceleration processor decrypting the ciphertext model weights, and storing the decrypted plaintext model weights in the protected storage area, the general processing device cannot access the protected storage area.

[0010] In the above technical solution, the general processing device can send model information to the data acceleration processing device, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the corresponding computing task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related technology, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0011] In a possible implementation of the first aspect, the specific process of the acceleration processor decrypting the model weights in ciphertext form can be performed by a corresponding accelerator or operator in the acceleration processor. For example, the acceleration processor may include a decryption operator AICPU, which can be used to decrypt the model weights in ciphertext form. In addition, the data acceleration processing device is also provided with a TEE environment, and the process of the acceleration processor decrypting the model weights in ciphertext form can be performed in the TEE environment.

[0012] In a possible implementation of the first aspect, the data acceleration processing device further includes a persistent memory storing a decryption key; and the acceleration processor decrypting the encrypted model weights includes: the acceleration processor obtaining the decryption key from the persistent memory and decrypting the encrypted model weights based on the decryption key. In this possible implementation, the security of the model weights is ensured by storing the decryption key in the persistent memory and decrypting the encrypted model weights based on the decryption key.

[0013] In one possible implementation of the first aspect, the data acceleration processing device further includes a persistent memory storing a signature verification public key, and the model information further includes signature information. The method further includes: the acceleration processor obtaining the signature verification public key from the persistent memory, and determining, based on the signature verification public key, that verification of the signature information of the model information has been successful. In this possible implementation, the integrity of the model information can be ensured by signing and verifying the signature of the model information.

[0014] In one possible implementation of the first aspect, the decryption key and / or signature verification public key are stored encrypted, and the data acceleration processing device further includes a one-time programmable memory storing a hardware-unique key HUK. The method further includes: the acceleration processor obtaining the HUK from the one-time programmable memory and encrypting and decrypting the decryption key and / or signature verification public key based on a key derived from the HUK. In this possible implementation, by encrypting and storing the decryption key and / or signature verification public key, the confidentiality of the decryption key and / or signature verification public key is ensured, thereby ensuring the security of the model weights.

[0015] In a possible implementation manner of the first aspect, the model weight and signature information in the encrypted form in the model information may be generated in a development environment.

[0016] In a possible implementation of the first aspect, the model weight is stored continuously in the protected storage area; the method further includes: the acceleration processor sends the base address of the model weight in plain text in the protected storage area to the general processing device, and the base address is used to determine the logical address of the model weight; the acceleration processor receives the logical address of the model weight sent by the general processing device, and obtains the model weight in plain text from the protected storage area according to the logical address. In the above possible implementation, the acceleration processor can send the base address of the model weight in plain text in the protected storage area to the general processing device, so that the general processing device determines the logical address according to the base address and the address offset value of the model weight, and sends it to the acceleration processor, so that the general processing device does not need to read the model weight when scheduling the data acceleration processing device to perform a computing task, but only needs to send the logical address of the model weight, thereby ensuring the security of the model weight.

[0017] In a possible implementation of the first aspect, the method further includes: the acceleration processor receiving an address offset value of the model weight sent by the general processing device; and the acceleration processor determining a logical address of the model weight based on a base address of the plaintext model weight in the protected storage area and the address offset value. In this possible implementation, when scheduling the data acceleration processing device to perform a computing task, the general processing device does not need to read the model weight and only needs to send the address offset value of the model weight, thereby ensuring the security of the model weight.

[0018] In one possible implementation of the first aspect, the method further includes: the acceleration processor receiving memory configuration information, and configuring the unprotected storage area and / or the protected storage area in the memory according to the memory configuration information. In this possible implementation, by configuring a protected storage area in the memory of the data acceleration processing device that is inaccessible to the general processing device, the security of the model weights can be ensured.

[0019] According to a second aspect, a data processing method is provided, which is applied to a general processing device, wherein the general processing device communicates with a data acceleration processing device via a bus, and the memory of the data acceleration processing device includes an unprotected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device. The method includes: obtaining model information, wherein the model information includes model weights in ciphertext form; sending the model information to the acceleration processor so that the memory of the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor stores the model weights in plaintext obtained by decrypting the ciphertext model weights in the protected storage area.

[0020] In the above technical solution, the general processing device can transmit model information to the data acceleration processing device, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the calculation task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related technology, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0021] In a possible implementation of the second aspect, the model information further includes signature information. In the above possible implementation, the integrity of the model information can be ensured by signing and verifying the signature of the model information.

[0022] In a possible implementation of the second aspect, the method further includes: the general processing device receives the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; the general processing device determines the logical address of the model weight based on the base address and the address offset value of the model weight; the general processing device sends the logical address of the model weight to the data acceleration processing device. In the above possible implementation, the general processing device can receive the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device, determine the logical address based on the base address and the address offset value of the model weight, and send it to the data acceleration processing device, so that the general processing device does not need to read the model weight when scheduling the data acceleration processing device to perform a computing task, but only needs to read the logical address of the model weight, thereby ensuring the security of the model weight.

[0023] In a possible implementation of the second aspect, the method further includes: the general processing device sending the address offset value of the model weight to the data acceleration processing device, so that the data acceleration processing device determines the logical address of the model weight based on the base address of the plaintext model weight in the protected storage area and the address offset value. In this possible implementation, when scheduling the data acceleration processing device to perform a computing task, the general processing device does not need to read the model weight and only needs to send the address offset value of the model weight, thereby ensuring the security of the model weight.

[0024] In one possible implementation of the second aspect, the method further includes: sending memory configuration information to the data acceleration processing device via the management channel, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area. In this possible implementation, by configuring a protected storage area in the memory of the data acceleration processing device that is inaccessible to the general processing device, the security of the model weights can be ensured.

[0025] According to a third aspect, a data processing method is provided, which is applied to a data acceleration processing device, the data acceleration processing device including an acceleration processor and a memory, the memory including an unprotected storage area and a protected storage area, the unprotected storage area including model information, the model information including model weights in ciphertext form, the protected storage area including model weights in plaintext form after decryption of the ciphertext model weights, the method including: the acceleration processor obtaining a target task sent by a general processing device based on the model information; the acceleration processor obtaining a first logical address of the target weight, and reading the target weight from the protected storage area according to the first logical address, the target weight being the weight in the model weight required to be used when executing the target task, and the general processing device cannot access the protected storage area; the acceleration processor executes the target task according to the target weight.

[0026] In the above technical solution, the data acceleration processing device stores the model information including the model weights in ciphertext form in a non-protected storage area, and stores the corresponding decrypted model weights in plaintext form in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the model weights in plaintext form. In this way, when the data acceleration processing device executes the corresponding target task according to the model weights in plaintext form, it can ensure the security of the model weights during execution.

[0027] In a possible implementation of the third aspect, the acceleration processor obtains the logical address of the target weight, including: the acceleration processor determines the base address of the model weight in plain text in the protected storage area; the acceleration processor sends the base address to the general processing device, and the base address is used to determine the first logical address of the target weight; the acceleration processor receives the first logical address of the target weight sent by the general processing device. In the above possible implementation, when the data acceleration processing device executes the corresponding target task, it does not need to read the target weight, but only needs to send the first logical address of the target weight, thereby ensuring the security of the model weight.

[0028] In a fourth aspect, a data acceleration processing device is provided, which includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; the storage unit is used to receive model information sent by a general processing device and store the model information in the non-protected storage area, and the general processing device can access the non-protected storage area, and the model information includes model weights in ciphertext form; the processing unit is used to decrypt the model weights in ciphertext form and store the decrypted model weights in plaintext form in the protected storage area, and the general processing device cannot access the protected storage area.

[0029] In a possible implementation of the fourth aspect, the data acceleration processing device also includes a persistent memory, which stores a decryption key; the processing unit is also used to: obtain the decryption key from the persistent memory, and decrypt the model weight in ciphertext form according to the decryption key.

[0030] In a possible implementation of the fourth aspect, the data acceleration processing device also includes a persistent memory, which stores a signature verification public key, and the model information also includes signature information; the processing unit is also used to: obtain the signature verification public key from the persistent memory, and determine whether the signature information verification of the model information is successful based on the signature verification public key.

[0031] In a possible implementation of the fourth aspect, the decryption key and / or signature verification public key are stored in an encrypted manner, and the data acceleration processing device also includes a one-time programmable memory, which stores a hardware-unique key HUK; the processing unit is further used to: obtain the HUK from the one-time programmable memory, and encrypt and decrypt the decryption key and / or signature verification public key according to a derived key of the HUK.

[0032] In a possible implementation of the fourth aspect, the model weight is stored continuously in the protected storage area; the processing unit is also used to: send the base address of the model weight in plain text in the protected storage area to the general processing device, and the base address is used to determine the logical address of the model weight; receive the logical address of the model weight sent by the general processing device, and obtain the model weight in plain text from the protected storage area according to the logical address.

[0033] In a possible implementation manner of the fourth aspect, the processing unit is further used to: receive memory configuration information, and configure the unprotected storage area and / or the protected storage area according to the memory configuration information.

[0034] In a fifth aspect, a general processing device is provided, which is used to communicate with a data acceleration processing device. The data acceleration processing device includes an acceleration processor and a memory. The memory includes an unprotected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device. The general processing device includes: a processing unit for obtaining model information, wherein the model information includes model weights in ciphertext form; a sending unit for sending the model information to the data acceleration processing device, so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor decrypts the ciphertext model weights and stores them in the protected storage area in plaintext form.

[0035] In a possible implementation manner of the fifth aspect, the model information further includes signature information, and the signature information is used to verify the model information.

[0036] In a possible implementation of the fifth aspect, the model weight is stored continuously in the protected storage area, and the general processing device also includes a receiving unit; the receiving unit is used to receive the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; the processing unit is also used to determine the logical address of the model weight based on the base address and the address offset value of the model weight; the sending unit is also used to send the logical address of the model weight to the data acceleration processing device.

[0037] In a possible implementation manner of the fifth aspect, the sending unit is further used to: send memory configuration information to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

[0038] In a sixth aspect, a data acceleration processing device is provided, which includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; the storage unit is used to store model information in the non-protected storage area, and the model information includes model weights in ciphertext form, and the general processing device cannot access the protected storage area; the storage unit is also used to store model weights in plaintext corresponding to the model weights in ciphertext form in the protected storage area, and the general processing device cannot access the protected storage area; the processing unit is used to obtain a target task sent by the general processing device based on the model information; the processing unit is also used to obtain a first logical address of the target weight, and read the target weight from the protected storage area according to the first logical address, the target weight being the weight in the model weight that needs to be used when the target task is executed; the processing unit is also used to execute the target task according to the target weight.

[0039] In a possible implementation of the sixth aspect, the data acceleration processing device also includes a sending unit and a receiving unit; the processing unit is further used to determine the base address of the model weight in plain text in the protected storage area; the sending unit is further used to send the base address of the target weight to the general processing device, and the base address is used to determine the first logical address of the target weight; the receiving unit is used to receive the first logical address of the target weight sent by the general processing device.

[0040] In the seventh aspect, a data acceleration processing device is provided, which includes an acceleration processor and a memory, in which instructions are stored. When the acceleration processor runs the instructions, the device executes the data processing method provided by the first aspect or any possible implementation of the first aspect, or executes the data processing method provided by the third aspect or any possible implementation of the third aspect.

[0041] In an eighth aspect, a general processing device is provided, which includes a processor and a memory, wherein instructions are stored in the memory. When the processor runs the instructions, the device executes the data processing method provided in the second aspect or any possible implementation of the second aspect.

[0042] Another aspect of the present application provides an electronic device, which includes: a data acceleration processing device as provided in any of the above aspects, and a general processing device as provided in any of the above aspects.

[0043] Another aspect of the present application provides a terminal device, comprising: the data acceleration processing device provided in any of the above aspects, and the general processing device provided in any of the above aspects. For example, the terminal device can be a mobile phone, tablet, computer, camera, wearable device, or vehicle-mounted device.

[0044] In another aspect of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the data processing method provided by the first aspect or any possible implementation of the first aspect.

[0045] In another aspect of the present application, a computer-readable storage medium is provided, in which instructions are stored. When the instructions are executed by a device, the device executes the data processing method provided in the second aspect or any possible implementation of the second aspect.

[0046] Another aspect of the present application provides a computer-readable storage medium, which stores instructions. When the instructions are executed by a device, the device executes the data processing method provided by the third aspect or any possible implementation of the third aspect.

[0047] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method as provided in the first aspect or any possible implementation of the first aspect.

[0048] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method as provided in the second aspect or any possible implementation of the second aspect.

[0049] In another aspect of the present application, a computer program product is provided, which includes: a computer program (also referred to as code, or instructions), which, when executed, enables a computer to execute a data processing method as provided in the third aspect or any possible implementation of the third aspect.

[0050] It can be understood that the beneficial effects that can be achieved by any of the data acceleration processing devices, general processing devices, electronic devices, terminal devices, computer-readable storage media and computer program products provided above can correspond to the beneficial effects of the data processing methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] FIG1 is a schematic diagram of a method for implementing runtime protection of an AI model;

[0052] FIG2 is a schematic diagram of another method for implementing runtime protection of an AI model;

[0053] FIG3 is a schematic structural diagram of an electronic device provided in an embodiment of the present application;

[0054] FIG4 is a schematic structural diagram of another electronic device provided in an embodiment of the present application;

[0055] FIG5 is a flow chart of a data processing method provided in an embodiment of the present application;

[0056] FIG6 is a schematic diagram of model information provided in an embodiment of the present application;

[0057] FIG7 is a flow chart of another data processing method provided in an embodiment of the present application;

[0058] FIG8 is a schematic diagram of a model weight in plain text provided in an embodiment of the present application;

[0059] FIG9 is a schematic diagram of a model weight in ciphertext form provided in an embodiment of the present application;

[0060] FIG10 is a schematic diagram of data processing provided by an embodiment of the present application;

[0061] FIG11 is a schematic diagram of a configuration data acceleration processing device provided in an embodiment of the present application;

[0062] FIG12 is a schematic diagram of another data processing method provided by an embodiment of the present application;

[0063] FIG13 is a flow chart of another data processing method provided in an embodiment of the present application;

[0064] FIG14 is a schematic diagram of the structure of a data acceleration processing device provided in an embodiment of the present application;

[0065] FIG15 is a schematic diagram of the structure of another data acceleration processing device provided in an embodiment of the present application;

[0066] FIG16 is a schematic structural diagram of a general processing device provided in an embodiment of the present application;

[0067] FIG17 is a schematic structural diagram of another general processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. In the present application, "at least one" refers to one or more, and "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can represent: a, b, c, a and b, a and c, b and c or a, b and c, where a, b and c can be single or multiple.

[0069] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or effects. Those skilled in the art will understand that terms such as "first" and "second" do not limit the quantity or order of execution. The term "coupled" is used to indicate an electrical connection, including direct connection via wires or connectors or indirect connection via other devices. Therefore, "coupling" should be considered a broad term for electronic communication connection.

[0070] It should be noted that, in this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described in this application as "exemplary" or "for example" should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0071] Before introducing the embodiments of the present application, the relevant background technology involved in the present application is first introduced and explained.

[0072] With the widespread application of artificial intelligence (AI) in various industries, the protection of AI models is becoming increasingly important. The protection of AI models can mainly include protection in three stages: storage, transmission, and operation. Implementing protection during the operation stage has always been a difficult problem in the industry. In related technologies, the following two methods are commonly used to implement protection during the operation stage of AI models. This protection during the operation stage can also be referred to as runtime protection.

[0073] The first type is to use electronic devices with confidential computing capabilities. As shown in Figure 1, taking an electronic device including a central processing unit (CPU) and a graphics processing unit (GPU), and the CPU and GPU being connected via a peripheral component interconnect express (PCIe) bus as an example, the CPU is required to support trusted execution environment (TEE) capabilities, the GPU to support multi-instance isolation capabilities, and the PCIe bus to support encryption and decryption transmission capabilities. Specifically, each TEE of the CPU is configured with a confidential virtual machine (VM), and the confidential VM is provided with a GPU driver, which is encrypted and transmitted via the PCIe bus. The GPU includes a secure GPU instance isolated by a firewall, and each secure GPU instance is correspondingly configured with a PCIe virtual function (VF), which can be used to support the secure GPU instance for transmission via the PCIe bus. Therefore, in this electronic device, a confidential VM in the CPU, a secure GPU instance in the GPU, and the corresponding PCIe VF can form a hardware isolation environment (or called a secure environment) across the CPU and GPU, thereby ensuring the security of the AI ​​model running therein.

[0074] In the above solution, the CPU can be used to process the input of AI applications, load model files, etc.; the GPU can be used to perform corresponding AI computing tasks; and the PCIe bus can be used to transmit interactive data streams between the CPU and GPU. In other words, this solution covers three parts of data flow design. First, TEE isolation capabilities are used on the CPU, so that non-secure environments cannot access AI applications and AI models. By connecting to a remote attestation server or local attestation, it can be ensured that the operating status of the GPU environment connected to the TEE meets the requirements, such as the correct firmware version. Second, the operating status on the GPU can be measured and perceived by the CPU; multi-instance isolation provides hardware isolation capabilities to ensure that AI computing tasks in secure environments and AI computing tasks in non-secure environments are isolated and do not affect each other. Third, the PCIe bus uses encrypted transmission to ensure the confidentiality and integrity of the data flow between the CPU and GPU.

[0075] However, this solution has high hardware requirements for electronic devices, such as requiring the hardware of electronic devices to support isolation capabilities and encryption and decryption transmission capabilities. It is difficult to apply to related products already on the market (such as electronic devices or chips). In other words, it is impossible to achieve the runtime protection capability of the AI ​​model through software enhancement alone, and thus its use is limited.

[0076] The second method is to realize parameter protection of AI models during operation by abstracting the units of model-related operations in TEE and combining the existing computing and storage resources in TEE. Specifically, as shown in Figure 2, the electronic device includes a trusted execution environment TEE and a non-trusted execution environment non-TEE. The TEE includes a configuration manager, a model processing unit, a model management unit, an authentication manager, secure storage and a computation processing unit. The non-TEE includes AI services, applications and accelerators. Among them, the model management unit can be used to communicate with the configuration manager, the model processing unit, the authentication manager and the secure storage, and the model processing unit can be used to communicate with the AI ​​service, and schedule the computation processing unit and the accelerator to perform corresponding calculations. The AI ​​model in the above-mentioned electronic device can be configured after development and training.

[0077] The above solution makes significant changes to the software stack and does not protect the AI ​​model-related information sent between the model processing unit and the accelerator.

[0078] Based on this, an embodiment of the present application provides a data processing method, which can be used to protect the security of AI models and prevent them from being stolen during offline inference without using hardware isolation technologies such as TEE. The method can be applied to electronic devices, including but not limited to: mobile phones, tablet computers, laptop computers, PDAs, mobile internet devices (MIDs), wearable devices (such as smart watches and smart bracelets), vehicle-mounted devices (such as cars, bicycles, electric vehicles, airplanes, ships, trains, high-speed trains, etc.), virtual reality (VR) devices, augmented reality (AR) devices, wireless terminals in industrial control, smart home devices (such as refrigerators, televisions, air conditioners, electric meters, etc.), intelligent robots, workshop equipment, wireless terminals in self-driving, wireless terminals in remote medical surgery, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, and flying devices (such as intelligent robots, hot air balloons, drones, and airplanes). The above-mentioned electronic devices may also be referred to as terminal devices.

[0079] The specific structure of the electronic device is described below.

[0080] FIG3 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, wherein the electronic device includes: a general-purpose processing device 100, a data acceleration processing device 200, and a bus 300. The general-purpose processing device 100 and the data acceleration processing device 200 are coupled via the bus 300. The general-purpose processing device 100 may refer to a device having general data processing, instruction or command execution, and task delivery capabilities. The general-purpose processing device 100 may also be referred to as a host, a general-purpose server, or a master device. The general-purpose processing device 100. The data acceleration processing device 200 may refer to a device having data acceleration processing capabilities. The data acceleration processing device 200 may also be referred to as a device, a dedicated processing device, or a slave device.

[0081] The general-purpose processing device 100 may include one or more general-purpose processors, including but not limited to a central processing unit (CPU), a digital signal processor (DSP), a microcontroller, or a microprocessor. The general-purpose processor may be used to execute various functions of the electronic device and process data to monitor the electronic device as a whole, such as for processing the electronic device's operating system, user interface, and application programs. In some embodiments, the general-purpose processing device 100 may also include memory (e.g., internal memory and external memory), which may be used to store data, software programs, and modules related to the general-purpose processing device 100. For example, the software programs may include an inference framework. For example, the memory may include a program storage area and a data storage area. The program storage area may store software programs, including coded instructions, including but not limited to an operating system and application programs required for at least one function, such as a sound playback function or an image playback function. The data storage area may store data generated based on the use of the electronic device, such as audio data, image data, and text data. FIG3 illustrates the general-purpose processing device 100 as including a CPU and a first memory. The first memory may be DRAM, DDR, or other memory.

[0082] The data acceleration processing device 200 may include one or more acceleration processors, which may include but are not limited to: a graphics processing unit (GPU), a neural network processing unit (NPU), a DSP, a CPU, an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD), etc. The acceleration processor may include one or more accelerators (e.g., a vector calculation unit, a matrix calculation unit, and / or a scalar calculation unit), and one or more operators (e.g., a decryption operator AICPU, or other custom operators). The acceleration processor may be used to perform corresponding calculations (e.g., vector calculations, matrix calculations, and / or scalar calculations) under the call of the general processing device 100. In some embodiments, the data acceleration processing device 200 may also include a memory, which may be used to store data, software programs, and modules related to the data acceleration processing device 200. For example, the software program may include firmware. In one example, the memory in the data acceleration processing device 200 may include a second memory, which may include an unprotected storage area accessible to the general processing device 100 and a protected storage area inaccessible to the general processing device 100. Furthermore, the memory may also include a persistent memory and a one-time programmable memory (efuse), etc. For example, the persistent memory may include a flash memory, which may be used to store the decryption key and / or the signature verification public key mentioned below. The one-time programmable memory refers to a memory that is only allowed to be written once (or is called a memory that is only allowed to be programmed once). In an embodiment of the present application, the one-time programmable memory may be used to store the hardware unique key (HUK) mentioned below. Figure 3 takes the data acceleration processing device 200 including the NPU and the second memory as an example for explanation.

[0083] It is understandable that the memory (e.g., the first memory and the second memory) in the embodiments of the present application may refer to the internal memory of the corresponding device, which can be used to temporarily store data during the operation of the processor of the corresponding device (e.g., a general-purpose processor and an accelerated processor), has a high access efficiency and a small capacity, and the stored data will usually be lost after a power outage. The above-mentioned persistent memory and one-time programmable memory may refer to external memory, which can be used for long-term data storage, has a relatively low access efficiency and a large capacity, and the stored data will not be lost after a power outage. The persistent memory may be a memory that supports multiple writes, and the one-time programmable memory refers to a memory that allows only one write.

[0084] The bus 300 may be used to transmit information between the general processing device 100 and the data acceleration processing device 200, thereby enabling communication between the general processing device 100 and the data acceleration processing device 200. In some embodiments, the bus 300 may be a PCIe bus or an extended industry standard architecture (EISA) bus.

[0085] Although not shown, the general processing device 100 may also include other components such as sensor components (e.g., acceleration sensors, gyroscope sensors, pressure sensors and / or temperature sensors, etc.), multimedia components (e.g., display panels and cameras, etc.), input and output devices (e.g., mice and keyboards, etc.), communication modules (e.g., Bluetooth modules and WiFi modules) and power supply components, which are not described in detail in the embodiments of the present application.

[0086] Furthermore, the electronic device processing includes the above-mentioned hardware resources, and may also include a software architecture running on the above-mentioned hardware resources (such as processors and memories, etc.). In some embodiments of the present application, the software architecture of the general processing device 100 in the electronic device may include an application layer (such as AI applications), an application framework layer, a function library layer, and a kernel layer, etc. In one example, as shown in Figure 4, the software architecture may include an inference framework and a device management driver; the inference framework may be located in the above-mentioned application framework layer, and the inference framework may be used to provide various APIs used when the application accesses; the device management driver may be located in the above-mentioned kernel layer, and the device management driver may be used to configure the second memory in the data acceleration processing device 200, such as for configuring the above-mentioned protected storage area and unprotected storage area. In other embodiments of the present application, as shown in Figure 4, the data acceleration processing device 200 may also include a corresponding software architecture, such as the software architecture may include firmware running on an acceleration processor.

[0087] It will be understood that the hardware resources and software architecture of the electronic device shown in FIG3 and FIG4 are merely exemplary. In actual applications, the electronic device may also include more or different components or software than those shown in the figure. The above examples do not constitute a limitation to the embodiments of the present application.

[0088] Figure 5 is a flow chart of a data processing method provided in an embodiment of the present application, which includes the following steps. The data processing can be applied to the electronic device provided above, which includes a general processing device and a data acceleration processing device. The following is an example in which the general processing device includes a general processor (e.g., CPU) and a first memory, and the data acceleration processing device includes an acceleration processor (e.g., NPU) and a second memory.

[0089] S301: The general processing device obtains model information, where the model information includes model weights in ciphertext form.

[0090] Among them, the model information may be the model information of the AI ​​model. The model information may be stored in the first memory of the general processing device. The model weight (weight) in the ciphertext form in the model weight may refer to the encrypted model weight. Optionally, the model information may also include multiple computing tasks. The model weight includes the weights that need to be used by the multiple computing tasks during the calculation process. The multiple computing tasks may be in plain text in the model information, and the multiple computing tasks may specifically be computing tasks that need to be scheduled for execution by a data acceleration processing device, for example, the multiple computing tasks may be AI computing tasks. Optionally, the model information may also include other parameters in addition to the model weight, such as the input parameters and output parameters required by the model, etc. The other parameters may also be in ciphertext form. The embodiment of the present application is described using the model weight in ciphertext form as an example.

[0091] In one possible embodiment, the model information is stored in a first memory of the general-purpose processing device, and the general-purpose processor of the general-purpose processing device can obtain the model information from the first memory. Furthermore, the general-purpose processor of the general-purpose processing device can also parse and convert the model information so that the converted model information can be recognized and processed by the data acceleration processing device.

[0092] Optionally, the model weights may be stored continuously in the first memory and may be arranged according to the order of calculation in the multiple computing tasks. The model weights may include multiple weights, and the order of arrangement of the multiple weights in the first memory is consistent with the order of calculation of the multiple weights in the multiple computing tasks.

[0093] In addition, the model information may also include plaintext computational graph information, which may include multiple computational nodes and the computation order corresponding to the multiple computational nodes. The multiple weights of the model weight may include the weights used by the multiple computational nodes during the computation process. The multiple computational tasks may also be referred to as multiple computational tasks corresponding to the computational graph information. Therefore, the arrangement order of the model weights in the first memory may also be consistent with the computation order of the model weights in the computational graph information.

[0094] Optionally, the model information may also include signature information, which can be used to verify other information in the model information (e.g., computational graph information, model weights, and computational tasks) other than the signature information to ensure the integrity of the model information. For example, the signature information may be calculated using a preset algorithm using other information in the model information other than the signature information.

[0095] In one possible embodiment, the ciphertext model weights and signature information in the model information can be generated in a development environment. For example, as shown in FIG6 , when the model information is developed and obtained in the development environment, the plaintext model weights in the model information can be encrypted to obtain model information containing the ciphertext model weights. The encrypted model information can then be signed to obtain model information containing the signature information and the ciphertext model weights.

[0096] It will be appreciated that the encryption of the model weights in the model information in the development environment and the signing of the encrypted model information are merely exemplary. In actual applications, the model weights may not be encrypted or signed in the development environment, but may be encrypted and signed during a subsequent configuration process. The above example does not limit the embodiments of this application.

[0097] Furthermore, before acquiring the model information, the general processing device may also load the model information into the first memory. In one possible embodiment, the general processing device may include an external memory, and the model information may be configured (or stored) in the external memory after development, so that the general processor of the general processing device can load the model information from the external memory into the first memory.

[0098] Optionally, if the model weights are not encrypted and the encrypted signature is not performed in the development environment, the model weights in the model information can be encrypted and the encrypted model information can be signed during the process of configuring the model information in the external memory.

[0099] S302: The general processing device sends the model information to the data acceleration processing device.

[0100] In a possible embodiment, the general processing device can transmit the model information to the data acceleration processing device via a bus. Exemplarily, the bus can be a PCIe bus. When the CPU of the general processing device obtains the model information, the CPU of the general processing device can send the model information to the data acceleration processing device via the PCIe bus, or directly store the model information in the second memory of the data acceleration processing device via the bus, specifically in the non-protected storage area of ​​the second memory. The general processing device can access the non-protected storage area. For example, the CPU of the general processing device can copy the model information from the general processing device to the non-protected storage area in the second memory of the data acceleration processing device.

[0101] S303: The second memory of the data acceleration processing device receives the model information and stores the model information in a non-protected storage area of ​​the second memory.

[0102] The second memory may include an unprotected storage area accessible to the general-purpose processing device, which may also be referred to as an unprotected storage area operable by the general-purpose processing device, i.e., the general-purpose processing device may read and write in the unprotected storage area. In one possible embodiment, when the second memory of the data acceleration processing device receives the model information via the bus, the model information may be stored in the unprotected storage area; alternatively, the acceleration processor of the data acceleration processing device may copy the model information from the general-purpose processing device via direct memory access (DMA) and store it in the unprotected storage area.

[0103] In addition, the model information stored in the unprotected storage area may include model weights in ciphertext. Further, the model information stored in the unprotected storage area may also include at least one of a plurality of computing tasks and graph computing information, and the plurality of computing tasks and the graph computing information may be in plaintext.

[0104] Optionally, the model information may also include signature information. When the second memory of the data acceleration processing device stores the model information in the unprotected storage area, the data acceleration processing device may also verify the signature of the model information using the signature verification public key. In one possible embodiment, the data acceleration processing device also includes a persistent memory that stores the signature verification public key. The acceleration processor of the data acceleration processing device verifies the signature information of the model information. Specifically, the acceleration processor may obtain the signature verification public key from the persistent memory and determine, based on the signature verification public key, that the signature information of the model information has been successfully verified.

[0105] Optionally, the signature verification public key may be encrypted and stored in the persistent memory, so that the general processing device can first decrypt the signature verification public key after obtaining it. The key used to decrypt the signature verification public key may be a derived key of the hardware unique key HUK. The HUK may be stored in the one-time programmable memory.

[0106] In one possible example, the general processing device can decrypt the encrypted signature verification public key using the HUK's derived key to obtain a decrypted signature verification public key. In another possible example, when the signature verification public key is configured in the persistent memory of the data acceleration processing device, the signature verification public key can be encrypted using the HUK's derived key, and the encrypted signature verification public key can then be stored in the persistent memory.

[0107] S304: The acceleration processor of the data acceleration processing device decrypts the model weight in the ciphertext form, and stores the decrypted model weight in the plaintext form to the protected storage area of ​​the second memory. The general processing device cannot access the protected storage area.

[0108] The second memory may further include a protected storage area that is inaccessible to the general purpose processing device, which may also be referred to as a protected storage area that is inoperable by the general purpose processing device, that is, the general purpose processing device cannot read or write in the protected storage area.

[0109] In addition, the model weights in the model information are in ciphertext form. The acceleration processor of the data acceleration processing device can decrypt the ciphertext model weights stored in the unprotected storage area to obtain the model weights in plaintext. To protect the security of the plaintext model weights, the acceleration processor of the data acceleration processing device can also store the plaintext model weights in a protected storage area that is inaccessible to the general processing device.

[0110] Optionally, the specific process of the acceleration processor of the data acceleration processing device decrypting the model weights in the ciphertext form can be performed by a corresponding accelerator or operator. For example, the data acceleration processing device may include a decryption operator AICPU, which can be used to decrypt the model weights in the ciphertext form. In addition, the data acceleration processing device is also provided with a TEE environment, and the process of the data acceleration processing device decrypting the model weights in the ciphertext form can be performed in the TEE environment.

[0111] Furthermore, in a possible embodiment, the data acceleration processing device also includes a persistent memory, for example, the persistent memory may include a flash memory, and a decryption key is stored in the persistent memory. Accordingly, the acceleration processor of the data acceleration processing device decrypts the model weight in ciphertext form, which may specifically include: the acceleration processor obtains the decryption key from the persistent memory, and decrypts the model weight in ciphertext form according to the decryption key to obtain the model weight in plaintext form. The decryption key may be configured in the persistent memory, for example, in the process of configuring the model information in the external memory of the general processing device, the decryption key may be configured in the persistent memory of the data acceleration processing device.

[0112] Optionally, the decryption key may be encrypted and stored in the persistent memory, so that the acceleration processor of the data acceleration processing device can first decrypt the decryption key after obtaining it. The key used to decrypt the decryption key may be a derived key of the HUK. The data acceleration processing device further includes a one-time programmable memory, and the HUK may be stored in the one-time programmable memory.

[0113] In one possible example, the acceleration processor of the data acceleration processing device may decrypt the encrypted decryption key using the HUK derived key to obtain a decrypted decryption key. In another possible example, when the decryption key is configured in the persistent memory of the data acceleration processing device, the decryption key may be encrypted using the HUK derived key, and the encrypted decryption key may then be stored in the persistent memory.

[0114] After the acceleration processor of the data acceleration processing device stores the decrypted plaintext model weights in the protected storage area of ​​the second memory, as shown in FIG7 , the method may further include S305 .

[0115] S305: The acceleration processor of the data acceleration processing device executes multiple computing tasks corresponding to the model information according to the model weight in the plain text form.

[0116] When the acceleration processor of the data acceleration processing device decrypts the model weights in plain text, the acceleration processor can execute the multiple computing tasks based on the model weights. Optionally, after completing the multiple computing tasks, the data acceleration processing device can also return the computing results corresponding to the multiple computing tasks to the general processing device.

[0117] In an embodiment of the present application, the general processing device can transmit model information to the data acceleration processing device through a bus, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the computing task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related art, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0118] Furthermore, as shown in FIG7 , before S305 , the method further includes: S306 to S309 .

[0119] S306: The acceleration processor of the data acceleration processing device sends the base address of the model weight in plain text in the protected storage area to the general processing device.

[0120] Among them, the model weight in plain text can be stored continuously in the protected storage area, and can be arranged according to the calculation order in the multiple computing tasks. For example, the model weight may include multiple weights, and the arrangement order of the multiple weights in the protected storage area is consistent with the calculation order of the multiple weights in the multiple computing tasks. The base address can also be called the first address, specifically refers to the logical address of the first weight among the multiple weights included in the model weight. The logical address can refer to the address used by the processor (for example, an accelerated processor) to find the data when accessing the data.

[0121] For example, taking the example that the graph calculation information in the model information includes multiple computing nodes and the calculation order corresponding to the multiple computing nodes, the model weight includes multiple weights and the multiple weights are the weights used by the multiple computing nodes in the calculation process, then the arrangement order of the model weights in plain text in the protected storage area can be as shown in Figure 8.

[0122] In a possible embodiment, after storing the model weights in plain text to the protected storage area, the acceleration processor of the data acceleration processing device can determine the base address of the model weights in plain text in the protected storage area based on the position of the first weight in the model weights in the protected storage area, and send the base address to the general processing device through the bus.

[0123] S307: The general processing device receives the base address of the model weight in plain text form in the protected storage area sent by the data acceleration processing device, and determines the logical address of the model weight according to the base address and the address offset value of the model weight.

[0124] Among them, the model weights in the encrypted form in the general processing device are also stored continuously and arranged according to the calculation order in the multiple computing tasks. For example, the model weights may include multiple weights, and the arrangement order of the multiple weights in the general processing device is consistent with the calculation order of the multiple weights in the computing task. The address offset value of the model weight may include the address offset value of any one of the multiple weights. Taking the i-th weight among the multiple weights as an example, the address offset value of the i-th weight may refer to the offset value between the i-th logical address of the i-th weight and the base address corresponding to the first weight, and i is an integer in the value range.

[0125] For example, taking the example that the graph calculation information in the model information includes multiple computing nodes and the calculation order corresponding to the multiple computing nodes, the model weight includes multiple weights and the multiple weights are the weights used by the multiple computing nodes in the calculation process, then the arrangement order of the model weights in ciphertext form in the general processing device can be as shown in Figure 9.

[0126] When the encrypted model weights are continuously stored in the first memory of the general-purpose processing device, the general-purpose processing device may determine the address offset value of any one of the model weights based on the order in which the encrypted model weights are stored. The address offset value of a weight may be the offset value between the logical address of the weight and the base address corresponding to the first weight in the order in which the model weights are arranged.

[0127] In a possible embodiment, when the general processing device receives the base address of the model weight in plain text in the protected storage area, the general processing device can determine the logical address of the weight in the protected storage area based on the base address and the address offset value of any one of the model weights.

[0128] S308: The general processing device sends the logical address of the model weight to the data acceleration processing device.

[0129] When the general processing device determines the logical address of each weight in the model weights, the general processing device may send the logical address of each weight in the model weights to the data acceleration processing device via a bus. In one possible embodiment, the general processing device may send the logical address of each weight in the model weights required for executing each computing task to the data acceleration processing device in the order in which the data acceleration processing device executes the aforementioned multiple computing tasks.

[0130] S309: The acceleration processor of the data acceleration processing device receives the logical address of the model weight, and obtains the model weight in plain text from the protected storage area according to the logical address.

[0131] When the acceleration processor of the data acceleration processing device receives the logical address of each weight in the model weight, the acceleration processor of the data acceleration processing device can obtain the corresponding model weight in plain text from the protected storage area according to the logical address when executing the multiple computing tasks, and execute the multiple computing tasks according to the model weight in plain text.

[0132] In another possible embodiment, the acceleration processor of the data acceleration processing device may also determine the logical address of the model weight in the following manner: the general processing device sends the address offset value of the model weight to the data acceleration processing device through the bus; the acceleration processor of the data acceleration processing device receives the address offset value of the model weight through the bus, and determines the logical address of the model weight based on the base address of the model weight in plain text in the protected storage area and the address offset value.

[0133] Furthermore, the unprotected storage area and the protected storage area may be pre-configured by those skilled in the art. Optionally, the protected storage area may be a statically divided storage area with continuous physical addresses, and the capacity of the protected storage area may be configurable.

[0134] Exemplarily, a non-protected storage area accessible to the general processing device is pre-configured in the second memory, and in this case, the storage area other than the non-protected storage area in the second memory can be the protected memory; or, a protected storage area inaccessible to the general processing device is pre-configured in the second memory, and in this case, the storage area other than the protected storage area in the second memory can be the non-protected memory; or, both the non-protected storage area and the protected storage area are configured in the second memory at the same time.

[0135] In one possible embodiment, the bus includes a management channel. Before obtaining the model information, the general-purpose processing device may also send memory configuration information to the data acceleration processing device via the management channel. The memory configuration information is used to configure the unprotected storage area and / or the protected storage area. In this way, the acceleration processor of the data acceleration processing device can receive the memory configuration information via the management channel and configure the unprotected storage area and / or the protected storage area in the second memory based on the memory configuration information. The following explains the relevant content of configuring the protected storage area.

[0136] When configuring the unprotected storage area and the protected storage area, the second memory of the data acceleration processing device may be cleaned first, that is, the data in the second memory may be cleaned to prevent the residual data in the second memory from being leaked.

[0137] In addition, the size of the protected storage area is configured through the management channel, and a storage area with continuous physical addresses is statically divided as the protected storage area.

[0138] Secondly, after configuring the protected storage area, address mapping is performed. The address of the protected storage area does not participate in the address mapping corresponding to the bus. For example, if the bus is a PCIe bus, the page table corresponding to the PCIe bus (also referred to as a PCIe page table) does not include the address mapping of the protected storage area, so that the general processing device cannot access the protected storage area through the bus. However, when mapping the page table of the acceleration processor in the data acceleration processing device, the address of the protected storage area and the address of the unprotected storage area can both participate in the mapping. That is, the page table corresponding to the acceleration processor in the data acceleration processing device includes both the address mapping of the protected storage area and the address mapping of the unprotected storage area. This ensures that the acceleration processor of the data acceleration processing device can access both the unprotected storage area and the protected storage area.

[0139] In addition, when configuring the non-protected storage area and the protected storage area, if there are other running services in the electronic device, the device can be reset to forcibly interrupt the execution of the service.

[0140] In the above embodiment, the technical solution of the embodiment of the present application is described from the perspective of the interaction between the general processing device and the data acceleration processing device. In actual applications, the steps corresponding to the general processing device can be implemented by the processor in the general processing device and the inference framework running on the processor, and the steps corresponding to the data acceleration processing device can be implemented by the acceleration processor in the data acceleration processing device and the firmware running on the acceleration processor. For ease of understanding, the technical solution of the embodiment of the present application is illustrated below by taking the structures of the electronic devices shown in Figures 10 to 12 as examples. Figures 10 to 12 are taken as an example for explanation, with the processor in the general processing device including a CPU.

[0141] In one example, as shown in FIG10 , the general processing device includes a CPU and a first memory, the data acceleration processing device includes an acceleration processor and a second memory, the second memory includes a protected storage area and an unprotected storage area, and the acceleration processor includes multiple accelerators and a decryption operator AICPU. Accordingly, the method may include: S11. The CPU of the general processing device copies the model information to the non-protected storage area of ​​the data acceleration processing device through the bus, and the model information includes calculation graph information (plaintext), model weight (ciphertext) and calculation task (plaintext); S12. The acceleration processor of the data acceleration processing device schedules the decryption operator AICPU to decrypt the model weight, and stores the decrypted model weight (plaintext) to the protected storage area in the memory of the data acceleration processing device; S13. The acceleration processor of the data acceleration processing device sends the base address of the model weight in the protected storage area to the general processing device through the bus; S14. When receiving the base address, the CPU of the general processing device determines the address offset value of the model weight; S15. The CPU of the general processing device determines the logical address of the model weight based on the base address and the address offset value, and sends the logical address to the data acceleration processing device through the bus; S16. The acceleration processor of the data acceleration processing device obtains the model weight from the protected memory according to the logical address, and executes the calculation task according to the model weight.

[0142] In another example, in combination with FIG10 , as shown in FIG11 , the general processing device is coupled to the data acceleration processing device via a PCIe bus. The general processing device further includes an inference architecture and a device management driver. The data acceleration processing device further includes flash memory, a one-time programmable memory, and firmware. Accordingly, the method may further include: S21. Configuring the data acceleration processing device via the device management driver. The configuration may include configuring a decryption key and a signature verification public key in the flash memory, and encrypting and protecting the decryption key and the signature verification public key using a derived key of the HUK in the one-time programmable memory; S22. Configuring a protected storage area in the memory of the data acceleration processing device via the device management driver. The PCIe page table only maps the address of the unprotected storage area, and the page table of the acceleration processor in the data acceleration processing device simultaneously maps the address of the unprotected storage area and the address of the protected storage area.

[0143] In another example, in combination with Figure 11, as shown in Figure 12, the method may also include: S31. The inference architecture in the general processing device loads the model information, and copies the model information to the non-protected storage area of ​​the data acceleration processing device through the bus; S32. The firmware in the data acceleration processing device obtains the signature verification public key from the flash memory and verifies the signature information of the model information; S33. After the signature verification is successful, the firmware in the data acceleration processing device obtains the decryption key from the flash memory to decrypt the model weights in ciphertext form stored in the non-protected storage area, and stores the decrypted model weights in plaintext form in the protected storage area; S34. The firmware in the data acceleration processing device loads the model weights in plaintext and executes the computing task.

[0144] In the above example, in order to configure the inference framework to protect the model information during runtime, the firmware in the data acceleration processing device provides the following functions: a. Model information signature verification: protects the integrity of the model information and prevents the model information from being tampered with and implanted with malicious operators; b. Model weight decryption: protects the confidentiality of the model weight; c. Protected storage area: isolates a storage area in the second memory of the data acceleration processing device, which can only be accessed by the acceleration processor of the data acceleration processing device and cannot be accessed by the general processing device through the bus; in addition, storing the decrypted plaintext model weights in the protected storage area in the memory can improve the reading efficiency of the acceleration processor; d. Key management: encrypts and manages the signature public key and decryption key to ensure the confidentiality of the key.

[0145] In an embodiment of the present application, the general processing device can transmit model information to the data acceleration processing device through a bus, and the model weights in the model information are in ciphertext form, thereby ensuring the security of the model weights in the general processing device and the security during the transmission process; in addition, after the data acceleration processing device decrypts the ciphertext model weights, it stores the decrypted plaintext model weights in a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the plaintext model weights, so that when the data acceleration processing device performs the computing task according to the plaintext model weights, it can ensure the security of the model weights during execution. Therefore, compared with the related art, the embodiment of the present application can be used to protect the security of model information during operation without using hardware isolation technologies such as TEE, thereby preventing the model information from being stolen.

[0146] Based on this, an embodiment of the present application also provides another data processing method, which can be applied to the data acceleration processing device described above. As shown in FIG13 , the data processing method includes the following steps.

[0147] S320: The acceleration processor of the data acceleration processing device obtains the target task sent by the general processing device based on the model information.

[0148] The target task may be any one of the multiple computing tasks corresponding to the model information. Optionally, the target task may include a computing type and operand information. For example, the computing type may be one of addition, subtraction, exclusive-or, or, and. The operand information may be used to indicate storage information of input data corresponding to the target task, and the storage information may be used to obtain the input data.

[0149] S321: The data acceleration processing device obtains a first logical address of the target weight, and reads the target weight from the protected storage area according to the first logical address. The general processing device cannot access the protected storage area.

[0150] In a possible embodiment, the acceleration processor obtains the first logical address of the target weight, which may include: the acceleration processor determines the base address of the model weight in plain text in the protected storage area; the acceleration processor sends the base address to the general processing device, and the base address is used to determine the first logical address of the target weight; the acceleration processor receives the first logical address of the target weight sent by the general processing device.

[0151] The target weight is the weight in the model weight that needs to be used when the target task is executed, that is, the model weight includes the target weight. The first logical address of the target weight specifically refers to the logical address of the target weight in the protected storage area.

[0152] S322: The acceleration processor of the data acceleration processing device executes the target task according to the target weight.

[0153] In one possible embodiment, the acceleration processor of the data acceleration processing device may further obtain corresponding operands based on operand information included in the target task; the acceleration processor may execute the target task based on the obtained target weight, operands, and the computation type included in the target task. The acceleration processor may then transmit the execution result corresponding to the target task to the general processing device.

[0154] It can be understood that all relevant contents of each step in the method embodiments corresponding to Figures 5 to 12 above can be referred to the description of the method embodiment corresponding to Figure 13, and the embodiments of this application will not be repeated here.

[0155] In an embodiment of the present application, the data acceleration processing device can store the model weight in plain text obtained by decrypting the model weight in cipher text into a protected storage area that is inaccessible to the general processing device, that is, the general processing device cannot access the model weight in plain text. In this way, when the data acceleration processing device executes the corresponding target task according to the model weight in plain text, it can ensure the security of the model weight during execution.

[0156] The above mainly introduces the solution provided by the embodiment of the present application from the perspective of the interaction between the general processing device and the data acceleration processing device. It can be understood that, in order to realize the above functions, the general processing device and the data acceleration processing device include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in a hardware or computer software driven hardware manner depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0157] The embodiment of the present application can divide the functional modules of the general processing device and the data acceleration processing device according to the above method example. For example, each functional module can be divided according to each function, or two or more functions can be integrated into one processing module. The above functional modules can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present application is schematic and is only a logical functional division. There may be other division methods in actual implementation. The following is an example of dividing each functional module according to the corresponding function:

[0158] In the case of adopting an integrated unit, Figure 14 shows a possible structural diagram of the data acceleration processing device involved in the above embodiment. The data acceleration processing device may include: a storage unit 401 and a processing unit 402. In a possible embodiment, the storage unit 401 can be used to support the data acceleration processing device to execute S303 in the above method embodiment; the processing unit 402 can be used to support the device to execute S304, S305, and S309 in the above method embodiment. Further, the data acceleration processing device also includes a sending unit 403; wherein the sending unit 403 can be used to support the data acceleration processing device to execute S306 in the above method embodiment. In a possible embodiment, the processing unit 402 can be used to support the data acceleration processing device to execute S320 to S322 in the above method embodiment.

[0159] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0160] Based on the hardware implementation, the processing unit 402 may be an acceleration processor, the sending unit 403 may be a transmitter, the transmitter and the receiver may be integrated into a transceiver, and the transceiver may also be called a communication interface.

[0161] Figure 15 is a possible structural diagram of the data acceleration processing device involved in an embodiment of the present application, and the data acceleration processing device may include: a memory 411 and an acceleration processor 412. Among them, the memory 411 is used to store the program code and data of the device, and the acceleration processor 412 is used to control the actions of the data acceleration processing device in the above method embodiment. For example, the acceleration processor 412 is used to support the device to execute S304, S305 and / or S309 in the above method embodiment, or execute S320 to S322 in the above method embodiment, and / or other processes for the technology described herein. Optionally, the data acceleration processing device may also include a communication interface 413, which is used to support the device to execute the steps of communicating with the general processing device in the above method embodiment.

[0162] The acceleration processor 412 may be a GPU, NPU, DSP, CPU, application-specific integrated circuit, processing chip, field programmable gate array or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute various logic blocks, modules, and circuits described in conjunction with the embodiments of this application. The acceleration processor 412 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like.

[0163] Memory 411 can be volatile memory or non-volatile memory. Optionally, memory 411 can be integrated into acceleration processor 412. In one example, memory 411 can include internal memory, persistent memory, and one-time programmable memory. For example, the persistent memory can include flash memory, which can be used to store the decryption key and / or signature verification public key described below, and the one-time programmable memory can be used to store the hardware unique key HUK described below.

[0164] In the case of an integrated unit, Figure 16 shows a possible structural diagram of the general processing device involved in the above-mentioned embodiments. The device may include: a processing unit 501 and a sending unit 502. In one possible embodiment, the processing unit 501 is used to support the device in executing S301 and the step of determining the logical address in S307 of the above-mentioned method embodiment; the sending unit 502 is used to support the device in executing S302 and S308 of the above-mentioned method embodiment. Furthermore, the general processing device also includes a receiving unit 503; wherein the receiving unit 503 is used to support the general processing device in executing the step of receiving the base address in S307 of the above-mentioned method embodiment.

[0165] All relevant contents of each step involved in the above method embodiment can be referred to the functional description of the corresponding functional module and will not be repeated here.

[0166] Based on hardware implementation, the processing unit 501 may be a processor, the sending unit 502 may be a transmitter, the receiving unit 503 may be a receiver, the receiver and the transmitter may be integrated into a transceiver, which may also be called a communication interface.

[0167] Figure 17 is a possible structural diagram of a general processing device involved in an embodiment of the present application, and the device includes: a memory 511 and a processor 512. Among them, the memory 511 is used to store the program code and data of the device, and the processor 512 is used to control the actions of the general processing device in the above method embodiment. For example, the processor 512 is used to execute the steps of determining the logical address in S301 and S307 in the above method embodiment, and / or other processes for the technology described herein. Optionally, the device may also include a communication interface 513, which is used to support the device in executing the steps of communicating with the data acceleration processing device in the above method embodiment.

[0168] The processor 512 may be a central processing unit, a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a processing chip, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. It may implement or execute various logic blocks, modules, and circuits described in conjunction with the disclosure of the embodiments of this application. The processor 512 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, and the like. The communication interface 513 may be a transceiver, a transceiver circuit, or a transceiver interface, and the like. The memory 511 may be a volatile memory or a non-volatile memory, and the like.

[0169] For example, communication interface 513, processor 512, and memory 511 are interconnected via bus 514; bus 514 may be a PCI bus or an EISA bus, for example. Bus 514 may be classified as an address bus, a data bus, a control bus, and the like. For ease of illustration, the figure uses only one thick line, but this does not necessarily indicate that there is only one bus or only one type of bus.

[0170] Optionally, the memory 511 may be included in the processor 512 .

[0171] In another aspect of the present application, an electronic device is provided, comprising any of the data acceleration processing devices provided above and any of the general processing devices provided above. The data acceleration processing device is configured to execute the steps of the data acceleration processing device in the above method embodiment; and the general processing device is configured to execute the steps of the general processing device in the above method embodiment.

[0172] In another aspect of the present application, a terminal device is provided, comprising any of the data acceleration processing devices provided above and any of the general processing devices provided above. The data acceleration processing device is configured to execute the steps of the data acceleration processing device in the above method embodiment; and the general processing device is configured to execute the steps of the general processing device in the above method embodiment.

[0173] Exemplarily, the terminal device may be a mobile phone, a tablet, a computer, a camera, a wearable device, or a vehicle-mounted device.

[0174] In the method provided in the embodiment of the present application, it can be implemented in whole or in part by software, hardware or a combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in accordance with the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a network device or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, a computer, a server or a data center to another website, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, twisted pair) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any medium that a computer can access or a data storage device such as a server or a data center that includes one or more media integrations. The medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., an optical disk) or a semiconductor medium (e.g., a solid-state hard disk), etc.

[0175] In another aspect of the present application, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes one or more steps of the data acceleration processing device in the above method embodiment.

[0176] In another aspect of the present application, a computer-readable storage medium is provided, which includes computer instructions. When the computer instructions are executed by a device, the device executes one or more steps of the general processing device in the data processing method provided by the above method embodiment.

[0177] In another aspect of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute one or more steps of the data acceleration processing device in the above method embodiment.

[0178] In another aspect of the present application, a computer program product comprising instructions is provided, which, when executed on a computer, enables the computer to execute one or more steps of the general processing device in the above method embodiment.

[0179] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that: Applied in a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area and a protected storage area, the method includes: The memory receives model information sent by the general processing device, and stores the model information in the unprotected storage area, the general processing device can access the unprotected storage area, and the model information includes model weights in ciphertext form; The acceleration processor decrypts the model weights in ciphertext form, and stores the decrypted model weights in plaintext form in the protected storage area, and the general processing device cannot access the protected storage area.

2. The method according to claim 1, characterized in that The data acceleration processing device also includes a persistent memory, wherein the persistent memory stores a decryption key; The acceleration processor decrypts the ciphertext model weights, including: The acceleration processor obtains the decryption key from the persistent memory, and decrypts the ciphertext model weights according to the decryption key.

3. The method according to claim 1 or 2, characterized in that: The data acceleration processing device further includes a persistent memory, wherein the persistent memory stores a signature verification public key, and the model information further includes signature information; the method further includes: The acceleration processor obtains the signature verification public key from the persistent memory, and determines, based on the signature verification public key, that the signature information of the model information has been successfully verified.

4. The method according to claim 2 or 3, characterized in that: The decryption key and / or the signature verification public key are stored in encrypted form, the data acceleration processing device further includes a one-time programmable memory, and the one-time programmable memory stores a hardware unique key HUK; the method further includes: The acceleration processor obtains the HUK from the one-time programmable memory, and encrypts and decrypts the decryption key and / or the signature verification public key according to a derived key of the HUK.

5. The method according to any one of claims 1 to 4, characterized in that: The model weights are stored continuously in the protected storage area, and the method further comprises: The acceleration processor sends the base address of the model weight in plain text in the protected storage area to the general processing device, where the base address is used to determine the logical address of the model weight; The acceleration processor receives the logical address of the model weight sent by the general processing device, and obtains the model weight in plain text from the protected storage area according to the logical address.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The acceleration processor receives memory configuration information, and configures the unprotected storage area and / or the protected storage area according to the memory configuration information.

7. A data processing method, characterized in that: Applied in a general processing device, the general processing device is used to communicate with a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device, the method includes: Obtaining model information, wherein the model information includes a model weight in ciphertext form; The model information is sent to the data acceleration processing device so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor stores the model weight in plain text after decrypting the ciphertext model weight in the protected storage area.

8. The method according to claim 7, characterized in that The model information also includes signature information, and the signature information is used to verify the model information.

9. The method according to claim 7 or 8, characterized in that: The model weights are stored continuously in the protected storage area, and the method further comprises: Receiving the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; Determining a logical address of the model weight according to the base address and the address offset value of the model weight; The logical address of the model weight is sent to the data acceleration processing device.

10. The method according to any one of claims 7 to 9, characterized in that: The method further comprises: Memory configuration information is sent to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

11. A data processing method, characterized in that: The method is applied to a data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes an unprotected storage area and a protected storage area, the unprotected storage area includes model information, the model information includes a model weight in ciphertext form, and the protected storage area includes a model weight in plaintext form after the ciphertext model weight is decrypted. The method includes: The acceleration processor obtains a target task sent by the general processing device based on the model information; The acceleration processor obtains a first logical address of a target weight, and reads the target weight from the protected storage area according to the first logical address, wherein the target weight is a weight in the model weight required to be used when the target task is executed, and the general processing device cannot access the protected storage area; The acceleration processor executes the target task according to the target weight.

12. The method according to claim 11, characterized in that The acceleration processor obtains the logical address of the target weight, including: The acceleration processor determines a base address of the model weight in plain text in the protected storage area; The acceleration processor sends the base address to the general processing device, where the base address is used to determine the first logical address of the target weight; The acceleration processor receives the first logical address of the target weight sent by the general processing device.

13. A data acceleration processing device, characterized in that: The data acceleration processing device includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; The storage unit is used to receive the model information sent by the general processing device and store the model information in the unprotected storage area, the general processing device can access the unprotected storage area, and the model information includes the model weight in ciphertext form; The processing unit is used to decrypt the model weights in ciphertext form and store the decrypted model weights in plaintext form in the protected storage area. The general processing device cannot access the protected storage area.

14. The data acceleration processing device according to claim 13, characterized in that: The data acceleration processing device also includes a persistent memory, wherein the persistent memory stores a decryption key; The processing unit is further configured to obtain the decryption key from the persistent memory, and decrypt the ciphertext model weight according to the decryption key.

15. The data acceleration processing device according to claim 13 or 14, characterized in that: The data acceleration processing device also includes a persistent memory, the persistent memory stores a signature verification public key, and the model information also includes signature information; The processing unit is further configured to obtain the signature verification public key from the persistent storage, and determine, based on the signature verification public key, whether the signature information of the model information has been successfully verified.

16. The data acceleration processing device according to claim 14 or 15, characterized in that: The decryption key and / or the signature verification public key are stored in encrypted form, and the data acceleration processing device further includes a one-time programmable memory, which stores a hardware unique key HUK; The processing unit is further configured to obtain the HUK from the one-time programmable memory, and encrypt and decrypt the decryption key and / or the signature verification public key according to a derived key of the HUK.

17. The data acceleration processing device according to any one of claims 13 to 16, characterized in that: The model weights are stored continuously in the protected storage area; The processing unit is further used to send the base address of the model weight in plain text form in the protected storage area to the general processing device, where the base address is used to determine the logical address of the model weight; The processing unit is further used to receive the logical address of the model weight sent by the general processing device, and obtain the model weight in plain text from the protected storage area according to the logical address.

18. The data acceleration processing device according to any one of claims 13 to 17, characterized in that: The processing unit is further used to receive memory configuration information and configure the unprotected storage area and / or the protected storage area according to the memory configuration information.

19. A general processing device, characterized in that: The general processing device is used to communicate with the data acceleration processing device, the data acceleration processing device includes an acceleration processor and a memory, the memory includes a non-protected storage area accessible to the general processing device and a protected storage area inaccessible to the general processing device, and the general processing device includes: A processing unit, configured to obtain model information, wherein the model information includes a model weight in a ciphertext form; A sending unit is used to send the model information to the data acceleration processing device, so that the memory in the data acceleration processing device stores the model information in the unprotected storage area, and the acceleration processor stores the model weight in plain text after decrypting the ciphertext model weight in the protected storage area.

20. The universal processing device according to claim 19, characterized in that: The model information also includes signature information, and the signature information is used to verify the model information.

21. The universal processing device according to claim 19 or 20, characterized in that: The model weights are stored continuously in the protected storage area, and the general processing device further comprises a receiving unit; The receiving unit is used to receive the base address of the model weight in plain text in the protected storage area sent by the data acceleration processing device; The processing unit is further used to determine the logical address of the model weight according to the base address and the address offset value of the model weight; The sending unit is also used to send the logical address of the model weight to the data acceleration processing device.

22. The universal processing device according to any one of claims 19 to 21, characterized in that: The sending unit is further used to send memory configuration information to the data acceleration processing device, where the memory configuration information is used to configure the unprotected storage area and / or the protected storage area.

23. A data acceleration processing device, characterized in that: The data acceleration processing device includes a processing unit and a storage unit, and the storage unit includes a non-protected storage area and a protected storage area; The storage unit is used to store model information in the unprotected storage area, wherein the model information includes model weights in ciphertext form, and the protected storage area is not accessible to a general processing device; The storage unit is further used to store the model weight in plain text corresponding to the model weight in cipher text in the protected storage area, and the general processing device cannot access the protected storage area; The processing unit is used to obtain the target task sent by the general processing device based on the model information; The processing unit is further used to obtain a first logical address of a target weight, and read the target weight from the protected storage area according to the first logical address, wherein the target weight is a weight in the model weight required to be used when the target task is executed, and the general processing device cannot access the protected storage area; The processing unit is further configured to execute the target task according to the target weight.

24. The data acceleration processing device according to claim 23, characterized in that: The data acceleration processing device also includes a sending unit and a receiving unit; The processing unit is further used to determine a base address of the model weight in plain text in the protected storage area; The sending unit is further used to send the base address to the general processing device, where the base address is used to determine the first logical address of the target weight; The receiving unit is used to receive the first logical address of the target weight sent by the general processing device.

25. A data acceleration processing device, characterized in that: The data acceleration processing device includes an acceleration processor and a memory, wherein instructions are stored in the memory. When the acceleration processor runs the instructions, the device executes the data processing method as described in any one of claims 1 to 6, or executes the data processing method as described in any one of claims 11 to 12.

26. A general processing device, characterized in that: The general processing device includes a processor and a memory, wherein instructions are stored in the memory. When the processor executes the instructions, the device executes the data processing method according to any one of claims 7 to 10.

27. An electronic device, characterized in that: The electronic device comprises: a data acceleration processing device as described in any one of claims 13-18, any one of claims 23-24, or claim 25, and a general processing device as described in any one of claims 19-22 or claim 26.

28. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed by a device, the device executes the data processing method according to any one of claims 1 to 12.

Images