Trusted delivery method of OVD multimodal large model parameters based on device-edge-cloud collaboration

Abstract

This application relates to a method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration, wherein the method includes: dividing the OVD multimodal large model through the cloud center and caching part of the model to the edge server, the edge device performing model obfuscation and saving the obfuscated model to the TEE; when the smart device requests the model from the edge server, the edge server delivers the cached model to the smart device, and the smart device loads part of the OVD model for real-time inference, thereby making full use of the resources at all layers of the end-edge-cloud and realizing the trusted delivery of OVD multimodal large model parameters. This solves the problems that the existing technology is not applicable to smart security scenarios and OVD multimodal large model deployment of massive smart devices, and lacks an effective model asset protection mechanism.

Description

Technical Field

[0001] The present application relates to the field of information security technology, and in particular to a method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration. Background Art

[0002] With the rapid development of artificial intelligence large models, the Open Vocabulary Detection (OVD) large model has become a new paradigm in the field of target detection. It combines multimodal visual-language large models (Visual Language Models, VLMs) with generalized region detectors to align image region features with target description text across modalities, thereby achieving target detection for unknown objects in the open world. Compared with traditional target detection methods (such as the YOLO series), the significant advantages of the OVD large model can be summarized as follows: only natural language input is required to locate the corresponding target in the video. The core requirement in smart security scenarios is the ability to accurately and quickly lock on to the target person in complex scenarios, such as Figure 1 The figure shows the difference between using the OVD large model and traditional target detection in the search for the specified target task of "young men wearing red shirts and riding balance bikes" in smart security scenarios. The significant advantage of the OVD large model can be summarized as: it can quickly respond to public security investigation tasks only through natural language descriptions. Because the OVD multimodal large model can further unleash its performance advantages, reduce labor costs and improve response speed, it is imperative to empower smart security with the OVD multimodal large model.

[0003] Smart security scenarios are generally composed of cloud centers, edge servers and smart devices. The cloud center has massive computing resources and is responsible for the pre-training and segmentation of OVD multimodal large models and the access authentication of smart devices. It is composed of a proprietary cloud service platform; the edge server has certain computing resources and is equipped with a trusted execution environment (TEE), which is responsible for message aggregation and forwarding. It is composed of base stations; smart devices have limited computing resources and are equipped with TEE, which is responsible for model inference and reporting of inference results. It is composed of police terminals such as Skynet cameras, police communications, and police cars.

[0004] Currently, the deployment of large models in smart security scenarios and the implementation of OVD multimodal large models are key steps in realizing the empowerment of smart security by large models. The existing deployment solution uses a cloud center to directly deploy to smart devices in a point-to-point manner, which has the following shortcomings:

[0005] (1) Poor scalability and does not support one-to-many batch deployment. The existing solution uses a point-to-point deployment method, which is only suitable for scenarios with a small number of smart devices and cannot be applied to smart security scenarios with massive smart devices;

[0006] (2) It does not support the deployment of large OVD models. Existing solutions deploy the entire model to smart devices. However, resource-constrained smart devices cannot support the computational overhead of a complete OVD multimodal large model. Therefore, existing methods are not suitable for the deployment of large OVD multimodal models.

[0007] (3) Lack of effective model asset protection mechanisms. Existing deployment solutions assume that models are transmitted in a secure transmission environment. However, the network transmission environment in actual smart security scenarios is complex and subject to various potential attacks. For example, adversaries can steal or tamper with models by monitoring open network data packets. Therefore, the security assumptions of deployment solutions are too idealistic and do not match the actual scenarios, and cannot meet the security requirements of smart security scenarios.

[0008] In summary, existing technologies are not applicable to smart security scenarios with massive smart devices and the deployment of large multimodal OVD models, and lack an effective model asset protection mechanism, which needs to be urgently addressed. Summary of the Invention

[0009] This application provides a method for trusted distribution of OVD multimodal large model parameters based on end-edge-cloud collaboration to solve the problems that existing technologies are not applicable to smart security scenarios of massive smart devices and OVD multimodal large model deployment, and lack an effective model asset protection mechanism.

[0010] The first aspect of the present application provides a method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration, comprising the following steps: signing the image encoder of the OVD multimodal large model through a preset cloud center, and caching the signed image encoder to a target edge server, and the target edge server generates a blinding factor, and uses the blinding factor to perform model confusion on the image encoder to obtain confused image encoder model parameters, and at the same time, caches the blinding factor and the confused image encoder model parameters to the TEE and REE in the target edge server respectively; using the cloud center to perform batch identity authentication on preset smart devices to obtain smart device identity authentication results, the target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the confused image encoder model parameters and the encrypted blinding factor to the smart device, and sends the confused image encoder model parameters and the encrypted blinding factor to the smart device. The blinding factors are loaded into the smart device REE and the smart device TEE respectively, and the credibility and integrity of the obfuscated image encoder model parameters and the encrypted blinding factors are verified; when the obfuscated image encoder model parameters and the encrypted blinding factors pass the credibility and integrity verification, the target text task is generated through the cloud center, and the text encoder of the OVD multimodal large model is used to obtain the text embedding features of the target text task, and the text embedding features are sent to the target edge server, and the target edge server is used to send an inference request to the smart device, so as to perform model inference on the image encoder through the smart device to obtain image embedding features, and at the same time, the image embedding features and the text embedding features are compared using a preset cross-modal feature comparison strategy to obtain target image embedding features that meet the preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task.

[0011] Optionally, in one embodiment of the present application, before signing the image encoder of the OVD multimodal large model through a preset cloud center and caching the signed image encoder to the target edge server, it also includes: registering with a preset key management center through the cloud center, the target edge server and the smart device respectively to generate a public key and a private key corresponding to the cloud center and the target edge server and a prefabricated shared secret value of the smart device; based on the public key and the private key, establishing a session handshake between the cloud center and the target edge server to complete the identity authentication of the target edge server by the cloud center; using the cloud center to perform a partition operation on the OVD multimodal large model to deploy the text encoder of the OVD multimodal large model to the cloud center, and deploy the image encoder to the smart device.

[0012] Optionally, in one embodiment of the present application, the image encoder of the OVD multimodal large model is signed by a preset cloud center, and the signed image encoder is cached to a target edge server, and the target edge server generates a blinding factor to use the blinding factor to perform model confusion on the image encoder to obtain the confused image encoder model parameters, including: obtaining a target network layer in the image encoder that meets the preset importance requirements through the cloud center, and constructing a corresponding network layer index set based on the target network layer; randomly selecting random values ​​corresponding to the target edge server and the cloud center, and based on the network layer index set and the random value, sign the network layer parameters in the image encoder, encrypt the network layer parameters to obtain encrypted network layer parameters, and cache the signed image encoder model parameters to the target edge server; decrypt the encrypted network layer parameters by using the prefabricated shared secret value through the target edge server, and verify the validity of the signature of the network layer parameters according to a preset signature validity condition; when the signature of the network layer parameters meets the preset signature validity condition, generate the blinding factor in the TEE of the edge server to obfuscate the network layer parameters by the blinding factor to obtain the obfuscated image encoder model parameters.

[0013] Optionally, in one embodiment of the present application, the cloud center is used to perform batch authentication on preset smart devices to obtain smart device identity authentication results, and the target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and at the same time verifies the credibility and integrity of the obfuscated image encoder model parameters and the encrypted blinding factor, including: through the smart device Send a model request response to the target edge server, so that the target edge server sends the aggregated model request response to the cloud center after aggregating the model request response; use the cloud center to generate a challenge value according to the aggregated model request response, and send the challenge value to the smart device through the target edge server, so that the smart device uses the challenge value to calculate the initialization program measurement value of the smart device TEE to generate a response value, and send the response value to the cloud center; batch authenticate the identity of the smart device through the cloud center to obtain the smart device identity authentication result, and The target edge server generates a group session key with the target device, signs the group session key, and sends the signed target message to the target device. The target device calculates the group session key with the target edge server based on the target message, authenticates the target edge server, and performs hash processing on the group session key to obtain a hash processing result, and returns the hash processing result to the target edge server that has passed the authentication. Based on the hash processing result, the target edge server generates a group session key with the target device, and signs the group session key. The method comprises the following steps: performing identity authentication on the smart device, encrypting the blinding factor by using the group session key, and sending the obfuscated image encoder model parameters and the encrypted blinding factor to the authenticated smart device; loading the encrypted blinding factor into the TEE of the smart device by using the smart device, decrypting the encrypted blinding factor by using the group session key in the TEE of the smart device to obtain the target blinding factor, and saving the obfuscation model parameters to the REE of the smart device; and performing credibility and integrity verification on the target blinding factor stored in the TEE of the smart device and the obfuscation model parameters stored in the REE of the smart device.

[0014] Optionally, in one embodiment of the present application, a target text task is generated by the cloud center, and the text encoder of the OVD multimodal large model is used to obtain the text embedding features of the target text task, and the text embedding features are sent to the target edge server, and the target edge server is used to send an inference request to the smart device, so that the smart device performs model inference on the image encoder to obtain image embedding features, and at the same time uses a preset cross-modal feature comparison strategy to compare the image embedding features and the text embedding features to obtain target image embedding features that meet preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task, including: generating the target text task by the cloud center, and inputting the target text task into the text encoder of the OVD multimodal large model to obtain the text embedding corresponding to the target text task. feature; encrypting the text embedding feature to generate text encryption embedding information, and sending the text encryption embedding information to the target edge server, so that the target edge server sends an inference request to the smart device according to the text encryption embedding information; based on the inference request, the smart device uses the obfuscated image encoder and performs an inference operation in combination with the target blinding factor to obtain the image embedding feature; encrypting the image embedding feature to obtain an image embedding encryption feature, and sending the image embedding encryption feature to the target edge server; decrypting the image embedding encryption feature through the target edge server to obtain an image embedding decryption feature; calculating the cross-modal feature similarity between the text embedding feature and the image embedding decryption feature, and using the image embedding decryption feature with the highest cross-modal feature similarity as the inference result of the OVD multimodal large model.

[0015] The second embodiment of the present application provides a device for trusted issuance of OVD multimodal large model parameters based on end-edge-cloud collaboration, including: an obfuscation module for signing the image encoder of the OVD multimodal large model through a preset cloud center, and caching the signed image encoder to a target edge server, and the target edge server generates a blinding factor to use the blinding factor to perform model obfuscation on the image encoder, obtain the obfuscated image encoder model parameters, and cache the blinding factor and the obfuscated image encoder model parameters to the TEE and REE in the target edge server respectively; a verification module for performing batch identity authentication on preset smart devices using the cloud center to obtain smart device identity authentication results, and the target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device. The encrypted blinding factors are loaded into the smart device REE and the smart device TEE respectively, and the credibility and integrity of the obfuscated image encoder model parameters and the encrypted blinding factors are verified; an inference module is used to generate a target text task through the cloud center when the obfuscated image encoder model parameters and the encrypted blinding factors pass the credibility and integrity verification, and obtain the text embedding features of the target text task by using the text encoder of the OVD multimodal large model, and send the text embedding features to the target edge server, and use the target edge server to send an inference request to the smart device, so as to perform model inference on the image encoder through the smart device to obtain image embedding features, and at the same time use a preset cross-modal feature comparison strategy to compare the image embedding features and the text embedding features to obtain target image embedding features that meet the preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task.

[0016] Optionally, in one embodiment of the present application, it also includes: a registration module, which is used to sign the image encoder of the OVD multimodal large model through the preset cloud center, and register with the preset key management center through the cloud center, the target edge server and the smart device respectively before caching the signed image encoder to the target edge server, so as to generate the public key and private key corresponding to the cloud center and the target edge server and the prefabricated shared secret value of the smart device; a handshake module, which is used to establish a session handshake between the cloud center and the target edge server based on the public key and the private key, so as to complete the identity authentication of the target edge server by the cloud center; a partitioning module, which is used to use the cloud center to perform a partition operation on the OVD multimodal large model, so as to deploy the text encoder of the OVD multimodal large model to the cloud center, and deploy the image encoder to the smart device.

[0017] Optionally, in one embodiment of the present application, the obfuscation module includes: a construction unit for obtaining a target network layer in the image encoder that meets a preset importance requirement through the cloud center, and constructing a corresponding network layer index set based on the target network layer; a first signature unit for randomly selecting random values ​​corresponding to the target edge server and the cloud center, respectively, and signing the network layer parameters in the image encoder based on the network layer index set and the random value, and encrypting the network layer parameters to obtain encrypted network layer parameters, and caching the signed image encoder model parameters to the target edge server; a first decryption unit for decrypting the encrypted network layer parameters using the prefabricated shared secret value through the target edge server, and verifying the validity of the signature of the network layer parameters according to a preset signature validity condition; a generation unit for generating the blinding factor in the TEE of the edge server when the signature of the network layer parameter meets the preset signature validity condition, so as to obfuscate the network layer parameters by the blinding factor to obtain the obfuscated image encoder model parameters.

[0018] Optionally, in one embodiment of the present application, the verification module includes: a convergence unit, which is used to send a model request response to the target edge server through the smart device, so that the target edge server sends the aggregated model request response to the cloud center after aggregating the model request response; a first calculation unit, which is used to use the cloud center to generate a challenge value according to the aggregated model request response, and send the challenge value to the smart device through the target edge server, so that the smart device uses the challenge value to calculate the initialization program measurement value of the smart device TEE to generate a response value, and send the response value to the cloud center; a second signature unit, which is used to batch authenticate the identity of the smart device through the cloud center to obtain a smart device identity authentication result, and send the smart device identity authentication result to the target edge server, so that the target edge server constructs a group session key between the smart device and the group session key, and signs the group session key, and sends the signed target message to the smart device; a hash unit, which is used to generate a challenge value based on the target The target edge server receives a target message, calculates a group session key with the target edge server through the smart device, authenticates the target edge server, and hashes the group session key to obtain a hash result, and returns the hash result to the authenticated target edge server; an authentication unit is used to enable the target edge server to authenticate the smart device based on the hash result, encrypt the blinding factor using the group session key, and send the obfuscated image encoder model parameters and the encrypted blinding factor to the authenticated smart device; a loading unit is used to load the encrypted blinding factor into the smart device TEE using the smart device, decrypt the encrypted blinding factor in the smart device TEE using the group session key to obtain a target blinding factor, and save the obfuscation model parameters to the smart device REE; an analysis unit is used to verify the credibility and integrity of the target blinding factor stored in the smart device TEE and the obfuscation model parameters stored in the smart device REE.

[0019] Optionally, in one embodiment of the present application, the reasoning module includes: an acquisition unit, which is used to generate the target text task through the cloud center, and input the target text task into the text encoder of the OVD multimodal large model to obtain the text embedding feature corresponding to the target text task; a first encryption unit, which is used to encrypt the text embedding feature to generate text encryption embedding information, and send the text encryption embedding information to the target edge server, so that the target edge server sends an inference request to the smart device according to the text encryption embedding information; an execution unit, which is used to enable the smart device to use the hybrid The obfuscated image encoder is used, and an inference operation is performed in combination with the target blinding factor to obtain the image embedding feature; a second encryption unit is used to encrypt the image embedding feature to obtain the encrypted image embedding feature, and send the encrypted image embedding feature to the target edge server; a second decryption unit is used to decrypt the encrypted image embedding feature through the target edge server to obtain the decrypted image embedding feature; a second calculation unit is used to calculate the cross-modal feature similarity between the text embedding feature and the decrypted image embedding feature, and use the image embedding feature with the highest cross-modal feature similarity as the inference result of the OVD multimodal large model.

[0020] The third aspect of the present application provides an electronic device, comprising: a memory, a processor, and a computer program stored on the memory and runnable on the processor, wherein the processor executes the program to implement the method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration as described in the above embodiment.

[0021] The fourth aspect embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the program is executed by a processor, it implements the above-mentioned OVD multimodal large model parameter trusted delivery method based on end-edge-cloud collaboration.

[0022] Therefore, the embodiments of the present application have the following beneficial effects:

[0023] The embodiments of the present application can split the OVD multimodal large model through the cloud center, and cache part of the model to the edge server. The edge device performs model obfuscation and saves the obfuscated model to the TEE. When the smart device requests the model from the edge server, the edge server sends the cached model to the smart device, and the smart device loads part of the OVD model for real-time inference, so as to make full use of the resources of each layer of the end-edge cloud and realize the trusted delivery of the parameters of the OVD multimodal large model. This application is based on purified signatures, aggregated message authentication code technology with detection functions, and a trusted execution environment, which effectively realizes the secure deployment of the OVD multimodal large model under the end-edge cloud architecture, greatly improving the deployment efficiency of the model. As a result, it solves the problems that the existing technology cannot be applied to the smart security scenarios of massive smart devices and the deployment of OVD multimodal large models, and lacks an effective model asset protection mechanism.

[0024] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0026] Figure 1 This is a schematic diagram comparing the differences between traditional fixed target detection and OVD multimodal large model target detection in one embodiment of the present application;

[0027] Figure 2 This is a flowchart of a method for trusted delivery of OVD multimodal large model parameters based on device-edge-cloud collaboration according to an embodiment of the present application;

[0028] Figure 3 This is a schematic diagram of the execution logic of a method for trusted delivery of OVD multimodal large model parameters based on device-edge-cloud collaboration according to an embodiment of the present application;

[0029] Figure 4 This is a flow chart of the model caching phase according to one embodiment of the present application;

[0030] Figure 5 This is a flowchart of the smart device model request authentication phase according to one embodiment of the present application;

[0031] Figure 6 This is a flow chart of the model deployment phase of an embodiment of the present application;

[0032] Figure 7 This is a logical architecture diagram of a method for trusted delivery of OVD multimodal large model parameters based on device-edge-cloud collaboration according to an embodiment of the present application;

[0033] Figure 8 This is an example diagram of a device for trustedly issuing parameters of an OVD multimodal large model based on device-edge-cloud collaboration according to an embodiment of the present application;

[0034] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0035] Among them, 10-OVD multimodal large model parameter trusted distribution device based on end-edge-cloud collaboration; 100-obfuscation module, 200-verification module, 300-inference module; 901-memory, 902-processor, 903-communication interface. DETAILED DESCRIPTION

[0036] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0037] The following describes the trusted method for sending parameters of an OVD multimodal large model based on end-edge-cloud collaboration according to an embodiment of the present application with reference to the accompanying drawings. In response to the problems mentioned in the above background technology, the present application provides a trusted method for sending parameters of an OVD multimodal large model based on end-edge-cloud collaboration. In this method, the OVD multimodal large model is divided by the cloud center, and part of the model is cached to the edge server. The edge device performs model obfuscation and saves the obfuscated model to the TEE. When the smart device requests the model from the edge server, the edge server sends the cached model to the smart device, and the smart device loads part of the OVD model for real-time inference to make full use of the resources at each layer of the end-edge-cloud, thereby realizing the trusted sending of parameters of the OVD multimodal large model. The present application effectively realizes the secure deployment of the OVD multimodal large model under the end-edge-cloud architecture based on purified signatures, aggregated message authentication code technology with detection function, and trusted execution environment, greatly improving the deployment efficiency of the model. Thus, it solves the problems that the existing technology cannot be applied to the smart security scenarios of massive smart devices and the deployment of OVD multimodal large models, and lacks an effective model asset protection mechanism.

[0038] In order to facilitate those skilled in the art to understand the execution logic of the OVD multimodal large model parameter trusted delivery method based on end-edge-cloud collaboration of this application, the mathematical basic content and mathematical symbols involved in this application are explained below.

[0039] 1. Basic Mathematics Content

[0040] (1) Elliptic Curve Cryptography

[0041] Elliptic Curve Cryptography (ECC) is a technology that uses the mathematical properties of elliptic curves on finite fields to build public key cryptosystems. The keys it creates are mathematically difficult to crack, and compared to RSA, shorter keys can be used to achieve the same security capabilities. p The elliptic curve on is the one that satisfies the equation y 3 =(x 3 +ax+b)modp, where p is a prime number and a and b satisfy 4a 3 +27b 2 ≠0. The security of ECC relies on the following two mathematical problems:

[0042] 1) Elliptic Curve Discrete Logarithm Problem: Given a point P and an integer k on an elliptic curve, calculate Q = k·P. Based on the properties of ECC, calculating Q is easy, but k cannot be inferred from Q and P.

[0043] 2) Elliptic Curve Diffie-Hellman Problem: Given points P, k1P, and k2P on an elliptic curve, it is impossible to calculate k1k2P without knowing the values ​​of k1 and k2.

[0044] (2) Purification signature

[0045] The sanitization signature consists of a signer, a sanitizer, and a verifier. Compared to traditional signature methods, the sanitizer is a newly introduced role that can modify the message signed by the signer to hide sensitive information in the message without invalidating the original signature of the message, ensuring that the source of the modified message is credible. The verifier cannot determine whether the message has been modified by the sanitizer and where the modification was made. The sanitization signature consists of a series of algorithms {KGen, Sign, Sanit, Verify, Proof, Judge}. The following introduces the four algorithms {KGen, Sign, Sanit, Verify} respectively:

[0046] 1) KGen: used to generate the public and private keys of the signer and purifier: (pk sig ,sk sig )←KGen sig (1 λ ) and (pk san ,sk san )←KGen san (1 λ );

[0047] 2)Sign: Input message m∈{0,1} * , the signer's private key sksig , the purifier's public key pk san And the modifiable message segment identifier ADM, the signature algorithm output σ←Sign(m,sk sig ,pk san ,ADM);

[0048] 3) Sanit: Input message m, signature σ, signer public key pk sig , purifier private key sk san And the modifiable message segment identifier ADM, by executing (m * ,σ * )←Sanit(m,ADM,σ,pk sig ,sk san ) Complete the purification and output the message m after hiding the sensitive information * and the corresponding signature σ * ;

[0049] 4) Verify: Input message m, signature σ, signer public key pk sig , purifier public key pk san , by executing the algorithm b←Verify(m,σ,pk sig ,pk san ) Complete the verification. When b=1, it indicates that the verification is successful, otherwise, the verification fails.

[0050] (3) Aggregate Message Authentication Code with Detection Function

[0051] The Aggregate Message Authentication Code (AMAD) with detection function can authenticate multiple devices simultaneously and accurately locate the illegal user identity corresponding to the wrong MAC, which can effectively resist DoS attacks.

[0052] AMAD is generated by the MAC function And biorthogonal code is constructed, here we can use AMAD construction algorithm II, which consists of polynomial time algorithm (KGen, Tag, Agg, TVrfy). Let S be the biorthogonal code generator matrix of order l, (n, k, d muin )=(2 l ,l+1,2 l-1 ), where l≥3, k, n, l are integers, d min is the minimum distance; let Σ be the syndrome generator matrix of the extended matrix, ε be the syndrome, ψ be the extended syndrome; for each k=1,2,…,l+1, let S k =(S k,1 ,S k,2 ,..,S k,n )∈{0,1} n is the k-th row value of S; define X k =(Xk,1 ,X k,2 ,X k,3 ,…,X k,n )=(S k,1 ,αS k,2 ,α 2 X k,3 ,..,α n- 1 S k,n ), where α∈GF(2 h ), define X as a (l+1)×n matrix whose kth row is X k , let Γ be (2 l+1 +1)×n matrix, whose row elements consist of all non-zero code words in the row of matrix X. Its construction process is as follows:

[0053] 1) KGen: Send the stored key, K id ←KGen(1 λ ,id). This key is the pre-shared key between the cloud center and the smart device;

[0054] 2) Tag: K id and message m as input, and use MAC function M to calculate t=M(K id ,m), that is, t←Tag(K id ,m);

[0055] 3) Agg: Send all messages sent by smart devices (ID1, m1, t1), ..., (ID n ,m n ,t n ) as input, output aggregate message authentication code T; for t=(t1,t2,…,t n ), calculate T1=(T 1,1 ,T 1,2 ,…,T 1,l+1 )=tS T ; For the j∈[n]th device, extract t j The last hbit is After converging Calculate T2 = (T 2,1 ,T 2,2 ,…,T 2,l+1 )=t * X T ; Message authentication code T = (T1, T2) after executing the Agg algorithm;

[0056] 4) TVrfy: When the receiver receives the message Then, for each j∈[n] device sender, calculate ), we get t′=(t′1,t′2,…,t′n ). Then calculate ε = Tt'S T ; If ε = 0, it indicates that all device identity information is legal; otherwise, it is considered that malicious or illegal identity information exists; by performing the following steps, the illegal device identity number is locked.

[0057] a) The device ID number set is D = {1, 2, ..., n}, L = {1, 2, ..., 2 l+1 -1}.

[0058] b) Calculate the extended syndrome by using the syndrome ε and the extended check generator matrix Σ Where e is the error vector, satisfying ε=eS T ;

[0059] c) For k = 1, 2, ..., 2 l+1 -1, if ψ k =0, set and L←L\{k}, where is an integer, and Σ in the kth row satisfies

[0060] d) For each device j∈[n], extract t j The following hbit is After aggregation, we can get Calculate g=(g1,g2,…,g l+1 )=T2-t * X T =e′X T , calculated using g and Γ

[0061] e) For k = 1, 2, ..., 2 l+1 -1, extract ψ k ∈GF(2 h ) is ψ′ k , traverse k∈L,j∈D, calculate J={j|Σ k,j α j-1 ψ′ k =ξ k};

[0062] f) Output illegal device ID list D′={ID j},j∈J.

[0063] 2. Mathematical symbols

[0064] The mathematical symbols used in this application are shown in Table 1:

[0065] Table 1

[0066]

[0067] like Figure 2 As shown, the method for trusted delivery of OVD multimodal large model parameters based on device-edge-cloud collaboration includes the following steps:

[0068] In step S201, the image encoder of the OVD multimodal large model is signed by the preset cloud center, and the signed image encoder is cached to the target edge server. The target edge server generates a blinding factor to use the blinding factor to perform model obfuscation on the image encoder of the OVD multimodal large model to obtain the obfuscated image encoder model parameters. At the same time, the blinding factor and the obfuscated image encoder model parameters are cached to the TEE and REE in the target edge server respectively.

[0069] It should be noted that the embodiment of the present application mainly includes five execution stages in the specific implementation process: initialization and disclosure of parameter information stage, model purification cache stage, smart device model request authentication stage, model deployment stage and model inference stage. Its execution logic is as follows: Figure 3 shown.

[0070] Among them, in the embodiment of the present application, during the model purification and caching stage, the OVD multimodal large model can be signed by the cloud center and sent to the edge server. The edge server generates a blinding factor to perform model obfuscation on the image encoder of the OVD multimodal large model through the blinding factor, and caches the obfuscated model network parameters in the TEE and REE in the edge server respectively.

[0071] Therefore, the embodiment of the present application effectively ensures the credibility of the source of the OVD multimodal large model parameters through the purification signature mechanism and trusted execution environment, while also achieving privacy protection of the model parameters and avoiding the security risks of model tampering.

[0072] Optionally, in one embodiment of the present application, before signing the image encoder of the OVD multimodal large model through the preset cloud center and caching the signed image encoder to the target edge server, it also includes: registering with the preset key management center through the cloud center, the target edge server and the smart device respectively to generate the public key and private key corresponding to the cloud center and the target edge server and the pre-made shared secret value of the smart device; based on the public key and the private key, establishing a session handshake between the cloud center and the target edge server to complete the cloud center's identity authentication of the target edge server; using the cloud center to perform a partition operation on the OVD multimodal large model to deploy the text encoder of the OVD multimodal large model in the cloud center, and deploy the image encoder to the smart device.

[0073] It should be noted that in the initialization and public parameter information phase before the model purification cache phase, the embodiment of the present application first needs to register with the Key Management Center (KGC) through the cloud center, edge server and smart device respectively to generate the public and private keys of the cloud center and edge server and the pre-made shared secret value of the smart device; secondly, the cloud center establishes a session handshake with the edge server to complete the cloud center CS to the edge server ES i Finally, the cloud center splits the OVD multimodal large model, deploys the text encoder of the OVD multimodal large model to the local cloud center, and deploys the image encoder of the OVD multimodal large model to the smart device. The specific process is as follows:

[0074] 1. The cloud center, edge server, and smart device register with the key management center respectively:

[0075] (1) KGC randomly selects a As the system private key, calculate the system public key P pub =sP, and choose the following one-way hash function:

[0076] (2) The cloud center CS sends the real identity ID through a secure channel CS After receiving the information, KGC randomly selects a Compute the private key Sk of the cloud center CS =x CS +h1(ID CS ) s and public key Pk CS =Sk CS ·P, and {ID CS ,Pk CS ,Sk CS}Send to cloud center;

[0077] (3) Edge Server ES i Send its real identity to KGC through a secure channel After receiving the information, KGC randomly selects Calculating ES i Private key and public key At the same time, the shared key is calculated Used to decrypt model parameters. Store it in the KGC database, and then Sent to the edge server ES through a secure channel i , Edge Server ES i The public key Place the private key in REE and symmetric keys Placed in TEE;

[0078] (4) Smart device ED j Send its real identity to KGC through a secure channel After KGC receives the information, it calculates Through a secure channel Send to smart device j , ED j Will Store in TEE;

[0079] (5) KGC will and The message is sent to the cloud center CS through a secure channel. After receiving the message, the cloud center stores it in the database. Then, the file hash summary value F of the cloud center enclave initialization program is Enc Download to smart device j In the TEE, it is used to initialize and start the TEE;

[0080] (6) The cloud center establishes a session handshake with the edge server and completes the cloud center CS to the edge server ES through remote proof. i Identity authentication;

[0081] (7) The OVD multimodal large model is composed of a text encoder and an image encoder. The text encoder is composed of an OVD multimodal large model such as CLIP and BERT, and the image encoder is composed of Transformer and Faster-RCNN. The OVD multimodal large model is partitioned according to its function, and the text encoding module is transferred to the cloud center, and the image encoding module is transferred to the smart device.

[0082] Therefore, the embodiment of the present application is based on the OVD multimodal large model deployment framework with end-edge-cloud collaboration, taking into account the characteristics of the large number of parameters of the OVD multimodal large model and the limited computing power resources of the device. The signed OVD multimodal image encoder is cached to the edge server through the cloud center, and the device private key and secret parameters are saved in a trusted execution environment. Therefore, compared with the inference method in which smart devices request the cloud center, this framework takes into account the trustworthiness and efficient distribution of model parameters, which can effectively reduce the bandwidth overhead and request load of the cloud center and reduce the latency of model inference.

[0083] Optionally, in one embodiment of the present application, the image encoder of the OVD multimodal large model is signed by a preset cloud center, and the signed image encoder is cached to a target edge server, and the target edge server generates a blinding factor to use the blinding factor to perform model obfuscation on the image encoder to obtain obfuscated image encoder model parameters, including: obtaining a target network layer in the image encoder that meets the preset importance requirements through the cloud center, and constructing a corresponding network layer index set based on the target network layer; randomly selecting random values ​​corresponding to the target edge server and the cloud center respectively, and signing the network layer parameters in the image encoder based on the network layer index set and the random value, and encrypting the network layer parameters to obtain encrypted network layer parameters, and caching the signed image encoder model parameters to the target edge server; decrypting the encrypted network layer parameters using a prefabricated shared secret value through the target edge server, and verifying the validity of the signature of the network layer parameters according to a preset signature validity condition; when the signature of the network layer parameters meets the preset signature validity condition, generating a blinding factor in the TEE of the edge server to obfuscate the network layer parameters using the blinding factor to obtain obfuscated image encoder model parameters.

[0084] During the actual execution process, in the model purification and caching stage, the embodiment of the present application can first sign the image encoder through the cloud center and encrypt it and send it to the edge server; secondly, the edge server decrypts and signs the signed model parameters in the TEE; then, the model parameters are purified and obfuscated using the obfuscation factor; finally, the obfuscated model parameters and obfuscation factors are saved in the REE and TEE memory respectively.

[0085] In the embodiment of the present application, the image encoder is assumed to be Net={m1,m2,…m L}, there are L layers of convolutional neural network, m l (l∈[1,L]) represents the parameters of the jth layer network. The network weight parameters within the top K range are considered important parameters, otherwise they are unimportant parameters. Let the index set φ={0,1} L Indicates F net Whether the network parameters of each layer of the image encoder contain important parameters, "1" means that the layer has important parameters, and "0" means that the layer parameters do not have important parameters.

[0086] like Figure 4 As shown, the specific execution steps of the embodiment of the present application in the model purification cache stage are as follows:

[0087] 1. The Cloud Center CS performs the following steps:

[0088] (1) Determine whether each layer of the image encoder Net contains important parameters to generate a network layer index set φ;

[0089] (2) For each edge server ES i Pick a random value And calculate R i =r i ·P;

[0090] (3) Let the weight parameter of the lth convolutional layer be m l , l∈[1,L], the cloud center first randomly selects Calculate D l =d l ·P,U=h3(ID CS ||φ||R i )as well as Afterwards, the network layer parameter m l Signature σ l =r i U+d l ·S l ;Finally, the network layer parameters are encrypted using the pre-made key

[0091] (4) For the index that meets φ[l]=1, calculate Afterwards, the index set φ is calculated

[0092] (5) The cloud center sends the message Send to different edge servers ES respectively i ,i∈[1,m].

[0093] 2. Edge Server ES i After receiving the message from the cloud center, the following operations will be performed:

[0094] (1) Check the timestamp If the sexual freshness is satisfied, continue, otherwise, terminate;

[0095] (2) Edge Cloud Center ES i Utilizing a shared key Decrypt model weight parameters

[0096] (3) Calculation U′=h 10 (ID CS ||R i ),

[0097] (4) Batch verify the signature σ of all network layer weight parameters l ,l∈[1,L], using the following formula:

[0098]

[0099] If the above formulas are equal, it indicates that the signature of the cloud center CS for the image encoder Net network layer weight parameters is valid. Otherwise, the signature is considered invalid and the model parameters are modified.

[0100] (5) Calculation Generate blinding factors in TEE Among them, |BF| is equal to the number of elements in the index set φ′ that are 1, which is used to purify the model parameters and purify the important parameters in each layer of the network to obtain the purified model parameters Net′;

[0101] (6) Using the purified signature, the modified network layer parameter signature is converted as follows:

[0102]

[0103] (7) The calculated blinding factor BF is stored in TEE, and the purified network layer parameters Net′={m′1,m′2,…,m′ n} is stored in REE storage memory.

[0104] Therefore, the embodiment of the present application uses a purification signature mechanism based on the edge-cloud collaborative model caching mechanism to ensure the source credibility and privacy security of the model parameters. Compared with the traditional digital signature mechanism, the purification signature in the embodiment of the present application can securely obfuscate the model parameters by generating a blinding factor without destroying the signature invalidation of the model parameters by the cloud center, effectively taking into account the credibility of the source of the model parameters and the privacy security.

[0105] In step S202, the cloud center is used to perform batch identity authentication on the preset smart devices to obtain the smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and at the same time, the obfuscated image encoder model parameters and the encrypted blinding factor are verified for credibility and integrity.

[0106] Furthermore, in the embodiment of the present application, during the smart device model request authentication stage, the cloud center can batch authenticate the identity of the smart device through the edge server, and establish a session key between the cloud center and the edge server, as well as a group session key between the edge server and the smart device; then, during the model deployment stage, after the smart device identity authentication is successful, the blinding factor is first encrypted with the group session key and then sent to the smart device and stored in the TEE; secondly, the obfuscated model parameters are sent to the smart device and stored in the REE; finally, the smart device verifies that the model source is credible and the parameters are complete.

[0107] Understandably, when counterfeit smart devices send model requests to edge servers to steal model assets, while identity authentication can effectively resist counterfeit smart device attacks, existing smart device authentication schemes based on bilinear pairings incur high response delays and computational overhead, making it impossible to quickly authenticate the identities of massive smart devices in real time, making it difficult to meet the real-time requirements of smart security. Therefore, the embodiments of the present application use an Aggregate Message Authentication Code (AMAD) with detection capabilities to complete the authentication of batches of smart devices, thereby effectively resisting identity counterfeiting attacks.

[0108] Optionally, in one embodiment of the present application, a cloud center is used to perform batch authentication on preset smart devices to obtain smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and at the same time verifies the credibility and integrity of the obfuscated image encoder model parameters and the encrypted blinding factor, including: sending a model request response to the target edge server through the smart device, so that the target edge server sends the aggregated model request response to the cloud center after aggregating the model request response; using the cloud center to generate a challenge value based on the aggregated model request response, and sending the challenge value to the smart device through the target edge server, so that the smart device uses the challenge value to calculate the initialization program measurement value of the smart device TEE to generate a response value, and sending the response value to the cloud center; batch authenticating the identity of the smart device through the cloud center to obtain the smart device identity authentication result, and the identity of the smart device The authentication result is sent to the target edge server, so that the target edge server builds a group session key with the smart device, signs the group session key, and sends the signed target message to the smart device; based on the target message, the smart device calculates the group session key with the target edge server, authenticates the target edge server, and hashes the group session key to obtain a hash result, and returns the hash result to the authenticated target edge server; based on the hash result, the target edge server authenticates the smart device, encrypts the blinding factor using the group session key, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the authenticated smart device; the smart device is used to load the encrypted blinding factor into the smart device TEE, decrypts the encrypted blinding factor using the group session key in the smart device TEE to obtain the target blinding factor, and saves the obfuscation model parameters to the smart device REE; the credibility and integrity of the target blinding factor stored in the smart device TEE and the obfuscation model parameters stored in the smart device REE are verified.

[0109] During the smart device model request authentication stage, the embodiment of the present application can first send a model request response to the edge server through the smart device, and the edge server aggregates the request responses and sends them to the cloud center; secondly, the cloud center generates a challenge value, which is sent to the smart device through the edge server; thirdly, the smart device uses the challenge value to calculate the TEE initialization program measurement value to generate a response value, and aggregates it through the edge server and sends it to the cloud center; thereafter, the cloud center batch authenticates the identity of the smart devices and calculates the session key between the edge server, and sends the authentication result to the edge server; further, the edge server constructs a group session key with the smart device, and signs the authentication result of the cloud center and sends it to the smart device; finally, the smart device authenticates the identity of the edge server, calculates the session key, and sends a session key reception confirmation message to the edge server to complete the cloud center's batch authentication of the smart devices and the mutual authentication of the smart devices and the edge server.

[0110] like Figure 5 As shown, the specific execution steps of the smart device model authentication phase in the embodiment of the present application are as follows:

[0111] 1. Smart device ED j Generate a model deployment request and send it to the edge server. The specific steps are as follows:

[0112] (1) Generate model deployment request tag j , calculate the message request authentication code

[0113] (2) Sending a message To the edge server ES to which it belongs i ;

[0114] 2. Edge Server ES i Aggregate model deployment requests and send them to the cloud center. The specific steps are as follows:

[0115] (1) Collect message requests from n smart devices within a fixed time;

[0116] (2) Check the timestamp of the smart message if Then continue, otherwise terminate;

[0117] (3) Calculate the MAC value of the aggregated message,

[0118] (4) Sending a message to the cloud center.

[0119] 3. The cloud center generates challenges and sends them to edge servers in batches:

[0120] (1) Check the timestamp of the message if If satisfied, continue, otherwise terminate;

[0121] (2) Calculate based on the received converged message Determine MAC′ i Is it equal to MAC? i ; If they are not equal, terminate, otherwise, continue;

[0122] (3) For ES belonging to edge server i Each smart device ED j , the cloud center searches the database After the information, calculate in, and Tag′ j Obtain from the received message; determine MAC′ in sequence j Is it equal to MAC? j ,If they are equal, it indicates that the model request of the ,smart device is a legal request, and the request continues, otherwise it terminates;

[0123] (4) For each smart device Generate a challenge value nonce j , choose a random number Calculate Y j =y j · and as well as

[0124] (5) Send the message Sent to the edge server ES i .

[0125] 4. Edge Server ES i Forward the challenge value to each smart device:

[0126] (1) Check the message timestamp if Then continue, otherwise terminate;

[0127] (2) Calculation as well as If AH′1 is equal to AH1, it indicates that the challenge message has not been tampered with, and the process continues; otherwise, the process terminates.

[0128] (3) Smart devices calculate And construct the message

[0129] (4) Send the message Send to different smart devices where j∈[1,n].

[0130] 5. The smart device calculates the TEE metric and sends it to the edge server:

[0131] (1) Check the message timestamp if Then continue, otherwise terminate;

[0132] (2) Calculation and calculate the measure

[0133] (3) Select a random number and calculate calculate

[0134] (4) Calculation calculate

[0135] (5) Send the message Sent to the edge server ES i .

[0136] 6. The edge server uses AMAD to batch aggregate response messages and send them to the cloud center.

[0137] (1) After gathering the messages sent by all smart devices, check the timestamps of the messages sent by the smart devices Freshness; if If it is fresh, continue; otherwise, terminate.

[0138] (2) Extraction Save to database;

[0139] (3) ED for smart devices j ,calculate Will Save to database;

[0140] (4) Calculate the aggregated message authentication code T = tS T , where t=(t1,t2,…,t n );

[0141] (5) Random selection And calculate V i =v i P, calculation

[0142] (6) Sending a message to the cloud center.

[0143] 7. The cloud center batch verifies the identity of smart devices and negotiates session keys:

[0144] (1) Check the message timestamp if Then continue, otherwise terminate;

[0145] (2) Calculate based on the received converged message

[0146] (3)Judgment Is it equal to If they are not equal, terminate, otherwise continue;

[0147] (4) Calculation We get t′=(t′1,t′2,…,t′ n ); by calculating ∈=Tt′S T Complete batch identity authentication of smart devices. If ∈ = 0, it indicates that all smart identities that initiate batch authentication of the model are legal; otherwise, it is considered that there are malicious or illegal smart devices connected, and the illegal device search algorithm in TVrfy is executed to output the illegal device ID list D′ i , the list contains invalid or illegal smart device identity IDs;

[0148] (5) Computing and Edge Server ES i Calculate the session key

[0149] (6) Calculate the message hash value

[0150] (7) Sending a message To the edge server ES i , i∈[m].

[0151] 8. The edge server constructs the group session key and sends it to the smart device:

[0152] (1) Check the message timestamp if Then continue, otherwise terminate;

[0153] (2) Calculate the session key with the cloud center

[0154] (3) Verify the message hash value if This indicates that the session key with the cloud center is established correctly and the device ID list has not been tampered with;

[0155] (4) According to the Chinese remainder theorem, calculate the group session key with the smart device; first, D′i γ corresponding to the smart devices involved k Delete, where k∈D′ i , then calculate γ=γ1·γ2·…·γ n , β j ×u j ≡1modγ j , where var j =β j ×u j ; Random selection The group session key is calculate

[0156] (5) Random selection And calculate W j =w j P, signs the group session key,

[0157] (6) Send the message Broadcast to each smart device requested by the model.

[0158] 9. The smart device authenticates the edge server and calculates the session key:

[0159] (1) Check the message timestamp if Then continue, otherwise terminate;

[0160] (2) According to Gsk i Calculate the group session key

[0161] (3) Verify the edge server ES i The signed message: If the signature is established, the smart device successfully authenticates the edge server and successfully establishes the group session key;

[0162] (4) Calculate the hash value returned by the session,

[0163] (5) Send key confirmation message to the edge server.

[0164] 10. Edge server authenticates smart device identity:

[0165] (1) Check the message timestamp if Then continue, otherwise terminate;

[0166] (2) Calculation if Indicates edge server ES i Certified smart device ED j Success, indicating that the smart device has calculated the correct group session key.

[0167] During the model deployment phase, upon receiving the smart device key confirmation message, the embodiment of the present application may first send the obfuscation factor and purified network layer parameters to the smart device through the edge server; secondly, after the smart device receives it, the encrypted obfuscation factor is loaded into the TEE and decrypted using the group session key, while the network layer parameters are saved into the REE; finally, the obfuscation factor in the TEE and the purified model parameters in the REE are verified to determine whether the sources of the model parameters and obfuscation factor are credible and complete, thereby completing the deployment of the model parameters.

[0168] The model deployment phase verifies the secure deployment of the confounding factors, the cleansed model parameters, and the credibility of the sources of the confounding factors, e.g. Figure 6 The specific execution steps are as follows:

[0169] 1. The edge server sends network parameters and confusion factors to the smart device.

[0170] (1) Encrypt the blinding factor BF using the group session key Gk E =Enc Gk (BF′);

[0171] (2) Calculation

[0172] (3) Sending a message To the smart device, where {R i ,{D l ,σ′ l} l∈[1,L]} is the relevant parameter information after the signature is purified in the model purification cache stage.

[0173] 2. The smart device stores and verifies network parameters and obfuscation factors:

[0174] (1) Check the message timestamp if Then continue, otherwise terminate;

[0175] (2) Encrypt the blinding factor BF E Put it into TEE and put the purified network parameters Net′ into REE;

[0176] (3) Decrypt the blinding factor BF′=Dec using the session key in TEEGk′ (BF E ) for calculation for comparison to check if it is equal to If they are equal, continue; otherwise, terminate.

[0177] (4) Randomly select w (w < L) elements from the numbers [1, L] of the image encoder network layer to form a network layer index set I. For each element c in the set where c ∈ [1, L], select a random value

[0178] (5) Calculate the linear combination for signature verification of the network layer weight parameters corresponding to the network layer index set I U′ = h 10 (ID CS ||R i ) to verify the signature validity of the index set I. Its mathematical expression is as follows:

[0179]

[0180] If the above equation holds, it indicates that the model parameters have not been tampered with, proving that their source is trustworthy.

[0181] Thus, based on the lightweight batch intelligent device access authentication method in the embodiments of this application, the AMAD function is used to achieve batch aggregation of authentication messages, and the cloud center can quickly batch authenticate a large number of devices, avoiding the model being sent to smart devices forged by adversaries; at the same time, smart devices complete the authentication of edge servers, effectively resisting edge server impersonation attacks, preventing malicious models from being deployed to smart devices, and avoiding serious security incidents; in addition, in terms of efficiency, compared with the one-to-one authentication mechanism, the embodiments of this application can perform one-to-many batch identity authentication of intelligent devices, effectively improving the efficiency of the authentication and model parameter distribution processes, avoiding the use of bilinear calculation operations, reducing the calculation and communication overhead of the smart device authentication process, and at the same time strongly resisting various protocol attacks, with excellent security performance.

[0182] In step S203, when the obfuscated image encoder model parameters and the encrypted blinding factor pass the credibility and integrity verification, the target text task is generated through the cloud center, and the text embedding features of the target text task are obtained using the text encoder of the OVD multimodal large model, and the text embedding features are sent to the target edge server. The target edge server is used to send an inference request to the smart device to perform model inference on the image encoder through the smart device to obtain the image embedding features. At the same time, the image embedding features and the text embedding features are compared using the preset cross-modal feature comparison strategy to obtain the target image embedding features that meet the preset similarity requirements, so that the target image embedding features are used as the inference results of the target text task.

[0183] Furthermore, in the model reasoning stage, the embodiment of the present application can generate text tasks through the cloud center and output text embedding features through the large language model, such as Figure 7 As shown, it is encrypted and sent to the edge server. The edge server loads the text embedding features into TEE and sends an inference request to the smart device, thereby executing the image encoder through the smart device.

[0184] Therefore, the embodiments of the present application can effectively protect the intellectual property rights of model parameters, realize the trusted distribution of model parameters, solve the scalability problem of model distribution, and meet the real-time requirements of smart scenarios.

[0185] Optionally, in one embodiment of the present application, a target text task is generated through a cloud center, and a text encoder of an OVD multimodal large model is used to obtain text embedding features of the target text task, and the text embedding features are sent to a target edge server, and an inference request is sent to an intelligent device using the target edge server to perform model inference on the image encoder through the intelligent device to obtain image embedding features, and at the same time, a preset cross-modal feature comparison strategy is used to compare the image embedding features and the text embedding features to obtain target image embedding features that meet preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task, including: generating a target text task through a cloud center, and inputting the target text task into the text encoder of an OVD multimodal large model to obtain the text embedding features corresponding to the target text task. The invention discloses an OVD multimodal large model that is based on the OVD multimodal large model. ...

[0186] Specifically, the process of performing device-edge-cloud collaborative reasoning in the model reasoning stage in the embodiment of the present application is as follows:

[0187] 1. The cloud center generates a text task, inputs the text encoder to obtain text embedding features, encrypts it with the session key, obtains the encrypted text embedding information, and sends it to the edge server;

[0188] 2. After receiving the encrypted embedded text information, the edge server sends image inference instructions to each smart device;

[0189] 3. After receiving the inference instruction, the smart device performs the image encoder inference by combining the purified model with the confusion factor in the TEE, and sends the inference result to the edge server using the group session key.

[0190] 4. The edge server performs cross-modal embedding feature calculations.

[0191] To sum up, the embodiment of the present application innovatively utilizes the end-edge cloud architecture and the purification signature mechanism to design an OVD multimodal large model caching and deployment method, which effectively ensures the source credibility and privacy security of the OVD multimodal large model parameters; at the same time, the embodiment of the present application utilizes an aggregated message authentication code with a detection function to realize one-to-many batch smart device identity authentication, reducing the computational and communication overhead of the authentication process and improving the efficiency of the model parameter distribution process.

[0192] According to the method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration proposed in the embodiment of this application, the OVD multimodal large model is divided by the cloud center, and part of the model is cached to the edge server. The edge device performs model obfuscation and saves the obfuscated model to the TEE. When the smart device requests the model from the edge server, the edge server sends the cached model to the smart device, and the smart device loads part of the OVD model for real-time inference, so as to make full use of the resources of each layer of the end-edge-cloud and realize the trusted delivery of OVD multimodal large model parameters. This application effectively realizes the secure deployment of the OVD multimodal large model in the end-edge-cloud architecture based on purified signatures, aggregated message authentication code technology with detection function, and trusted execution environment, greatly improving the deployment efficiency of the model.

[0193] Secondly, refer to the attached Figure 8 Describe the trusted sending device for OVD multimodal large model parameters based on end-edge-cloud collaboration proposed in an embodiment of the present application.

[0194] Figure 8 It is a block diagram of an OVD multimodal large model parameter trusted delivery device based on end-edge-cloud collaboration in an embodiment of the present application.

[0195] like Figure 8 As shown, the OVD multimodal large model parameter trusted delivery device 10 based on end-edge-cloud collaboration includes: an obfuscation module 100, a verification module 200 and an inference module 300.

[0196] Among them, the obfuscation module 100 is used to sign the image encoder of the OVD multimodal large model through a preset cloud center, and cache the signed image encoder to the target edge server, and the target edge server generates a blinding factor to use the blinding factor to perform model obfuscation on the image encoder of the OVD multimodal large model, obtain the obfuscated image encoder model parameters, and cache the blinding factor and the obfuscated image encoder model parameters to the TEE and REE in the target edge server respectively.

[0197] The verification module 200 is used to use the cloud center to perform batch authentication on preset smart devices to obtain smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and at the same time verifies the credibility and integrity of the obfuscated image encoder model parameters and the encrypted blinding factor.

[0198] The inference module 300 is used to generate a target text task through the cloud center when the obfuscated image encoder model parameters and the encrypted blinding factor pass the credibility and integrity verification, and use the text encoder of the OVD multimodal large model to obtain the text embedding features of the target text task, and send the text embedding features to the target edge server, and use the target edge server to send the inference request to the smart device to perform model inference on the image encoder through the smart device to obtain the image embedding features, and at the same time use the preset cross-modal feature comparison strategy to compare the image embedding features and the text embedding features to obtain the target image embedding features that meet the preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task.

[0199] Optionally, in one embodiment of the present application, the OVD multimodal large model parameter trusted delivery device 10 based on end-edge-cloud collaboration in the embodiment of the present application further includes: a registration module, a handshake module and a partitioning module.

[0200] Among them, the registration module is used to sign the image encoder of the OVD multimodal large model through the preset cloud center, and register with the preset key management center through the cloud center, the target edge server and the smart device before caching the signed image encoder to the target edge server, so as to generate the public key and private key corresponding to the cloud center and the target edge server and the prefabricated shared secret value of the smart device.

[0201] The handshake module is used to establish a session handshake between the cloud center and the target edge server based on the public key and private key to complete the identity authentication of the cloud center to the target edge server.

[0202] The partitioning module is used to perform partitioning operations on the OVD multimodal large model using the cloud center, so as to deploy the text encoder of the OVD multimodal large model in the cloud center and deploy the image encoder to the smart device.

[0203] Optionally, in one embodiment of the present application, the obfuscation module 100 includes: a construction unit, a first signature unit, a first decryption unit and a generation unit.

[0204] Among them, the construction unit is used to obtain the target network layer that meets the preset importance requirements in the image encoder through the cloud center, and construct a corresponding network layer index set based on the target network layer.

[0205] The first signature unit is used to randomly select random values ​​corresponding to the target edge server and the cloud center respectively, sign the network layer parameters in the image encoder based on the network layer index set and the random value, and encrypt the network layer parameters to obtain encrypted network layer parameters, and cache the signed image encoder model parameters to the target edge server.

[0206] The first decryption unit is used to decrypt the encrypted network layer parameters using a pre-made shared secret value through a target edge server, and verify the validity of the signature of the network layer parameters according to a preset signature validity condition.

[0207] A generation unit is configured to generate a blinding factor in the TEE of the edge server when the signature of the network layer parameter meets a preset signature validity condition, so as to confuse the network layer parameter by the blinding factor to obtain the obfuscated image encoder model parameter.

[0208] Optionally, in one embodiment of the present application, the verification module 200 includes: a convergence unit, a first calculation unit, a second signature unit, a hash unit, an authentication unit, a loading unit and an analysis unit.

[0209] Among them, the aggregation unit is used to send the model request response to the target edge server through the smart device, so that the target edge server sends the aggregated model request response to the cloud center after aggregating the model request response.

[0210] The first computing unit is used to use the cloud center to generate a challenge value based on the aggregated model request response, and send the challenge value to the smart device through the target edge server, so that the smart device uses the challenge value to calculate the initialization program measurement value of the smart device TEE to generate a response value, and send the response value to the cloud center.

[0211] The second signature unit is used to batch authenticate the identities of smart devices through the cloud center to obtain smart device identity authentication results, and send the smart device identity authentication results to the target edge server, so that the target edge server builds a group session key between itself and the smart device, and signs the group session key at the same time, and sends the signed target message to the smart device.

[0212] The hash unit is used to calculate the group session key with the target edge server through the smart device based on the target message, authenticate the target edge server, hash the group session key to obtain a hash result, and return the hash result to the authenticated target edge server.

[0213] The authentication unit is used to enable the target edge server to authenticate the smart device based on the hash processing result, encrypt the blinding factor using the group session key, and send the obfuscated image encoder model parameters and the encrypted blinding factor to the authenticated smart device.

[0214] The loading unit is used to load the encrypted blinding factor into the TEE of the smart device using the smart device, decrypt the encrypted blinding factor in the TEE of the smart device using the group session key to obtain the target blinding factor, and save the obfuscation model parameters to the REE of the smart device.

[0215] The analysis unit is used to verify the credibility and integrity of the target blinding factor stored in the TEE of the smart device and the obfuscation model parameters stored in the REE of the smart device.

[0216] Optionally, in one embodiment of the present application, the reasoning module 300 includes: an acquisition unit, a first encryption unit, an execution unit, a second encryption unit, a second decryption unit, and a second calculation unit.

[0217] Among them, the acquisition unit is used to generate a target text task through the cloud center and input the target text task into the text encoder of the OVD multimodal large model to obtain the text embedding features corresponding to the target text task.

[0218] The first encryption unit is used to encrypt the text embedding feature to generate text encryption embedding information, and send the text encryption embedding information to the target edge server, so that the target edge server sends an inference request to the smart device according to the text encryption embedding information.

[0219] The execution unit is configured to enable the intelligent device to perform an inference operation using the obfuscated image encoder in combination with a target blinding factor based on an inference request to obtain an image embedding feature.

[0220] The second encryption unit is used to encrypt the image embedding feature to obtain the image embedding encryption feature, and send the image embedding encryption feature to the target edge server.

[0221] The second decryption unit is configured to decrypt the image embedded encryption feature through the target edge server to obtain the image embedded decryption feature.

[0222] The second calculation unit is used to calculate the cross-modal feature similarity between the text embedding feature and the image embedding decryption feature, and use the image embedding decryption feature with the highest cross-modal feature similarity as the inference result of the OVD multimodal large model.

[0223] It should be noted that the above explanation of the embodiment of the method for trusted issuance of OVD multimodal large model parameters based on end-edge-cloud collaboration is also applicable to the device for trusted issuance of OVD multimodal large model parameters based on end-edge-cloud collaboration in this embodiment, and will not be repeated here.

[0224] According to the device for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration proposed in the embodiment of the present application, the OVD multimodal large model is divided by the cloud center, and part of the model is cached to the edge server. The edge device performs model obfuscation and saves the obfuscated model to the TEE. When the smart device requests the model from the edge server, the edge server delivers the cached model to the smart device, and the smart device loads part of the OVD model for real-time inference, so as to make full use of the resources at all layers of the end-edge-cloud and realize the trusted delivery of OVD multimodal large model parameters. This application effectively realizes the secure deployment of the OVD multimodal large model in the end-edge-cloud architecture based on purified signatures, aggregated message authentication code technology with detection functions, and a trusted execution environment, greatly improving the deployment efficiency of the model.

[0225] Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device may include:

[0226] A memory 901 , a processor 902 , and a computer program stored in the memory 901 and executable on the processor 902 .

[0227] When the processor 902 executes the program, the OVD multimodal large model parameter trusted delivery method based on end-edge-cloud collaboration provided in the above embodiment is implemented.

[0228] Furthermore, the electronic device further includes:

[0229] The communication interface 903 is used for communication between the memory 901 and the processor 902 .

[0230] The memory 901 is used to store computer programs that can be run on the processor 902 .

[0231] The memory 901 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0232] If the memory 901, processor 902, and communication interface 903 are implemented independently, the communication interface 903, memory 901, and processor 902 can be connected to each other via a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 9 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0233] Optionally, in a specific implementation, if the memory 901, the processor 902 and the communication interface 903 are integrated on a chip, the memory 901, the processor 902 and the communication interface 903 can communicate with each other through an internal interface.

[0234] The processor 902 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.

[0235] An embodiment of the present application also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, it implements the above-mentioned method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration.

[0236] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or N embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in this specification and features of different embodiments or examples without contradiction.

[0237] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be understood to indicate or imply relative importance or implicitly specify the number of technical features indicated. Thus, a feature specified as "first" or "second" may explicitly or implicitly include at least one such feature. In the description of this application, "N" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0238] Any process or method description in a flowchart or otherwise described herein may be understood to represent a module, fragment or portion of code comprising one or N executable instructions for implementing a custom logical function or process step, and the scope of the preferred embodiments of the present application includes alternative implementations in which functions may be performed in a different order than shown or discussed, including performing functions in a substantially simultaneous manner or in a reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present application pertain.

[0239] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or N wires (electronic devices), a portable computer disk cartridge (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program can be obtained electronically by optically scanning the paper or other medium and then editing, interpreting or processing it in other suitable ways as necessary, and then storing it in a computer memory.

[0240] It should be understood that various parts of the present application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiment, the N steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. If implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0241] Those skilled in the art will understand that all or part of the steps in the method of the above embodiment can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium. When the program is executed, it includes one or a combination of the steps of the method embodiment.

[0242] In addition, the functional units in the various embodiments of the present application may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into a module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0243] The storage medium mentioned above may be a read-only memory, a magnetic disk, or an optical disk, etc. Although the embodiments of the present application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present application. Persons skilled in the art may make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present application.

Claims

1. A method for reliable parameter distribution of OVD multimodal large model based on device-edge-cloud collaboration, characterized by: The following steps are involved: The image encoder of the OVD multimodal large model is signed by a preset cloud center, and the signed image encoder is cached to a target edge server. The target edge server generates a blinding factor to use the blinding factor to perform model obfuscation on the image encoder to obtain obfuscated image encoder model parameters. The blinding factor and the obfuscated image encoder model parameters are cached to the TEE and REE in the target edge server respectively. The cloud center is used to perform batch identity authentication on preset smart devices to obtain smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device. The obfuscated image encoder model parameters and the encrypted blinding factor are loaded into the smart device REE and the smart device TEE respectively, and the obfuscated image encoder model parameters and the encrypted blinding factor are verified for credibility and integrity. When the obfuscated image encoder model parameters and the encrypted blinding factor pass the credibility and integrity verification, the target text task is generated through the cloud center, and the text embedding features of the target text task are obtained by using the text encoder of the OVD multimodal large model, and the text embedding features are sent to the target edge server. The target edge server is used to send an inference request to the smart device, so that the image encoder is subjected to model inference by the smart device to obtain image embedding features, and the image embedding features and the text embedding features are compared using a preset cross-modal feature comparison strategy to obtain target image embedding features that meet the preset similarity requirements, so that the target image embedding features are used as the inference result of the target text task.

2. The method according to claim 1, characterized in that Before signing the image encoder of the OVD multimodal large model through the preset cloud center and caching the signed image encoder to the target edge server, the method further includes: Registering with a preset key management center through the cloud center, the target edge server, and the smart device respectively to generate a public key and a private key corresponding to the cloud center and the target edge server and a prefabricated shared secret value of the smart device; Establishing a session handshake between the cloud center and the target edge server based on the public key and the private key to complete the identity authentication of the target edge server by the cloud center; The cloud center is used to perform a partitioning operation on the OVD multimodal large model, so as to deploy the text encoder of the OVD multimodal large model to the cloud center and deploy the image encoder to the smart device.

3. The method according to claim 2, characterized in that The method includes signing the image encoder of the OVD multimodal large model through a preset cloud center, caching the signed image encoder to a target edge server, and generating a blinding factor by the target edge server to perform model obfuscation on the image encoder using the blinding factor to obtain obfuscated image encoder model parameters, including: Acquire, through the cloud center, a target network layer in the image encoder that meets a preset importance requirement, and construct a corresponding network layer index set according to the target network layer; Randomly selecting random values ​​corresponding to the target edge server and the cloud center respectively, signing the network layer parameters in the image encoder based on the network layer index set and the random values, encrypting the network layer parameters to obtain encrypted network layer parameters, and caching the signed image encoder model parameters to the target edge server; Decrypting the encrypted network layer parameters by using the prefabricated shared secret value through the target edge server, and verifying the validity of the signature of the network layer parameters according to a preset signature validity condition; When the signature of the network layer parameter meets the preset signature validity condition, the blinding factor is generated in the TEE of the edge server to obfuscate the network layer parameter by the blinding factor to obtain the obfuscated image encoder model parameter.

4. The method according to claim 2, characterized in that The cloud center is used to perform batch authentication on preset smart devices to obtain smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and at the same time performs credibility and integrity verification on the obfuscated image encoder model parameters and the encrypted blinding factor, including: Sending a model request response to the target edge server through the smart device, so that the target edge server aggregates the model request responses and then sends the aggregated model request responses to the cloud center; Utilizing the cloud center to generate a challenge value based on the aggregated model request response, and sending the challenge value to the smart device through the target edge server, so that the smart device calculates the initialization program metric value of the smart device TEE using the challenge value to generate a response value, and sending the response value to the cloud center; The cloud center batch authenticates the identities of the smart devices to obtain smart device identity authentication results, and sends the smart device identity authentication results to the target edge server, so that the target edge server establishes a group session key with the smart devices, signs the group session key, and sends the signed target message to the smart devices; Based on the target message, the smart device calculates a group session key with the target edge server, authenticates the target edge server, hashes the group session key to obtain a hash result, and returns the hash result to the authenticated target edge server; Based on the hash processing result, the target edge server authenticates the smart device, encrypts the blinding factor using the group session key, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the authenticated smart device; Loading the encrypted blinding factor into the TEE of the smart device using the smart device, decrypting the encrypted blinding factor in the TEE of the smart device using the group session key to obtain a target blinding factor, and saving the obfuscated image encoder model parameters to the REE of the smart device; The target blinding factor stored in the TEE of the smart device and the obfuscated image encoder model parameters stored in the REE of the smart device are verified for credibility and integrity.

5. The method according to claim 4, characterized in that Generate a target text task through the cloud center, and use the text encoder of the OVD multimodal large model to obtain text embedding features of the target text task, and send the text embedding features to the target edge server, use the target edge server to send an inference request to the smart device, so that the smart device performs model inference on the image encoder to obtain image embedding features, and simultaneously compare the image embedding features with the text embedding features using a preset cross-modal feature comparison strategy to obtain target image embedding features that meet preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task, including: Generate the target text task through the cloud center, and input the target text task into the text encoder of the OVD multimodal large model to obtain text embedding features corresponding to the target text task; Encrypting the text embedding feature to generate text encryption embedding information, and sending the text encryption embedding information to the target edge server, so that the target edge server sends an inference request to the smart device according to the text encryption embedding information; Based on the inference request, the smart device uses the obfuscated image encoder and the target blinding factor to perform an inference operation to obtain the image embedding feature; Encrypting the image embedding feature to obtain an encrypted image embedding feature, and sending the encrypted image embedding feature to the target edge server; decrypting the image embedded encryption feature by the target edge server to obtain the image embedded decryption feature; The cross-modal feature similarity between the text embedding feature and the image embedding decryption feature is calculated, and the image embedding decryption feature with the highest cross-modal feature similarity is used as the inference result of the OVD multimodal large model.

6. A device for credibly issuing parameters of an OVD multimodal large model based on device-edge-cloud collaboration, characterized in that: include: An obfuscation module is configured to sign the image encoder of the OVD multimodal large model through a preset cloud center, cache the signed image encoder to a target edge server, and generate a blinding factor on the target edge server to perform model obfuscation on the image encoder using the blinding factor to obtain obfuscated image encoder model parameters, and cache the blinding factor and obfuscated image encoder model parameters to the TEE and REE in the target edge server, respectively. A verification module is used to use the cloud center to perform batch identity authentication on preset smart devices to obtain smart device identity authentication results. The target edge server encrypts the blinding factor based on the smart device identity authentication results, and sends the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device, and loads the obfuscated image encoder model parameters and the encrypted blinding factor to the smart device REE and the smart device TEE respectively, and performs credibility and integrity verification on the obfuscated image encoder model parameters and the encrypted blinding factor. An inference module is used to generate a target text task through the cloud center when the obfuscated image encoder model parameters and the encrypted blinding factor pass credibility and integrity verification, and use the text encoder of the OVD multimodal large model to obtain the text embedding features of the target text task, and send the text embedding features to the target edge server, use the target edge server to send an inference request to the smart device, so as to perform model inference on the image encoder through the smart device to obtain image embedding features, and at the same time use a preset cross-modal feature comparison strategy to compare the image embedding features with the text encoder to obtain target image embedding features that meet preset similarity requirements, so as to use the target image embedding features as the inference result of the target text task.

7. The device according to claim 6, characterized in that Also includes: a registration module, configured to sign the image encoder of the OVD multimodal large model through a preset cloud center and register with a preset key management center through the cloud center, the target edge server, and the smart device before caching the signed image encoder to the target edge server, so as to generate public keys and private keys corresponding to the cloud center and the target edge server and a prefabricated shared secret value for the smart device; A handshake module is used to establish a session handshake between the cloud center and the target edge server based on the public key and the private key to complete the identity authentication of the target edge server by the cloud center; A partitioning module is used to perform a partitioning operation on the OVD multimodal large model using the cloud center to deploy the text encoder of the OVD multimodal large model to the cloud center and deploy the image encoder to the smart device.

8. The device according to claim 7, characterized in that The obfuscation module includes: A construction unit, configured to obtain, through the cloud center, a target network layer in the image encoder that meets a preset importance requirement, and construct a corresponding network layer index set according to the target network layer; a first signing unit, configured to randomly select random values ​​corresponding to the target edge server and the cloud center, respectively, sign the network layer parameters in the image encoder based on the network layer index set and the random values, encrypt the network layer parameters to obtain encrypted network layer parameters, and cache the signed image encoder model parameters to the target edge server; A first decryption unit is configured to decrypt the encrypted network layer parameters using the prefabricated shared secret value through the target edge server, and verify the validity of the signature of the network layer parameters according to a preset signature validity condition; A generating unit is configured to generate the blinding factor in the TEE of the edge server when the signature of the network layer parameter satisfies the preset signature validity condition, so as to obfuscate the network layer parameter by the blinding factor to obtain the obfuscated image encoder model parameter.

9. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method for trusted delivery of OVD multimodal large model parameters based on end-edge-cloud collaboration as described in any one of claims 1 to 5.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: The program is executed by a processor to implement the OVD multimodal large model parameter trusted delivery method based on end-edge-cloud collaboration as described in any one of claims 1-5.

Images