Image recognition method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present invention provide an image recognition method and apparatus, an electronic device, a storage medium and a program product, the method being applied to a server, the method comprising: receiving ciphertext data sent by a terminal, the ciphertext data being obtained by encrypting image data by the terminal; inputting the ciphertext data into a privacy calculation model; performing feature extraction on the ciphertext data through a privacy calculation operator in the privacy calculation model according to a homomorphic encryption algorithm and a secret sharing technology to obtain a secret state feature; calculating the similarity between the secret state feature and each encryption feature vector in a preset image feature database; and sending the encrypted feature vector with the highest similarity to the terminal, so that the terminal decrypts the encrypted feature vector with the highest similarity to obtain an identification result. The data security can be improved.

Description

Technical Field

[0001] This invention belongs to the field of privacy computing technology, and particularly relates to an image recognition method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In existing data encryption technologies, encryption and decryption services are typically deployed on both the client and server sides, with an image recognition service also deployed on the server side. After acquiring an image, the client encrypts it to obtain ciphertext data. The client sends the ciphertext data to the server, which decrypts it to obtain plaintext data. The plaintext data is then input into the image recognition service, which extracts features to obtain plaintext feature vectors. These plaintext feature vectors are then used to search for matching feature vectors in a database to obtain a matching result. The server encrypts the matching result and sends it to the client. The client decrypts the received ciphertext data to obtain the matching result.

[0003] However, after the server decrypts the encrypted data sent by the client to obtain the plaintext data, the plaintext data is at risk of being leaked, which reduces data security. Summary of the Invention

[0004] This invention provides an image recognition method, apparatus, electronic device, storage medium, and program product that can improve data security.

[0005] In a first aspect, embodiments of the present invention provide an image recognition method applied to a server, the method comprising: The receiver sends encrypted data, which is obtained by encrypting image data by the terminal. Input the encrypted data into the privacy computing model; The privacy computing model uses privacy computing operators to extract features from the ciphertext data using homomorphic encryption algorithms and secret sharing techniques to obtain encrypted features. Calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database; The terminal sends the encrypted feature vector with the highest similarity to the terminal so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0006] In one possible implementation, the step of extracting features from the ciphertext data using the privacy computation operator in the privacy computation model according to a homomorphic encryption algorithm and a secret sharing technique to obtain encrypted features includes: The target ciphertext in the ciphertext data is decrypted based on the first privacy computation operator to obtain a decrypted share fragment. The decrypted segment is calculated using the first privacy calculation operator to obtain the first feature vector; The first feature vector is encrypted to obtain the first encrypted feature vector; Input the first encrypted feature vector into the second privacy computation operator; The first encrypted feature vector is calculated using the second privacy computation operator according to the homomorphic encryption algorithm to obtain the encrypted feature.

[0007] In one possible implementation, after calculating the similarity between the encrypted features and each encrypted feature vector in a preset image feature database, the method further includes: The dense state features are stored in the preset image feature database.

[0008] In one possible implementation, before inputting the ciphertext data into the privacy computing model, the method further includes: Extract computational operators from image recognition models; In response to the user's operation, the computation operator is modified to obtain a privacy computation operator; Based on the correspondence between privacy computation operators and computation operators, a target computation operator corresponding to the privacy computation operator is determined, and the operator in the image recognition model is replaced with the target computation operator to obtain the privacy computation model.

[0009] Secondly, embodiments of this application provide an image recognition method applied to a terminal, the method comprising: The acquired image data is secretly shared and fragmented to obtain multiple fragmented image data. For each fragment image data, the fragment image data is encrypted using the public key corresponding to the server to obtain the ciphertext data corresponding to the fragment image data; The encrypted data is sent to the server so that the server can input the encrypted data into the privacy computing model; the encrypted data is then processed by the privacy computing operator in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology to obtain encrypted features; the similarity between the encrypted features and each encrypted feature vector in the preset image feature database is calculated. Receive the encrypted feature vector with the highest similarity sent by the server; The encrypted feature vector with the highest similarity is decrypted to obtain the recognition result.

[0010] In one possible implementation, the process of secretly sharing and fragmenting the acquired image data to obtain multiple fragmented image data includes: The image data is converted to obtain tensor data; When the tensor data is integer data, a preset number of integer random numbers are generated, the preset number being the number of the server-side random numbers minus one. Calculate the difference between the tensor data and the preset number of integer random numbers to obtain the first fragment data; The first fragment data and the preset number of integer random numbers are used as the multiple fragment image data.

[0011] One possible implementation also includes: When the tensor data is floating-point data, the product of the tensor data and the preset scaling factor is calculated to obtain the scaled data; Remove the decimal places from the scaled data to obtain integer scaled data; Generate a preset number of integer random numbers, wherein the preset number is the number of numbers on the server minus one; Calculate the difference between the integer-scaled data and the preset number of integer random numbers to obtain the second fragment data; The second fragment data and the preset number of integer random numbers are used as the multiple fragment image data.

[0012] In one possible implementation, after sending the encrypted data to the server, the method further includes: Receive the encrypted feature vector sent by each server; For each encrypted feature vector sent by the server, the encrypted feature vector is decrypted according to the public key corresponding to the server to obtain plaintext image data; The sum of each plaintext image data is calculated to obtain the recognition result.

[0013] Thirdly, embodiments of this application provide an image recognition apparatus applied to a server, the apparatus comprising: The receiving module is used to receive encrypted data sent by the terminal, wherein the encrypted data is obtained by the terminal encrypting image data; The input module is used to input the encrypted data into the privacy computing model; The feature extraction module is used to extract features from the ciphertext data using the privacy computing operators in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology, so as to obtain the encrypted features; The calculation module is used to calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database; The sending module is used to send the encrypted feature vector with the highest similarity to the terminal, so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0014] Fourthly, embodiments of this application provide an image recognition device applied to a terminal, the device comprising: The processing module is used to perform secret sharing fragmentation processing on the acquired image data to obtain multiple fragmented image data. An encryption module is used to encrypt each fragment image data using the public key corresponding to the server, so as to obtain the ciphertext data corresponding to the fragment image data. A sending module is used to send the encrypted data to the server, so that the server can input the encrypted data into a privacy computing model; extract features from the encrypted data using a privacy computing operator in the privacy computing model according to a homomorphic encryption algorithm and a secret sharing technique to obtain encrypted features; and calculate the similarity between the encrypted features and each encrypted feature vector in a preset image feature database. The receiving module is used to receive the encrypted feature vector with the highest similarity sent by the server; The decryption module is used to decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0015] Fifthly, embodiments of the present invention provide an image recognition device, the device comprising: a processor, and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the image recognition method as described in the first or second aspect.

[0016] In a sixth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the image recognition method as described in the first or second aspect.

[0017] In a seventh aspect, embodiments of the present invention provide a computer program product, including a computer program that, when executed, implements the image recognition method as described in the first or second aspect.

[0018] This invention discloses an image recognition method, apparatus, device, medium, and program product. After receiving encrypted data from a terminal, the server inputs the encrypted data into a privacy computing model. Using a privacy computing operator within the model, features are extracted from the encrypted data according to a homomorphic encryption algorithm to obtain encrypted features. Thus, the privacy computing operator can extract features without decrypting the encrypted data, avoiding data leakage during processing. Subsequently, the similarity between the encrypted features and encrypted feature vectors in a preset image feature database is calculated. The similarity determines the encrypted feature vector that matches the encrypted data. The terminal then decrypts the encrypted feature vector to obtain the image recognition result. Therefore, the server does not decrypt the encrypted data throughout the image recognition process and does not store plaintext data, thereby improving data security. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments of the present invention will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a schematic diagram of an image recognition system structure provided in an embodiment of the present invention; Figure 2 This is a flowchart illustrating an image recognition method applied to a server, as provided in an embodiment of the present invention. Figure 3 This is a flowchart illustrating a feature extraction method provided in an embodiment of the present invention; Figure 4 This is an exemplary schematic diagram of an image recognition method provided in an embodiment of the present invention; Figure 5 This is a flowchart illustrating a method for constructing a privacy computing model according to an embodiment of the present invention; Figure 6 This is an exemplary schematic diagram of a model conversion method provided in an embodiment of the present invention; Figure 7 This is a schematic diagram of an image recognition process applied to a terminal according to an embodiment of the present invention; Figure 8 This is a flowchart illustrating a method for constructing fragmented image data according to an embodiment of the present invention; Figure 9 This is an exemplary schematic diagram of another image recognition method provided in an embodiment of the present invention; Figure 10 This is a schematic diagram of the structure of an image recognition device applied to a server, provided in an embodiment of the present invention; Figure 11This is a schematic diagram of the structure of an image recognition device applied to a terminal according to an embodiment of the present invention; Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] The features and exemplary embodiments of various aspects of the present invention will now be described in detail. To make the objectives, technical solutions, and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely intended to explain the present invention and not to limit the present invention. For those skilled in the art, the present invention can be practiced without some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present invention by illustrating examples of the invention.

[0022] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes said element.

[0023] It should be noted that the acquisition, storage, use, and processing of data in this application embodiment all comply with the relevant provisions of national laws and regulations.

[0024] It should be noted that in the embodiments of this application, certain software, components, models and other existing solutions in the industry may be mentioned. These should be regarded as exemplary and are only intended to illustrate the feasibility of implementing the technical solution of this application. However, it does not mean that the applicant has used or necessarily used the solution.

[0025] To address the problems of the prior art, embodiments of the present invention provide an image recognition method, apparatus, device, medium, and program product.

[0026] The system architecture for image recognition provided in the embodiments of the present invention will be described below. Figure 1 As shown, the system architecture includes a privacy-preserving online inference client, a privacy-preserving online inference server, and a high-performance privacy-preserving computing cluster.

[0027] The privacy-predicting online inference client includes a privacy prediction input source, the privacy-predicting online inference server includes a privacy prediction output source, and the high-performance privacy-preserving computing cluster includes multiple servers. In this embodiment, three servers are shown as examples: server 1, server 2, and server 3.

[0028] Specifically, the privacy prediction input source uses secret sharing technology to fragment the collected image data, obtaining fragmented data. The privacy prediction input source sends this fragmented data to the privacy online inference server, which then calls servers in a high-performance privacy-preserving computing cluster to perform privacy calculations, thereby extracting privacy features. Finally, the recognition result is determined by comparing the extracted privacy features with pre-set local privacy image feature data.

[0029] Figure 2 A flowchart illustrating an image recognition method according to an embodiment of the present invention is shown. Figure 2 As shown, this method is applied to the server, which is the privacy-preserving online inference server in the above embodiment. The method includes steps S201-S205: S201, Receive encrypted data sent by the terminal.

[0030] The encrypted data is obtained by encrypting the image data by the terminal. The terminal is the privacy-preserving online inference client in the above embodiment.

[0031] Specifically, after receiving image data, the terminal uses secret sharing technology to fragment the image data, obtaining multiple fragment data. Then, each fragment data is encrypted to obtain multiple ciphertext data.

[0032] S202. Input the encrypted data into the privacy computing model.

[0033] S203. Using the privacy computing operator in the privacy computing model, feature extraction is performed on the ciphertext data according to the homomorphic encryption algorithm and secret sharing technology to obtain the ciphertext features.

[0034] Specifically, when encrypting fragmented data, the terminal uses a public key pre-synchronized by the computing nodes in the high-performance privacy and security computing cluster for encryption. Then, after receiving the ciphertext data, the server distributes the ciphertext data to the corresponding computing node according to the public key used in the ciphertext data.

[0035] In this process, the server sends computation instructions to the computing nodes, directing them to perform computations according to the computational logic of the operators in the privacy-preserving computation model issued by the server. The privacy-preserving computation model consists of multiple computation operators.

[0036] S204. Calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database.

[0037] The encrypted feature vectors in the preset image feature database are obtained by encrypting historical image features.

[0038] Specifically, the cosine similarity between each encrypted feature vector and the encrypted feature can be calculated.

[0039] S205. Send the encrypted feature vector with the highest similarity to the terminal so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0040] Using the above method, after receiving the encrypted data sent by the terminal, the server inputs the encrypted data into the privacy computing model. The privacy computing operator in the model extracts features from the encrypted data using a homomorphic encryption algorithm, obtaining encrypted features. In this way, the privacy computing operator can extract features without decrypting the encrypted data, avoiding data leakage during data processing. Subsequently, the similarity between the encrypted features and encrypted feature vectors in a preset image feature database is calculated. The similarity determines the encrypted feature vector that matches the encrypted data. The terminal then decrypts the encrypted feature vector to obtain the image recognition result. Thus, the server does not decrypt the encrypted data throughout the image recognition process, and the server does not store plaintext data, thereby improving data security.

[0041] Regarding S203 above, the ciphertext data is feature-extracted using the privacy computation operator in the privacy computation model according to the homomorphic encryption algorithm and secret sharing technology to obtain the encrypted features. Specifically, this can be implemented as S2031-S2035, such as... Figure 3 As shown: S2031. Based on the first privacy computation operator, the target ciphertext in the ciphertext data is decrypted to obtain the decrypted share fragment.

[0042] The first privacy computation operator is set according to actual business needs. In practice, the first privacy computation operator is a computation operator using a method other than addition, subtraction, multiplication, and division. For example, the first privacy computation operator can be a convolution operation.

[0043] Specifically, the computing node allows the terminal to decrypt the target ciphertext through a secure protocol channel, obtaining the decrypted fragment.

[0044] S2032. Calculate the decrypted share fragment according to the first privacy calculation operator to obtain the first feature vector.

[0045] S2033. Encrypt the first feature vector to obtain the first encrypted feature vector.

[0046] Understandably, after the calculation is completed, the first feature vector of each computing node is re-encrypted to ensure that the data before and after the calculation is in an encrypted state, thereby ensuring data security.

[0047] S2034. Input the first encrypted feature vector into the second privacy calculation operator.

[0048] The second privacy computation operator is a computationally simple operator, which can be pre-set based on experience. For example, the first privacy computation operator can be a normalization and activation function.

[0049] S2035. The first encrypted feature vector is calculated using the second privacy computation operator according to the homomorphic encryption algorithm to obtain the encrypted feature.

[0050] Using the method provided in this application embodiment, the first privacy computation operator has relatively complex computational logic. Therefore, for the first privacy computation operator, the data required for computation by the first privacy computation operator in the ciphertext data is decrypted to obtain a decrypted share fragment. Then, calculations are performed according to the computational logic of the first privacy computation operator based on the decrypted share fragment to obtain the first feature vector. This reduces the computational load of the complex operator. Then, the first feature vector is re-encrypted to obtain the first encrypted feature vector. Thus, for the second privacy computation operator with simpler computational logic, the first encrypted feature vector is directly processed according to the homomorphic encryption algorithm to extract the encrypted features, avoiding the caching of plaintext data and improving data security.

[0051] In some embodiments of this application, after step S204 above, calculating the similarity between the encrypted features and each encrypted feature vector in the preset image feature database, the method further includes: Store dense features in a pre-defined image feature database.

[0052] The server stores the newly acquired dense features as encrypted feature vectors in a preset image feature database.

[0053] In this way, the preset image feature database stored locally is updated in real time by collecting image features in real time, thereby ensuring the accuracy of subsequent image recognition.

[0054] Based on the above update of the preset image feature database, combined with Figure 4 This application introduces an image recognition method provided in its embodiments, such as... Figure 4 As shown: After receiving the facial data, the local service inputs the facial recognition data into the privacy-preserving online inference client. In this embodiment, the facial data is image data collected by sensors.

[0055] The privacy-preserving online inference client sends the encrypted facial data (privacy data) to the privacy-preserving online inference server, which then executes the following two processes: The first process is the face registration process. The privacy-predicting online inference server inputs privacy data into the privacy computing prediction model, which is the privacy computing model in the above embodiment. The privacy computing prediction model schedules computing nodes in the privacy-secure computing cluster to perform feature extraction, thereby obtaining the encrypted features of the privacy data. The privacy-predicting online inference server then stores the encrypted features in the privacy face vector database. The privacy face vector database is the preset image feature database in the above embodiment.

[0056] The second process is the privacy-preserving computation recall and matching process. The online privacy inference server inputs privacy data into the privacy-preserving computation prediction model. This model then schedules computing nodes in the privacy-preserving computation cluster to extract features, obtaining a privacy data vector, which is the encrypted feature in the above embodiment. This privacy data vector is then matched with privacy face vectors in the privacy face vector database to determine the privacy result. The privacy face vector is the encrypted feature vector in the above embodiment. The privacy result is the encrypted feature vector with the highest similarity.

[0057] After the privacy-preserving online inference server sends the privacy results to the privacy-preserving online inference client, the privacy-preserving online inference client decrypts the privacy results to obtain the plaintext results, i.e., the identification results.

[0058] In some embodiments of this application, before step S202 above, where the encrypted data is input into the privacy computing model, the method for constructing the privacy computing model is as follows: Figure 5 As shown, it specifically includes: S501, Extract the computational operators of the image recognition model.

[0059] The image recognition model is an existing plaintext model. In one example, the image recognition model could be a face recognition model using the Facenet face recognition algorithm.

[0060] Specifically, the tf2onnx library is used to extract information such as computational operators, inputs, and outputs for each layer of the neural network in the image recognition model.

[0061] S502. In response to the user's operation, modify the computation operator to obtain the privacy computation operator.

[0062] In this process, users redefine each computational operator based on their actual business needs.

[0063] Specifically, for computationally complex operators, such as convolutional layers, a secret sharing technique is used to distribute the computational load. Specifically, after receiving the computation instructions from the server, the computing node, for computationally complex operators, allows the client to decrypt the target ciphertext through a secure protocol channel before executing the algorithm logic of the computation operator.

[0064] For computationally inefficient operators, such as normalization and activation functions, homomorphic encryption algorithms can be used to directly process the ciphertext data.

[0065] S503. Based on the correspondence between privacy computation operators and computation operators, determine the target computation operator corresponding to the privacy computation operator, and replace the operator in the image recognition model with the target computation operator to obtain the privacy computation model.

[0066] Specifically, the input-output tensor architecture, initialization function, initial computation, and backpropagation logic are defined based on the extracted computation operators.

[0067] The data, after being secretly shared and homomorphically encrypted, is encapsulated and inherits from TensorFlow's ExtensionType. This encapsulated data tensor possesses the characteristics of a computation graph, thus providing high-performance computation graph capabilities for privacy-preserving image recognition tasks. This effectively avoids directly using eager mode to execute computational tasks, significantly improving efficiency while maintaining data security.

[0068] The model weights are fragmented using a secret sharing technique, resulting in multiple sets of fragment weights. The number of fragment weights is the same as the number of computing nodes.

[0069] Ensuring that each computational operator can handle dense data, each operator is sequentially connected according to the network layer architecture of the plaintext model. This results in a privacy-preserving computational model that matches the image recognition model in both function and structure, providing similar prediction accuracy to the plaintext model while protecting data privacy.

[0070] Thus, without altering the original plaintext model's algorithmic logic, the computational operators within the plaintext model architecture are modified to receive encrypted data after secret sharing and homomorphic encryption. The modified operators output encrypted data, thereby maintaining data security throughout the entire computation process. This modification method not only preserves the integrity of the algorithmic logic but also ensures that the data remains encrypted before and after computation, thereby enhancing the strength of privacy protection.

[0071] Using the method provided in this application embodiment, after extracting the computational operators of the image recognition model, the computational operators are modified according to actual business needs to obtain the privacy computational operator corresponding to each computational operator. In this way, the privacy computational operator can process ciphertext data based on homomorphic encryption algorithm and secret sharing technology, thereby improving data security.

[0072] The following combination Figure 6 This section introduces the construction methods for privacy-preserving computation models, such as... Figure 6 As shown, after acquiring facial image data, data processing is performed. The processed data is then input into a neural network model for training, training a plaintext model to obtain an image recognition model. Next, the plaintext model weights are registered to a privacy weight input source. The privacy weight input source inputs the plaintext model weights into the plaintext model architecture, and then transforms the plaintext model architecture to obtain a privacy-predicting computation model.

[0073] Based on the same concept, embodiments of this application provide an image recognition method, which is applied to a terminal, such as... Figure 7 As shown, the method includes: S701. The acquired image data is secretly shared and fragmented to obtain multiple fragmented image data.

[0074] S702. For each fragment image data, encrypt the fragment image data using the public key corresponding to the computing node to obtain the ciphertext data corresponding to the fragment image data.

[0075] In this process, the public key of each computing node is pre-synchronized to the client. The computing nodes generate key pairs, which include a public key and a private key. The computing nodes synchronize the public key to the client, which uses the received public key to encrypt the fragmented image data, obtaining ciphertext data. The computing nodes then use their private keys to decrypt the received ciphertext data, thus obtaining the plaintext data.

[0076] S703. Send ciphertext data to the server so that the server can input the ciphertext data into the privacy computing model; extract features from the ciphertext data using the privacy computing operator in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology to obtain ciphertext features; calculate the similarity between the ciphertext features and each encrypted feature vector in the preset image feature database.

[0077] S704: Receive the encrypted feature vector with the highest similarity sent by the server.

[0078] S705. Decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0079] Using the method provided in this application, the terminal performs secret sharing fragmentation processing on image data to obtain fragmented image data. After encrypting the fragmented image data, it sends it to the server. Upon receiving the encrypted data from the terminal, the server inputs the encrypted data into a privacy computing model. Using the privacy computing operator in the privacy computing model, features are extracted from the encrypted data according to a homomorphic encryption algorithm to obtain encrypted features. Thus, the privacy computing operator can extract features without decrypting the encrypted data, avoiding data leakage during data processing. Subsequently, the similarity between the encrypted features and encrypted feature vectors in a preset image feature database is calculated. The similarity determines the encrypted feature vector that matches the encrypted data. The terminal decrypts the encrypted feature vector to obtain the image recognition result. Therefore, the server does not decrypt the encrypted data throughout the image recognition process, and the server does not store plaintext data, thus improving data security.

[0080] Regarding S701 above, the acquired image data is secretly shared and fragmented to obtain multiple fragmented image data, which can be specifically implemented as S7011-S7014, such as... Figure 8 As shown: S7011. Convert the image data to obtain tensor data.

[0081] Tensor data is a numerical data type, typically an integer or floating-point number.

[0082] S7012. When the tensor data is integer data, generate a preset number of integer random numbers.

[0083] The preset quantity is the number of computing nodes minus one. The dimensions of the integer random numbers and the tensor data are consistent.

[0084] S7013. Calculate the difference between the tensor data and a preset number of integer random numbers to obtain the first fragment data.

[0085] S7014. The first fragment data and a preset number of integer random numbers are used as multiple fragment image data.

[0086] Using the method provided in this application embodiment, after acquiring image data, the terminal converts the image data to obtain tensor data. Then, if the tensor data is integer data, a preset number of integer random numbers are generated, and the difference between the tensor data and the preset number of integer random numbers is calculated to obtain the first fragment data. Thus, the first fragment data and the preset number of integer random numbers are used as multiple fragmented image data. Through secret sharing technology, the image data is transformed into fragmented data, improving the computational efficiency of the privacy computing model.

[0087] In some embodiments of this application, for tensor data that is floating-point data, the following method is used for fragmentation processing: Step 1: When the tensor data is a floating-point number, calculate the product of the tensor data and the preset scaling factor to obtain the scaled data.

[0088] The preset scaling factor is set based on experience; for example, the preset scaling factor is 2^16.

[0089] Step 2: Delete the decimal places in the scaling data to obtain integer scaling data.

[0090] In this process, the integer part of the scaled tensor is increased, while the fractional part is reduced to about 1-2 digits. These fractional digits are discarded, resulting in a long integer tensor.

[0091] Step 3: Generate a preset number of integer random numbers.

[0092] The preset number is the number of computing nodes minus one.

[0093] Step 4: Calculate the difference between the integer-scaled data and the preset number of integer random numbers to obtain the second fragment data.

[0094] Step 5: Use the second fragment data and a preset number of integer random numbers as multiple fragment image data.

[0095] Using the method provided in this application embodiment, after acquiring image data, the terminal converts the image data to obtain tensor data. Then, if the tensor data is floating-point data, it is scaled to obtain scaled data. Finally, decimal places are removed from the scaled data to generate integer-type scaled data. Thus, fragmentation processing is performed based on the integer-type scaled data. A second fragment and a preset number of integer random numbers are used as multiple fragmented image data. Through secret sharing technology, the image data is transformed into fragmented data, improving the computational efficiency of the privacy computing model.

[0096] Regarding S705 above, the encrypted feature vector with the highest similarity is decrypted to obtain the recognition result, which can be specifically implemented as follows: Step A: Receive the encrypted feature vector sent by each computing node.

[0097] Step B: For each computing node's encrypted feature vector, decrypt the encrypted feature vector according to the public key corresponding to the computing node to obtain plaintext image data.

[0098] After decryption, the decryption result is divided by a preset scaling factor to obtain plaintext image data.

[0099] Specifically, plaintext image data is obtained by performing a secret sharing inverse operation on the encrypted feature vector.

[0100] Step C: Calculate the sum of each plaintext image data to obtain the recognition result.

[0101] Using the method provided in this application, after the encrypted data is distributed to multiple computing nodes through secret sharing technology, the computing nodes can determine the corresponding encrypted feature vector. The terminal can then decrypt the received encrypted feature vector and perform inverse operations on the plaintext image data using the secret sharing technology to construct the recognition result. This ensures data security.

[0102] The following combination Figure 9 This application introduces an image recognition method provided in its embodiments, such as... Figure 9 As shown, the method includes: S901, Cloud service plaintext model to privacy computing model.

[0103] The cloud service plaintext model is the image recognition model in the above embodiments.

[0104] S902, Local Service Privacy Computing Online Inference Client Data Collection and Encryption.

[0105] The local service privacy computing online inference client is the client in the above embodiments.

[0106] S903, face registration using cloud services to build a privacy-preserving face vector library.

[0107] The privacy face vector library is the preset image feature database in the above embodiments.

[0108] S904, Data collection and encryption of local service privacy computing online inference client.

[0109] The privacy computing online inference client, after acquiring a face image, fragments the face image using secret sharing technology to obtain fragment data, and then encrypts the fragment data using the public key of the computing node to obtain ciphertext data.

[0110] S905, privacy computing identification and matching for cloud services.

[0111] The cloud service is the server-side component described in the above embodiments.

[0112] S906, The local service receives the encrypted result and decrypts it.

[0113] The local service is the client in the above embodiments.

[0114] The core feature of the privacy-predicting computation model using the method provided in this application is that both input and output are encrypted fragmented data, ensuring privacy protection while maintaining the high accuracy of the original model. This provides strong security support for privacy-sensitive application scenarios. The client secretly shares the collected external facial image data through the privacy-predicting input source, ensuring that each frame of facial image is converted into reliable data fragments. These fragments are then encrypted using homomorphic encryption technology to form more secure ciphertext, ensuring the privacy and integrity of the data during transmission.

[0115] Based on the same concept, embodiments of this application provide an image recognition device applied to a server, such as... Figure 10 As shown, the device includes: The receiving module 1001 is used to receive encrypted data sent by the terminal, wherein the encrypted data is obtained by the terminal encrypting image data; Input module 1002 is used to input the encrypted data into the privacy computing model; The feature extraction module 1003 is used to extract features from the ciphertext data using the privacy computing operators in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology, so as to obtain the encrypted features; The calculation module 1004 is used to calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database; The sending module 1005 is used to send the encrypted feature vector with the highest similarity to the terminal, so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0116] In one possible implementation, the computing module 1004 is specifically used for: The target ciphertext in the ciphertext data is decrypted based on the first privacy computation operator to obtain a decrypted share fragment. The decrypted segment is calculated using the first privacy calculation operator to obtain the first feature vector; The first feature vector is encrypted to obtain the first encrypted feature vector; Input the first encrypted feature vector into the second privacy computation operator; The first encrypted feature vector is calculated using the second privacy computation operator according to the homomorphic encryption algorithm to obtain the encrypted feature.

[0117] In one possible implementation, the device further includes: A storage module is used to store the dense state features in the preset image feature database.

[0118] In one possible implementation, before inputting the ciphertext data into the privacy computing model, the device further includes: The extraction module is used to extract the computational operators of the image recognition model; The modification module is used to modify the computation operator in response to user operations to obtain a privacy computation operator; The replacement module is used to determine the target computation operator corresponding to the privacy computation operator according to the correspondence between the privacy computation operator and the computation operator, and replace the operator in the image recognition model with the target computation operator to obtain the privacy computation model.

[0119] Based on the same concept, embodiments of this application provide an image recognition device applied to a terminal, such as... Figure 11 As shown, the device includes: Processing module 1101 is used to perform secret sharing fragmentation processing on the acquired image data to obtain multiple fragmented image data; Encryption module 1102 is used to encrypt each fragment image data using the public key corresponding to the server, so as to obtain the ciphertext data corresponding to the fragment image data. The sending module 1103 is used to send the encrypted data to the server so that the server can input the encrypted data into the privacy computing model; extract features from the encrypted data using the privacy computing operator in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology to obtain encrypted features; and calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database. The receiving module 1104 is used to receive the encrypted feature vector with the highest similarity sent by the server; The decryption module 1105 is used to decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

[0120] In one possible implementation, processing module 1101 is used for: The image data is converted to obtain tensor data; When the tensor data is integer data, a preset number of integer random numbers are generated, the preset number being the number of computing nodes minus one; Calculate the difference between the tensor data and the preset number of integer random numbers to obtain the first fragment data; The first fragment data and the preset number of integer random numbers are used as the multiple fragment image data.

[0121] In one possible implementation, the device further includes: The calculation module is used to calculate the product of the tensor data and a preset scaling factor when the tensor data is floating-point data, so as to obtain scaled data; The deletion module is used to delete the decimal places in the scaled data to obtain integer scaled data; The generation module is used to generate a preset number of integer random numbers, wherein the preset number is the number of computing nodes minus one; The calculation module is also used to calculate the difference between the integer-type scaled data and the preset number of integer-type random numbers to obtain the second fragment data; The determining module is used to use the second fragment data and the preset number of integer random numbers as the multiple fragment image data.

[0122] In one possible implementation, the decryption module 1105 is used for: Receive the encrypted feature vector sent by each computing node; For each encrypted feature vector sent by a computing node, the encrypted feature vector is decrypted using the public key corresponding to the computing node to obtain plaintext image data; The sum of each plaintext image data is calculated to obtain the recognition result.

[0123] Figure 12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention is shown.

[0124] An electronic device may include a processor 1201 and a memory 1202 storing computer program instructions.

[0125] Specifically, the processor 1201 may include a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of the present invention.

[0126] Memory 1202 may include mass storage for data or instructions. For example, and not limitingly, memory 1202 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. In one instance, memory 1202 may include removable or non-removable (or fixed) media, or memory 1202 may be a non-volatile solid-state memory. Memory 1202 may be internal or external to an electronic device.

[0127] In one instance, memory 1202 may be read-only memory (ROM). In one instance, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0128] Memory 1202 may include read-only memory (ROM), random access memory (RAM), disk storage media device, optical storage media device, flash memory device, electrical, optical, or other physical / tangible memory storage device. Therefore, generally, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this disclosure.

[0129] The processor 1201 reads and executes computer program instructions stored in the memory 1202 to achieve... Figure 1 The image recognition method in the illustrated embodiment.

[0130] In one example, the electronic device may also include a communication interface 1203 and a bus 1204. For example, Figure 12 As shown, the processor 1201, memory 1202, and communication interface 1203 are connected through bus 1204 and complete communication with each other.

[0131] The communication interface 1203 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of the present invention.

[0132] Bus 1204 includes hardware, software, or both, that couples components of an electronic device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 1204 may include one or more buses. Although specific buses are described and illustrated in embodiments of the invention, the invention contemplates any suitable bus or interconnect.

[0133] Furthermore, in conjunction with the image recognition methods described in the above embodiments, this invention can be implemented using a computer storage medium. This computer storage medium stores computer program instructions; when these computer program instructions are executed by a processor, they implement any of the image recognition methods described in the above embodiments.

[0134] This application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the image recognition methods described in the above embodiments.

[0135] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0136] The functional blocks shown in the above-described structural diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memory (ROM), flash memory, erasable read-only memory (EROM), floppy disks, compact disc read-only memory (CD-ROM), optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0137] It should also be noted that the exemplary embodiments mentioned in this invention describe methods or systems based on a series of steps or apparatus. However, this invention is not limited to the order of the steps described above; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0138] The aspects of this disclosure have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by special-purpose hardware performing the specified functions or actions, or can be implemented by a combination of special-purpose hardware and computer instructions.

[0139] The above description is merely a specific embodiment of the present invention. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the protection scope of the present invention.

Claims

1. A method for image recognition, characterized in that, Applied to the server side, the method includes: The receiver sends encrypted data, which is obtained by encrypting image data by the terminal. Input the encrypted data into the privacy computing model; The privacy computing model uses privacy computing operators to extract features from the ciphertext data using homomorphic encryption algorithms and secret sharing techniques to obtain encrypted features. Calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database; The terminal sends the encrypted feature vector with the highest similarity to the terminal so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

2. The method according to claim 1, characterized in that, The privacy computing model extracts features from the ciphertext data using a homomorphic encryption algorithm and secret sharing technique to obtain encrypted features, including: The target ciphertext in the ciphertext data is decrypted based on the first privacy computation operator to obtain a decrypted share fragment. The decrypted segment is calculated using the first privacy calculation operator to obtain the first feature vector; The first feature vector is encrypted to obtain the first encrypted feature vector; Input the first encrypted feature vector into the second privacy computation operator; The first encrypted feature vector is calculated using the second privacy computation operator according to the homomorphic encryption algorithm to obtain the encrypted feature.

3. The method according to claim 1, characterized in that, After calculating the similarity between the encrypted features and each encrypted feature vector in the preset image feature database, the method further includes: The dense state features are stored in the preset image feature database.

4. The method according to claim 1, characterized in that, Before inputting the encrypted data into the privacy computing model, the method further includes: Extract computational operators from image recognition models; In response to the user's operation, the computation operator is modified to obtain a privacy computation operator; Based on the correspondence between privacy computation operators and computation operators, a target computation operator corresponding to the privacy computation operator is determined, and the operator in the image recognition model is replaced with the target computation operator to obtain the privacy computation model.

5. A method for image recognition, characterized in that, Applied to a terminal, the method includes: The acquired image data is secretly shared and fragmented to obtain multiple fragmented image data. For each fragment image data, the fragment image data is encrypted using the public key corresponding to the server to obtain the ciphertext data corresponding to the fragment image data; The encrypted data is sent to the server so that the server can input the encrypted data into the privacy computing model; the encrypted data is then processed by the privacy computing operator in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology to obtain encrypted features; the similarity between the encrypted features and each encrypted feature vector in the preset image feature database is calculated. Receive the encrypted feature vector with the highest similarity sent by the server; The encrypted feature vector with the highest similarity is decrypted to obtain the recognition result.

6. The method according to claim 5, characterized in that, The acquired image data is subjected to secret sharing fragmentation processing to obtain multiple fragmented image data, including: The image data is converted to obtain tensor data; When the tensor data is integer data, a preset number of integer random numbers are generated, the preset number being the number of computing nodes minus one; Calculate the difference between the tensor data and the preset number of integer random numbers to obtain the first fragment data; The first fragment data and the preset number of integer random numbers are used as the multiple fragment image data.

7. The method according to claim 6, characterized in that, Also includes: When the tensor data is floating-point data, the product of the tensor data and the preset scaling factor is calculated to obtain the scaled data; Remove the decimal places from the scaled data to obtain integer scaled data; Generate a preset number of integer random numbers, wherein the preset number is one less than the number of computing nodes; Calculate the difference between the integer-scaled data and the preset number of integer random numbers to obtain the second fragment data; The second fragment data and the preset number of integer random numbers are used as the multiple fragment image data.

8. The method according to claim 5, characterized in that, The process of decrypting the encrypted feature vector with the highest similarity to obtain the recognition result includes: Receive the encrypted feature vector sent by each computing node; For each encrypted feature vector sent by a computing node, the encrypted feature vector is decrypted using the public key corresponding to the computing node to obtain plaintext image data; The sum of each plaintext image data is calculated to obtain the recognition result.

9. An image recognition device, characterized in that, Applied to the server side, the device includes: The receiving module is used to receive encrypted data sent by the terminal, wherein the encrypted data is obtained by the terminal encrypting image data; The input module is used to input the encrypted data into the privacy computing model; The feature extraction module is used to extract features from the ciphertext data using the privacy computing operators in the privacy computing model according to the homomorphic encryption algorithm and secret sharing technology, so as to obtain the encrypted features; The calculation module is used to calculate the similarity between the encrypted features and each encrypted feature vector in the preset image feature database; The sending module is used to send the encrypted feature vector with the highest similarity to the terminal, so that the terminal can decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

10. An image recognition device, characterized in that, Applied to a terminal, the device includes: The processing module is used to perform secret sharing fragmentation processing on the acquired image data to obtain multiple fragmented image data. An encryption module is used to encrypt each fragment image data using the public key corresponding to the server, so as to obtain the ciphertext data corresponding to the fragment image data. A sending module is used to send the encrypted data to the server, so that the server can input the encrypted data into a privacy computing model; extract features from the encrypted data using a privacy computing operator in the privacy computing model according to a homomorphic encryption algorithm and a secret sharing technique to obtain encrypted features; and calculate the similarity between the encrypted features and each encrypted feature vector in a preset image feature database. The receiving module is used to receive the encrypted feature vector with the highest similarity sent by the server; The decryption module is used to decrypt the encrypted feature vector with the highest similarity to obtain the recognition result.

11. An image recognition device, characterized in that, The device includes: a processor and a memory storing computer program instructions; the processor reads and executes the computer program instructions to implement the image recognition method as described in any one of claims 1 to 4 or 5-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions that, when executed by a processor, implement the image recognition method as described in any one of claims 1 to 4 or 5-8.

13. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the image recognition method as described in any one of claims 1 to 4 or 5-8.

Images