A model processing method and apparatus, a storage medium, and an electronic device

Abstract

The specification discloses a model processing method and device, a storage medium and an electronic device, wherein the method comprises: obtaining multi-modal data corresponding to at least one target privacy transaction scenario in a divided transaction scenario cluster, and obtaining an initial scenario cluster privacy processing model of the transaction scenario cluster; and then performing model training on the initial scenario cluster privacy processing model based on the multi-modal data to obtain a trained scenario cluster privacy processing model, so that the scenario cluster privacy processing model can be used to perform privacy transaction processing on the at least one target privacy transaction scenario.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model processing method, apparatus, storage medium and electronic device. Background Technology

[0002] With the rapid development of computer technology, privacy-related scenarios such as biometric identification, confidential meetings, and user data transmission are becoming increasingly prevalent, necessitating careful consideration of privacy and security. Taking biometric identification as an example, to achieve biometric authentication, it is often necessary to collect, process, transmit, and store users' biometric information. Each step in this process carries the potential for privacy information leakage, which could lead to further financial and informational losses for users. Summary of the Invention

[0003] This specification provides a model processing method, apparatus, storage medium, and electronic device, the technical solution of which is as follows: Firstly, this specification provides a model processing method, the method comprising: Divide the privacy transaction scenario into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. Privacy transaction processing is performed on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0004] Secondly, this specification provides a model processing apparatus, the apparatus comprising: The scenario segmentation module is used to segment privacy transaction scenarios into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster. The model training module is used to obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and to train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. The transaction processing module is used to perform privacy transaction processing on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0005] Thirdly, this specification provides a computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the above-described method steps.

[0006] Fourthly, this specification provides an electronic device that may include: a processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and to execute the above-described method steps.

[0007] The beneficial effects of the technical solutions provided in some embodiments of this specification include at least the following: In one or more embodiments of this specification, by dividing privacy-related transaction scenarios into at least one transaction scenario cluster, obtaining multimodal data corresponding to at least one target privacy-related transaction scenario within each transaction scenario cluster, obtaining initial scenario cluster privacy processing models corresponding to each transaction scenario cluster, training the initial scenario cluster privacy processing models based on the multimodal data, and deploying the trained scenario cluster privacy processing models to several target privacy-related transaction scenarios for privacy-related transaction processing, this approach avoids the need for separate model training for each privacy-related transaction scenario, saving model processing resources; it also optimizes the model processing process under multiple transaction scenarios, avoiding performance degradation after the model is deployed to several target transaction scenarios, and effectively enhancing the model processing stability and scenario generalization ability. Attached Figure Description

[0008] To more clearly illustrate the technical solutions in this specification or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0009] Figure 1 This is a scene diagram of a model processing system provided in this manual; Figure 2 This is a flowchart illustrating a model processing method provided in this manual; Figure 3 This is a flowchart illustrating a model processing method provided in this manual; Figure 4 This is a flowchart illustrating another model processing method provided in this manual; Figure 5 This is a schematic diagram of the structure of a model processing device provided in this specification; Figure 6 This is a structural diagram of a scene division module provided in this manual; Figure 7 This is a schematic diagram of the structure of a scene division unit provided in this specification; Figure 8 This is a schematic diagram of the structure of a model training module provided in this manual; Figure 9 This is a schematic diagram of the structure of a model training unit provided in this manual; Figure 10 This is a schematic diagram of the structure of a transaction processing module provided in this specification; Figure 11 This is a schematic diagram of the structure of an electronic device provided in this manual. Detailed Implementation

[0010] The technical solutions in this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the description of this application, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. In the description of this application, it should be noted that, unless otherwise expressly specified and limited, "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances. Furthermore, in the description of this application, unless otherwise stated, "multiple" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship.

[0012] In practical privacy scenarios, deep learning-based privacy protection methods can be used. These methods train a privacy protection model, which then performs de-identification and de-identification of the source data, similar to encryption / decryption of the source data. Because these methods heavily rely on training data specific to the privacy scenario, they often perform well in a single scenario but struggle to achieve good performance across multiple scenarios. In practice, separate privacy protection models need to be trained for each scenario, which undoubtedly leads to significant resource consumption in model training and model dimensionality. Furthermore, deep learning-based privacy protection methods suffer from performance degradation across multiple scenarios.

[0013] The present application will now be described in detail with reference to specific embodiments.

[0014] Please see Figure 1 This is a scene diagram of a model processing system provided in this specification. Figure 1 As shown, the model processing system may include at least a client cluster and a service platform 100.

[0015] The client cluster may include at least one client, such as Figure 1 As shown, it specifically includes client 1 corresponding to user 1, client 2 corresponding to user 2, ..., client n corresponding to user n, where n is an integer greater than 0.

[0016] Each client in a client cluster can be an electronic device with communication capabilities, including but not limited to: wearable devices, handheld devices, personal computers, tablets, in-vehicle devices, smartphones, computing devices, or other processing devices connected to a wireless modem. Electronic devices may have different names in different networks, such as: user equipment, access terminal, user unit, user station, mobile station, mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device, cellular phone, cordless phone, personal digital assistant (PDA), and electronic devices in 5G networks or future evolved networks.

[0017] The service platform 100 can be a standalone server device, such as a rack-mount, blade, tower, or cabinet-type server device, or a workstation, mainframe, or other hardware device with strong computing power; or it can be a server cluster composed of multiple servers. The servers in the service cluster can be composed in a symmetrical manner, wherein each server is functionally and hierarchically equivalent in the transaction chain, and each server can provide services independently. The independent provision of services can be understood as not requiring the assistance of other servers.

[0018] In one or more embodiments of this specification, the service platform 100 may establish a communication connection with at least one client in the client cluster, and complete the data interaction during the model processing based on the communication connection, such as privacy transaction data interaction. For example, the service platform 100 may deploy the scene cluster privacy processing model obtained by the model processing method of this specification to the client; or the service platform 100 may obtain de-identified transaction data from the client.

[0019] It should be noted that the service platform 100 establishes a communication connection with at least one client in the client cluster via a network for interactive communication. This network can be a wireless network or a wired network. Wireless networks include, but are not limited to, cellular networks, wireless LANs, infrared networks, or Bluetooth networks. Wired networks include, but are not limited to, Ethernet, universal serial bus (USB), or controller area networks. In one or more embodiments of the specification, technologies and / or formats including Hyper Text Markup Language (HTML), Extensible Markup Language (XML), etc., are used to represent data exchanged over the network (such as target compressed packets). Furthermore, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), and Internet Protocol Security (IPsec) can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the aforementioned data communication technologies.

[0020] The model processing system embodiments provided in this specification and the model processing methods described in one or more embodiments belong to the same concept. The execution entity corresponding to the model processing method involved in one or more embodiments of this specification can be the aforementioned service platform 100; the execution entity corresponding to the model processing method involved in one or more embodiments of this specification can also be the electronic device corresponding to the client, which is determined based on the actual application environment. The implementation process of the model processing system embodiments can be found in the following method embodiments, which will not be repeated here.

[0021] based on Figure 1 The following is a detailed description of the model processing methods provided in one or more embodiments of this specification, as illustrated in the scene diagram.

[0022] Please see Figure 2 This document provides a flowchart illustrating a model processing method according to one or more embodiments of the present specification. This method can be implemented using a computer program and can run on a model processing device based on the von Neumann architecture. The computer program can be integrated into an application or run as a standalone utility application. The model processing device can be a service platform.

[0023] Specifically: Privacy-related scenarios such as biometric identification, confidential meetings, and user data transmission require careful consideration of privacy and security. For example, biometric identification often involves the collection, processing, transmission, and storage of users' biometric information for authentication purposes. Each step in this process carries the potential for leakage of users' biometric information (highly private information), potentially leading to further financial and informational losses. Therefore, privacy-related scenarios involve the protection of users' biometric privacy information, confidential meeting information, and personal data. In several privacy-related scenarios, the scenario cluster privacy processing model described in this specification can be used to de-identify multimodal data such as biometric images and privacy-related text, thereby achieving privacy protection. Taking images as an example of multimodal data, the scenario cluster privacy processing model can de-identify image data from multiple privacy-related scenarios within a corresponding scenario cluster, resulting in de-identified images that visually completely lack privacy features. S102: Divide the privacy transaction scenario into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; The privacy-related scenarios mentioned can include facial recognition payment, facial access control, user attendance, fingerprint (image) recognition, confidential data transmission, and so on. In practical applications, training a separate privacy protection model for each privacy-related scenario would result in significant resource consumption in model training and model dimensionality. Therefore, the model processing methods described in this specification can be used to train a scenario cluster privacy processing model, which can then be applied to multiple privacy-related scenarios.

[0024] The transaction scenario cluster contains several target privacy transaction scenarios. All target privacy transaction scenarios in the same transaction scenario cluster have certain commonalities with each other. The scenario clustering method is used to perform scenario clustering processing on "each privacy transaction scenario" so as to group several similar privacy transaction scenarios with commonalities into one transaction scenario cluster.

[0025] In one feasible implementation, a scenario clustering approach can be used to perform scenario clustering processing on "each privacy transaction scenario" so that similar privacy transaction scenarios with commonalities can be clustered to generate a transaction scenario cluster.

[0026] As an illustration, multimodal features corresponding to several privacy transaction scenarios can be extracted, and then these multimodal features can be clustered into the same transaction scenario cluster.

[0027] In one feasible implementation, scenario clusters can be formed based on scenario attributes of privacy transaction scenarios. Scenario attributes can include the target user group, transaction type, etc. Privacy transaction scenarios with similar scenario attributes are grouped into a single transaction scenario cluster.

[0028] Furthermore, after dividing the privacy transaction scenarios into at least one transaction scenario cluster, each transaction scenario cluster contains several target privacy transaction scenarios. Then, multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster can be obtained to train the scenario cluster privacy processing model.

[0029] Understandably, multimodal data, in terms of data type, typically consists of scenario transaction data with multiple modalities. For example, authentication materials in privacy authentication scenarios are usually proof materials of multiple different modalities, such as proof of funds, proof of assets, tax details, etc. Identifying these different types of proof materials involves corresponding modality recognition processing methods. Understandably, multimodal data, in terms of data element type, typically consists of data with multiple element types, such as multimodal data containing image elements or multimodal data containing text elements.

[0030] S104: Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. Indicatively, the service platform pre-constructs an initial scenario cluster privacy processing model for several target privacy scenarios within the transaction scenario cluster. The initial scenario cluster privacy processing model is set based on the image privacy protection task to adapt to the needs of several target privacy scenarios. The initial scenario cluster privacy processing model is trained by acquiring multimodal data corresponding to several target privacy scenarios under the transaction scenario cluster. When the model training termination condition is met, the trained scenario cluster privacy processing model is obtained. Furthermore, the initial scene cluster privacy processing model can be created by fitting one or more machine learning models such as Convolutional Neural Network (CNN), Deep Neural Network (DNN), Recurrent Neural Networks (RNN), embedding models, Gradient Boosting Decision Tree (GBDT) models, and Logistic Regression (LR) models.

[0031] In one or more embodiments of this specification, the model termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific model termination conditions can be determined based on actual circumstances and will not be elaborated here.

[0032] Understandably, this scenario cluster privacy processing model is applicable to each target privacy transaction scenario under a transaction scenario cluster, meaning that the scenario cluster privacy processing model can be deployed online to any target privacy transaction scenario under that transaction scenario cluster. Furthermore, if there are multiple transaction scenario clusters, an initial scenario cluster privacy processing model can be built for each transaction scenario cluster. Based on the multimodal data under each transaction scenario cluster, the corresponding initial scenario cluster privacy processing model can be trained to obtain a scenario cluster privacy processing model for each transaction scenario cluster.

[0033] S106: Perform privacy transaction processing on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0034] In one or more embodiments of this specification, the trained scene cluster privacy processing model may include support for multiple types of input / output image data formats for privacy-related data (e.g., JPEG, BMP, PNG, TIFF, PDF files captured by scanners, document cameras, mobile phones, and cameras), cluster deployment capability, support for custom output content formats (custom template files for a certain fixed format form, used for structured output of recognition results), support for multiple languages, support for multi-directional text recognition, support for multiple input / output file data formats for privacy-related data (including docx, xlsx, double-layer PDF, XML, TXT, CSV, etc.), memory capacity, and processing speed. In addition, it may also include support for high concurrency, support for input / output decoupling, support for Docker container installation, and support for different computing cores (e.g., support for GPU / CPU).

[0035] Understandably, the service platform can deploy the scenario cluster privacy processing model online to any target privacy transaction scenario under the scenario cluster.

[0036] As an illustration, the service platform can deploy the scene cluster privacy processing model to at least one client associated with any target privacy transaction scenario, so that the client processes the input transaction data in the target privacy transaction scenario based on the scene cluster privacy processing model to obtain model output data. For example, the service platform deploys the scene cluster privacy processing model to the target privacy transaction scenario, and then performs image desensitization on the image transaction data in the target privacy transaction scenario based on the scene cluster privacy processing model, and outputs desensitized images, etc.

[0037] This specification describes a method for handling privacy-related transactions by dividing the scenario into at least one cluster. Multimodal data corresponding to at least one target privacy-related transaction within each cluster is obtained. Initial scenario cluster privacy processing models are then developed for each cluster. These models are trained using the multimodal data, and the trained models can be deployed to several target privacy-related transactions. This approach avoids the need for separate model training for each scenario, saving processing resources. Furthermore, it optimizes the model processing across multiple transaction scenarios, preventing performance degradation after deployment to multiple target scenarios and effectively enhancing model stability and scenario generalization capabilities.

[0038] Please see Figure 3 , Figure 3 This is a flowchart illustrating another embodiment of a model processing method proposed in one or more embodiments of this specification. Specifically: S202: Obtain reference multimodal data corresponding to each privacy transaction scenario, and extract multimodal features from the reference multimodal data corresponding to the privacy transaction scenario; The reference multimodal data is also the privacy transaction data generated in privacy transaction scenarios; The multimodal features can be obtained by extracting features from the reference multimodal data using feature engineering methods; typically, multimodal features are represented in the form of feature vectors.

[0039] Indicatively, the reference multimodal data includes reference text data and reference image data, and the multimodal features include at least text features and image features, which can be understood as text features extracted from the reference text data and image features extracted from the reference image data.

[0040] In one feasible implementation, the service platform extracts multimodal features from reference multimodal data corresponding to privacy-related scenarios. This can be achieved by inputting reference text data and reference image data into a multimodal feature network for feature extraction, thereby obtaining text features and image features.

[0041] Understandably, a multimodal feature network can be trained for the multimodal feature extraction process, and multimodal features can be extracted from reference multimodal data in privacy-related scenarios based on the trained multimodal feature network.

[0042] Furthermore, the training process of the multimodal feature network can generally be regarded as a cross-modal self-supervised model training process.

[0043] Specifically, the service platform pre-creates an initial multimodal feature network for multimodal feature extraction, and then obtains at least one multimodal sample data under a reference privacy transaction scenario. The multimodal sample data includes sample text data and sample image data. Optionally, the initial multimodal feature network can be created based on a machine learning model. This initial multimodal feature network can consist of encoders corresponding to several modal data types; for example, the multimodal feature network can include a text encoder for text modal data and an image encoder for image modal data. Using a multimodal feature network facilitates feature extraction and improves the efficiency of model processing.

[0044] The reference privacy scenario can be understood as certain privacy scenarios used to train the multimodal feature network, from which multimodal sample data is obtained. Typically, multimodal sample data includes sample text data and sample image data. Sample text data and sample image data can better reflect the commonalities among several privacy scenarios, which is convenient for subsequent scenario clustering.

[0045] Furthermore, the service platform uses a first loss function to train the initial multimodal feature network based on the sample text data and the sample image data, and obtains the trained multimodal feature network when the network training termination condition is met. In one or more embodiments of this specification, the network training termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific network training termination conditions can be determined based on actual circumstances and will not be elaborated here.

[0046] Optionally, the network structure of the initial multimodal feature network can include a text encoder and an image encoder.

[0047] Optionally, the input to the (initial) multimodal feature network is multimodal sample data from a reference privacy scenario, such as data pairs consisting of sample text data and sample image data. The output of the (initial) multimodal feature network is the text extraction feature. Image feature extraction .

[0048] The first loss function satisfies the following formula:

[0049] Wherein, Loss A is the first loss. Extract features from text. Extract features from the image.

[0050] Understandably, the first loss function is to calculate the feature distance between "text extracted features and image extracted features" in each round of training, such as the Euclidean distance.

[0051] In one feasible implementation, since directly extracting image features from sample image data may result in image background noise, which can affect the network matching process, it is necessary to extract the loss of the feature closest to the text feature from the image features. This can be regarded as further screening candidate image features from the image features and taking the minimum loss of the region text feature among several candidate image features.

[0052] Optionally, to reduce the impact of image background noise on matching, a selective search algorithm is used to extract several candidate boxes. Feature extraction is then performed based on the candidate regions where the candidate boxes are located. After extracting the candidate box features (i.e., the aforementioned candidate image features) of the regions where the candidate boxes are located, only the loss of the features in the candidate image features that are closest to the text features is calculated. For example, assuming there are i candidate image features f... img-i In each round of training, the candidate image features f are calculated respectively. img-i The minimum loss among the feature losses Loss Ai for text extraction is taken as the first loss in each round of training. The (initial) multimodal feature network is adjusted based on the first loss until the (initial) multimodal feature network converges, which means that the network training termination condition is met.

[0053] Specifically, extracting multimodal features from the reference multimodal data corresponding to the privacy transaction scenario can be achieved by using the trained multimodal feature network to extract multimodal features, such as text features and image features, from the input reference multimodal data of each privacy transaction scenario, such as text and image pairs. S204: Divide each privacy transaction scenario into at least one transaction scenario cluster based on the multimodal features corresponding to the privacy transaction scenario.

[0054] As an illustration, scenario clustering can be used to perform scenario clustering on "each privacy transaction scenario" so that similar privacy transaction scenarios with commonalities can be clustered to generate a transaction scenario cluster.

[0055] In one feasible implementation, the service platform can perform feature clustering on the multimodal features corresponding to all privacy transaction scenarios, and obtain scenario modality labels for each privacy transaction scenario through feature clustering. As can be immediately understood, the scenario modality label is used to represent the cluster of transaction scenarios to which the privacy transaction scenario belongs. The scenario modality label is the clustering result of the above feature clustering process, which can be understood as the cluster of transaction scenarios into which the privacy transaction scenario is divided.

[0056] Furthermore, the service platform divides privacy-related transaction scenarios into at least one transaction scenario cluster based on scenario modality tags.

[0057] Understandably, scenario modality tags correspond to transaction scenario clusters, and several privacy transaction scenarios with the same scenario modality tag belong to the same transaction scenario cluster.

[0058] In one or more embodiments of this specification, the feature clustering method may be a fitting of one or more of the following algorithms: k-means clustering algorithm, fuzzy C-means clustering algorithm, Gaussian mixture model algorithm (GMM algorithm). The specific feature clustering method may be determined based on the actual application scenario and is not limited here.

[0059] As an illustration, multimodal features can include text-extracted features and image-extracted features. When performing feature clustering, it can be based on text-extracted features corresponding to the privacy-related scenario; it can also be based on image-extracted features corresponding to the privacy-related scenario; or it can be based on both image-extracted features and text-extracted features corresponding to the privacy-related scenario. For example, the GMM algorithm can be used to cluster the text extraction features and image extraction features respectively to obtain scene modality labels for privacy affairs scenarios based on text extraction features; and scene modality labels for privacy affairs scenarios based on image extraction features; privacy affairs scenarios whose scene modality labels obtained based on "image extraction features and / or text extraction features" belong to the same category are grouped into a scene cluster, and finally N transaction scene clusters can be obtained.

[0060] Indicative, specifically: The service platform performs GMM clustering based on image extraction features: by setting the clustering category K, that is, setting K single Gaussian mixture models, the parameter estimation part of the model uses the EM algorithm (expectation-maximization algorithm) to estimate the GMM model parameters in the image feature space corresponding to the image extraction features, and finally the scene image label of each scene can be obtained (i.e. the clustering result of GMM clustering). The service platform performs GMM clustering based on text extraction features: by setting the cluster category K, the EM algorithm is used to estimate the GMM model parameters in the text feature space corresponding to the text extraction features, and finally the scene text label of each scene (i.e. the clustering result of GMM clustering) is obtained. The service platform integrates transaction scenario clusters: privacy transaction scenarios that belong to the same category of image extraction features or text extraction features are grouped into one scenario cluster, resulting in N scenario clusters. It is understandable that image extraction features and text extraction features under the same privacy transaction scenario usually belong to the same transaction scenario cluster.

[0061] For example, the K-means algorithm can be used to cluster multimodal features, as follows: Clustering is performed on multimodal features (vectors). In practice, a cluster number x is preset for extracting keywords from the conference. The purpose of clustering is to obtain a set of cluster numbers x indicating the number of data points from all the multimodal features (vectors) through clustering.

[0062] During clustering: 1. Randomly select k multimodal features (vectors) from the dataset as centroids; 2. For each multimodal feature (vector) in the dataset, calculate the distance (such as Euclidean distance or Manhattan distance) between the multimodal feature (vector) and each centroid, and assign the multimodal feature (vector) to the set to which the centroid indicated by the shortest distance belongs; 3. Then, recalculate the centroid for each set based on the centroid calculation formula; 4. Calculate the target distance between the new centroid and the original centroid. Based on this distance, determine whether the clustering process should be terminated. If terminated, sort the cluster sizes of each multimodal feature (vector) within the category. Typically, each set represents a transaction scenario cluster. The set is represented by a scenario modality label. The privacy transaction scenario corresponding to the multimodal feature in the set can be assigned a scenario modality label. If not terminated, execute steps 2-4 above.

[0063] Optionally, the clustering process can be terminated based on this distance. This can be achieved by setting a distance threshold, where the process terminates when the target distance is less than the threshold, and otherwise continues with steps 2-4 above.

[0064] Optionally, the distances (between the two feature vectors) mentioned above can be calculated using at least one of the following formulas: Euclidean distance, Manhattan distance, cosine distance, correlation coefficient distance, etc.

[0065] S206: Obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster.

[0066] For details, please refer to the method steps of other embodiments involved in this specification, which will not be repeated here.

[0067] S208: Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. For details, please refer to the method steps of other embodiments involved in this specification, which will not be repeated here.

[0068] S210: The scenario cluster privacy processing model includes a privacy desensitization network and an anti-privacy desensitization network. At least one client corresponding to the target privacy transaction scenario is determined, and the privacy desensitization network is sent to the client so that the client performs privacy desensitization processing based on the privacy desensitization network. S212: Perform anti-privacy desensitization processing on the client in the target privacy transaction scenario based on the anti-privacy desensitization network.

[0069] The privacy processing model for the scenario cluster includes at least a privacy de-identification network and an anti-privacy de-identification network. The anti-privacy de-identification process of the anti-privacy de-identification network can be viewed as the reverse process of the privacy de-identification network's privacy de-identification process. Further, the privacy de-identification network performs privacy de-identification on the input data and outputs de-identified data (the model output data of the privacy de-identification network); the anti-privacy de-identification network uses the de-identified data as input to perform anti-privacy de-identification processing to obtain the original input data; the original input data corresponds to the input data.

[0070] Understandably, by executing the model processing method in this manual, a scenario cluster privacy processing model can be trained for each transaction scenario cluster. After training, the scenario cluster privacy processing model can be deployed to several target privacy transaction scenarios under its corresponding transaction scenario cluster.

[0071] To illustrate, by obtaining the transaction scenario cluster corresponding to the scenario cluster privacy processing model, the privacy desensitization network in the scenario cluster privacy processing model can be deployed to at least one client associated with any target privacy transaction scenario. This allows the client to process the multimodal transaction data in the target privacy transaction scenario based on the privacy desensitization network to obtain model output data. For example, the service platform can deploy the privacy desensitization network for the target privacy transaction scenario, and then perform image desensitization on the image transaction data in the target privacy scenario based on the privacy desensitization network to output desensitized images, and so on.

[0072] In a schematic way, model deployment can be as follows: the desensitized part of the trained scene cluster privacy processing model, i.e. the privacy desensitization network, is distributed to the corresponding target privacy transaction scene. For example, for a scene belonging to transaction scene cluster A, the privacy desensitization network of scene cluster A is distributed to the corresponding client of transaction scene cluster A.

[0073] Understandably, during the data interaction between the service platform and the client, there will be data interaction between transaction data involving privacy and security in the target privacy transaction scenario. In order to ensure privacy and security, the client can perform data anonymization on the multimodal transaction data to be transmitted to the service platform. That is, after the multimodal transaction data is input into the privacy anonymization network for privacy anonymization processing, the anonymized transaction data is output, and then the client sends the anonymized transaction data to the service platform.

[0074] In a specific implementation scenario, the service platform can obtain de-identified transaction data uploaded by the client. The de-identified transaction data is the de-identified data obtained after the client collects multimodal transaction data in the target privacy transaction scenario and performs privacy de-identification processing through a privacy de-identification network. The service platform can input anonymized transaction data into an anti-privacy anonymization network for anti-privacy anonymization processing, and output reference multimodal transaction data. This reference multimodal transaction data corresponds to the original multimodal transaction data.

[0075] To illustrate, using multimodal data as image data as an example, the privacy protection process based on the scene cluster privacy processing model is as follows: Data collection: The client collects raw images based on the target privacy scenario, such as collecting users' biometric images. Biometric images involve privacy and security, so they need to be anonymized accordingly.

[0076] Data encryption: The client uses a privacy desensitization network to desensitize and encrypt the original image to obtain a desensitized image. In other words, the original image is input into the privacy desensitization network for desensitization processing and a desensitized image is output. Data Upload: The client uploads the de-identified image to the cloud service platform. Since the de-identified image has already been privacy-de-identified, privacy can be guaranteed during data interaction. Data de-identification: The service platform uses an anti-privacy de-identification network to de-identify the de-identified images and outputs a reference original image; Processing result return: Based on the original reference image, appropriate target privacy transaction processing is performed to obtain the transaction processing result, which can be returned to the client. For example, if the target privacy transaction processing can be biometric recognition, the service platform performs biometric recognition on the original reference image, obtains the recognition result, and then sends the recognition result to the client.

[0077] In one or more embodiments of this specification, by dividing privacy-related transaction scenarios into at least one transaction scenario cluster, obtaining multimodal data corresponding to at least one target privacy-related transaction scenario within each transaction scenario cluster, obtaining initial scenario cluster privacy processing models corresponding to each transaction scenario cluster, training the initial scenario cluster privacy processing models based on the multimodal data, and deploying the trained scenario cluster privacy processing models to several target privacy-related transaction scenarios for privacy-related transaction processing, this approach avoids the need for separate model training for each privacy-related transaction scenario, saving model processing resources; it also optimizes the model processing process under multiple transaction scenarios, avoiding performance degradation after deploying the model to several target transaction scenarios, effectively enhancing model processing stability and scenario generalization ability; furthermore, it proposes a self-supervised multimodal model training and feature extraction method, resulting in better model processing performance and a wider range of applicable transaction scenarios.

[0078] Please see Figure 4 , Figure 4 This is a flowchart illustrating another embodiment of a model processing method proposed in one or more embodiments of this specification. Specifically: S302: Divide the privacy transaction scenario into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; For details, please refer to the method steps of other embodiments involved in this description, which will not be repeated here.

[0079] S304: Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster.

[0080] Furthermore, the service platform obtains the initial scenario cluster privacy processing model corresponding to each transaction scenario cluster. The service platform can train the initial scenario cluster privacy processing model based on multimodal data to obtain the scenario cluster privacy processing model for each transaction scenario cluster. According to some embodiments, the service platform pre-constructs an initial scenario cluster privacy processing model for several target privacy transaction scenarios in the transaction scenario cluster. The initial scenario cluster privacy processing model may be created by fitting one or more machine learning models.

[0081] Indicatively, the initial scene cluster privacy processing model may include an initial privacy desensitization network and an initial anti-privacy desensitization network, that is, it consists of a privacy desensitization processing part and an anti-privacy desensitization part.

[0082] Indicatively, there can be multiple transaction scenario clusters. Multimodal data (such as image data) from multiple transaction scenario clusters can be separated. An initial scenario cluster privacy processing model is created for each transaction scenario cluster for model training. This results in a trained scenario cluster privacy processing model for each transaction scenario cluster, thereby achieving scenario security.

[0083] S306: Input the multimodal data into the initial scene cluster privacy processing model, and determine the modal privacy processing data corresponding to the multimodal data and the modal restoration data corresponding to the modal privacy processing data through the initial scene cluster privacy processing model; Understandably, multimodal data is input into the initial scene cluster privacy processing model, and the multimodal data is subjected to privacy desensitization processing through the initial scene cluster privacy processing model to obtain modal privacy-processed data. Furthermore, the modal privacy-processed data is subjected to anti-privacy desensitization processing through the initial scene cluster privacy processing model to obtain modal restoration data.

[0084] Understandably, anti-privacy desensitization is the inverse process of privacy desensitization. Performing anti-privacy desensitization on modal privacy-processed data using the initial scene cluster privacy processing model is equivalent to: predicting the modal privacy-processed data based on the modal privacy-processed data input to the model, and predicting the modal privacy-processed data as the output. In other words, the modal privacy-processed data corresponds to the modal privacy-processed data.

[0085] In one feasible implementation, the created initial scene cluster privacy processing model includes at least an initial privacy desensitization network and an initial anti-privacy desensitization network.

[0086] Furthermore, multimodal data is input into the initial scene cluster privacy processing model, and the initial privacy desensitization network is used to determine the modal privacy processing data based on the multimodal data, and the initial anti-privacy desensitization network is used to determine the modal restoration data based on the modal privacy processing data.

[0087] Understandably, during the training of the initial scene cluster privacy processing model based on multimodal data, the training input data is multimodal data, and the training output data is at least modal privacy processing data and modal restoration data.

[0088] To illustrate, taking multimodal data as image data as an example, the image data is input into the initial scene cluster privacy processing model. The initial privacy desensitization network performs privacy desensitization on the image data to obtain a desensitized image. The initial anti-privacy desensitization network then performs anti-privacy desensitization processing on the desensitized image to obtain the image restoration image.

[0089] Furthermore, the model structure of the initial scene cluster privacy processing model: the model contains at least two parts, the first part is the initial privacy desensitization network E, and the other part is the initial anti-privacy desensitization network D; Taking multimodal data as image data as an example, the input and output are as follows: the input of the initial privacy desensitization network E is the original image data, and the output of the initial privacy desensitization network E is the desensitized image; the input of the initial anti-privacy desensitization network D is the desensitized image, and the output of the initial anti-privacy desensitization network D is the reconstructed anti-desensitized original image, that is, the image restoration image.

[0090] S308: Based on the multimodal data, the modal privacy processing data, and the modal restoration data, the initial scene cluster privacy processing model is trained to obtain the trained scene cluster privacy processing model for the transaction scene cluster.

[0091] Specifically, the initial scene cluster privacy processing model is trained based on the multimodal data, modal privacy processing data, and modal restoration data from each round of model training. The model loss during the training process is calculated, and the parameters of the model network are adjusted based on the model loss. For example, the parameters include the weight values ​​and thresholds of the model network. The training continues until the entire initial scene cluster privacy processing model meets the end-of-training conditions and the model network converges, thus obtaining the trained scene cluster privacy processing model for the transaction scene cluster.

[0092] In one feasible implementation, the service platform can calculate the privacy processing loss and the privacy reconstruction loss, and adjust the parameters of the model network based on the privacy processing loss and the privacy reconstruction loss, as follows: In practice, the service platform determines the privacy processing loss for the initial scene cluster privacy processing model based on multimodal data and modal privacy processing data; and determines the privacy reconstruction loss for the initial scene cluster privacy processing model based on multimodal data and modal reconstruction data. In practice, the service platform trains the initial scenario cluster privacy processing model based on the privacy processing loss and privacy reconstruction loss as the total model loss, thereby obtaining the trained scenario cluster privacy processing model for the transaction scenario cluster.

[0093] Furthermore, the total model loss consists of at least two parts: the first part is the privacy protection loss, and the second part is the privacy reconstruction loss. Optionally, the initial scene cluster privacy processing model can be trained based on the privacy processing loss and the privacy reconstruction loss. Once the model training termination condition is met, the trained scene cluster privacy processing model can be obtained.

[0094] In one or more embodiments of this specification, the model termination condition may include, for example, the value of the loss function being less than or equal to a preset loss function threshold, or the number of iterations reaching a preset threshold. Specific model termination conditions can be determined based on actual circumstances and will not be elaborated here.

[0095] Specifically, it could be: The scene classification loss and the privacy processing loss are input into the loss calculation formula to obtain the total model loss; The initial scene cluster privacy processing model is trained based on the total model loss. The loss calculation formula satisfies the following formula:

[0096] Among them, Loss privacy Let I be the total loss of the model, and let I be the multimodal data. privacyFor the modal privacy processing data, I reconstruction This is the modal reconstruction data.

[0097] Understandably, during model training: for the first part of the privacy processing loss, it is expected that the privacy-de-sensitized modal privacy-processed data and multimodal data are as inconsistent as possible; taking multimodal data as image data as an example, it is expected that the de-sensitized image and the original image are as inconsistent as possible; therefore, the first part of the privacy protection loss is calculated by using the negative feature distance (such as Euclidean distance) between the multimodal data and the modal privacy-processed data as the privacy processing loss. The calculation of the negative feature distance between the multimodal data and the modal privacy-processed data can be obtained by calculating the feature vectors of the multimodal data and the modal privacy-processed data.

[0098] Understandably, during model training: for the second part, privacy reconstruction loss, it is expected that the modal reconstruction data after privacy de-identification is as consistent as possible with the multimodal data; taking multimodal data as image data as an example, it is expected that the reconstructed image after de-identification is as similar as possible to the original. Figure 1 Therefore, in the second part, the privacy reconstruction loss is calculated by using the feature distance (such as Euclidean distance) between the multimodal data and the modal reconstruction data as the privacy reconstruction loss. The feature distance between the privacy-preserving data and the modal reconstruction data can be calculated by calculating the feature vectors of the privacy-preserving data and the modal reconstruction data.

[0099] In one feasible implementation, a privacy processing model is trained for scene clusters under multiple scenarios: a privacy protection model is trained for each scene cluster. In order to overcome the impact of different privacy transaction scenarios within each scene cluster, an adversarial classification loss can be added to resist the distribution differences of data in different scenarios, so that the trained model has good model robustness and optimizes the model training process. This can effectively enhance the model's stability and scenario generalization ability for several transaction scenarios after the model is deployed.

[0100] Optionally, a scene (adversarial) classification loss, also known as the third part loss, can be introduced during the calculation of the total loss of the model. The third part loss is the scene adversarial classification loss. By introducing the scene (adversarial) classification loss, the scene information of different scenes within the scene cluster can be decoupled.

[0101] Specifically as follows: Optionally, the service platform obtains the average predicted probability value for the transaction scenario cluster; based on the multimodal data, it uses the initial scenario cluster privacy processing model to output the scenario predicted probability value for the transaction scenario cluster. The service platform determines the scene classification loss for the privacy processing model of the initial scene cluster based on the average prediction probability value and the scene prediction probability value, which is the loss mentioned in the third part above. The average predicted probability value is prior data for a scenario cluster, determined in advance based on the number of all transaction scenario clusters. For example, if the number of transaction scenario clusters is N, the average predicted probability value can typically be 1 / N. (This is illustrative; alternatively, it can be a prior value set by the expert service based on the number of transaction scenario clusters.)

[0102] The scenario prediction probability value is the probability value that the initial scenario cluster privacy processing model determines based on the input data to belong to the corresponding transaction scenario cluster of the model. It is a scenario cluster discrimination data in the model training process to identify the probability value that the current multimodal data belongs to the transaction scenario cluster of the current model.

[0103] In other words, during the training of the initial scene cluster privacy processing model, the model output includes scene prediction probability values, modal privacy processing data, and modal reconstruction data. The total model loss is calculated based on the aforementioned model output, as follows: The service platform trains the initial scene cluster privacy processing model based on the calculated scene classification loss, privacy processing loss, and privacy reconstruction loss.

[0104] In specific implementation, the scene classification loss, the privacy processing loss, and the privacy reconstruction loss are input into the second loss calculation formula to obtain the total model loss; The initial scene cluster privacy processing model is trained based on the total model loss. The second loss calculation formula satisfies the following formula:

[0105] Among them, Loss privacy Let I be the total loss of the model, and let I be the multimodal data. privacy For the modal privacy processing data, I reconstruction For the modal reconstruction data, P pred P is the predicted probability value for the scenario. avg The average predicted probability value is denoted as .

[0106] Understandably, the second loss calculation formula above consists of three parts: the first part is the privacy protection loss, the second part is the reconstruction loss, and the third part is the scene adversarial classification loss.

[0107] Understandably, during model training: for the third part, scene classification loss, scene information of different scenes within a scene cluster is decoupled; the third part, scene classification loss, calculates the norm of the scene prediction probability value and the average prediction probability value.

[0108] S310: Perform privacy transaction processing on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0109] For details, please refer to the method steps of other embodiments involved in this description, which will not be repeated here.

[0110] In one or more embodiments of this specification, by dividing privacy-related transaction scenarios into at least one transaction scenario cluster, obtaining multimodal data corresponding to at least one target privacy-related transaction scenario within the transaction scenario cluster, obtaining initial scenario cluster privacy processing models corresponding to each transaction scenario cluster, training the initial scenario cluster privacy processing models based on the multimodal data, and deploying the trained scenario cluster privacy processing models to several target privacy-related transaction scenarios for privacy-related transaction processing. This avoids training a separate model for each privacy-related transaction scenario, saving model processing resources; it also optimizes the model processing process under multiple transaction scenarios, avoiding performance degradation after deploying the model to several target transaction scenarios, effectively enhancing model processing stability and scenario generalization ability; furthermore, it proposes a self-supervised multimodal model training and feature extraction method, resulting in better model processing performance and a wider range of applicable transaction scenarios; and the scenario cluster-based model processing method introduces scenario (adversarial) classification loss to resist the distribution differences of data in different scenarios, making the trained model have good robustness.

[0111] The following will combine Figure 5 This manual provides a detailed description of the model processing device provided. It should be noted that... Figure 5 The model processing apparatus shown is used to execute this application. Figures 1-4 The methods of the embodiments shown are illustrated only in the parts relevant to this specification for ease of explanation. For specific technical details not disclosed, please refer to this application. Figures 1-4 The example shown.

[0112] Please see Figure 5 This diagram illustrates the structure of the model processing device described in this specification. The model processing device 1 can be implemented as all or part of a user terminal through software, hardware, or a combination of both. According to some embodiments, the model processing device 1 includes a scene segmentation module 11, a model training module 12, and a transaction processing module 13, specifically used for: Scene segmentation module 11 is used to segment privacy transaction scenarios into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; Model training module 12 is used to obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and to train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. Transaction processing module 13 is used to perform privacy transaction processing on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0113] Optional, such as Figure 6 As shown, the scene segmentation module 11 includes: The feature extraction unit 111 is used to acquire reference multimodal data corresponding to each privacy transaction scenario and extract multimodal features from the reference multimodal data corresponding to the privacy transaction scenario. The scenario segmentation unit 112 is used to segment each privacy transaction scenario into at least one transaction scenario cluster based on the multimodal features corresponding to the privacy transaction scenario.

[0114] Optional, such as Figure 7 As shown, the scene segmentation unit 112 includes: The tag subunit 1121 is used to perform feature clustering on the multimodal features corresponding to all the privacy transaction scenarios to obtain scenario modality tags for each privacy transaction scenario; Scene subunit 1122 is used to divide each privacy transaction scenario into at least one transaction scenario cluster based on the scene modality label.

[0115] Optionally, the reference multimodal data includes reference text data and reference image data, and the multimodal features include at least text features and image features. The feature extraction unit 111 is specifically used for: The reference text data and reference image data are input into a multimodal feature network for feature extraction to obtain text features and image features.

[0116] Optionally, the feature extraction unit 111 is specifically used for: Acquire at least one multimodal sample data under a reference privacy transaction scenario, wherein the multimodal sample data includes sample text data and sample image data; Based on the sample text data and the sample image data, the initial multimodal feature network is trained by feature extraction to obtain the trained multimodal feature network; The first loss function satisfies the following formula:

[0117] Wherein, Loss A is the first loss, f textTo extract features from the text, f img Extract features from the image.

[0118] Optional, such as Figure 8 As shown, the model training module 12 includes: Data processing unit 121 is used to input the multimodal data into the initial scene cluster privacy processing model, and determine the modal privacy processing data corresponding to the multimodal data and the modal restoration data corresponding to the modal privacy processing data through the initial scene cluster privacy processing model; The model training unit 122 is used to train the initial scene cluster privacy processing model based on the multimodal data, the modal privacy processing data, and the modal restoration data, so as to obtain the trained scene cluster privacy processing model for the transaction scene cluster.

[0119] Optionally, the initial scene cluster privacy processing model includes an initial privacy de-identification network and an initial anti-privacy de-identification network, and the data processing unit 121 is specifically used for: The multimodal data is input into the initial scene cluster privacy processing model. Based on the multimodal data, the initial privacy desensitization network is used to determine the modal privacy processing data, and based on the modal privacy processing data, the initial anti-privacy desensitization network is used to determine the modal restoration data.

[0120] Optional, such as Figure 9 As shown, the model training unit 122 includes: The loss calculation subunit 1221 is used to determine the privacy processing loss for the initial scene cluster privacy processing model based on the multimodal data and the modal privacy processing data. The loss calculation subunit 1221 is used to determine the privacy reconstruction loss for the initial scene cluster privacy processing model based on the multimodal data and the modal reconstruction data. The model training subunit 1222 is used to train the initial scene cluster privacy processing model based on the privacy processing loss and the privacy reconstruction loss, so as to obtain the trained scene cluster privacy processing model for the transaction scene cluster.

[0121] Optionally, the loss calculation subunit 1221 is used for: Obtain the average predicted probability value for the transaction scenario cluster; based on the multimodal data, use the initial scenario cluster privacy processing model to output the scenario predicted probability value for the transaction scenario cluster; The scene classification loss for the initial scene cluster privacy processing model is determined based on the average predicted probability value and the scene predicted probability value. The model training subunit 1222 is specifically used to: train the initial scene cluster privacy processing model based on the scene classification loss, the privacy processing loss, and the privacy reconstruction loss.

[0122] Optionally, the model training subunit 1222 is specifically used to: input the scene classification loss, the privacy processing loss, and the privacy reconstruction loss into the second loss calculation formula to obtain the total model loss; The initial scene cluster privacy processing model is trained based on the total model loss. The second loss calculation formula satisfies the following formula:

[0123] Wherein, Lossprivacy is the total loss of the model, I is the multimodal data, Iprivacy is the modal privacy processing data, Ireconstruction is the modal reconstruction data, Ppred is the scene prediction probability value, and Pavg is the average prediction probability value.

[0124] Optionally, the scene cluster privacy processing model includes a privacy de-identification network and an anti-privacy de-identification network. like Figure 10 As shown, the transaction processing module 13 includes: Network deployment unit 131 is used to determine at least one client corresponding to the target privacy transaction scenario, and send the privacy desensitization network to the client so that the client performs privacy desensitization processing based on the privacy desensitization network; The transaction processing unit 132 is used to perform anti-privacy desensitization processing on the client in the target privacy transaction scenario based on the anti-privacy desensitization network.

[0125] Optionally, the transaction processing unit 132 is specifically used for: Obtain de-identified transaction data uploaded by the client. The de-identified transaction data is the de-identified data obtained after the multimodal transaction data in the target privacy transaction scenario has been processed by the privacy de-identification network. The de-identified transaction data is input into an anti-privacy de-identification network for anti-privacy de-identification processing, and reference multimodal transaction data is output.

[0126] It should be noted that the model processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when executing the model processing method. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the model processing apparatus and the model processing method embodiments provided in the above embodiments belong to the same concept, and the implementation process is detailed in the method embodiments, which will not be repeated here.

[0127] The serial numbers in this specification are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0128] In one or more embodiments of this specification, by dividing privacy-related transaction scenarios into at least one transaction scenario cluster, obtaining multimodal data corresponding to at least one target privacy-related transaction scenario within the transaction scenario cluster, obtaining initial scenario cluster privacy processing models corresponding to each transaction scenario cluster, training the initial scenario cluster privacy processing models based on the multimodal data, and deploying the trained scenario cluster privacy processing models to several target privacy-related transaction scenarios for privacy-related transaction processing. This avoids training a separate model for each privacy-related transaction scenario, saving model processing resources; it also optimizes the model processing process under multiple transaction scenarios, avoiding performance degradation after deploying the model to several target transaction scenarios, effectively enhancing model processing stability and scenario generalization ability; furthermore, it proposes a self-supervised multimodal model training and feature extraction method, resulting in better model processing performance and a wider range of applicable transaction scenarios; and the scenario cluster-based model processing method introduces scenario (adversarial) classification loss to resist the distribution differences of data in different scenarios, making the trained model have good robustness.

[0129] This specification also provides a computer storage medium capable of storing multiple instructions adapted to be loaded and executed by a processor as described above. Figures 1-4 The specific execution process of the model processing method described in the illustrated embodiment can be found in [reference needed]. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0130] This application also provides a computer program product storing at least one instruction, which is loaded and executed by the processor as described above. Figures 1-4 The specific execution process of the model processing method described in the illustrated embodiment can be found in [reference needed]. Figures 1-4 The specific details of the illustrated embodiments will not be elaborated here.

[0131] Please see Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 11 As shown, the electronic device 1000 may include: at least one processor 1001, at least one network interface 1004, a user interface 1003, a memory 1005, and at least one communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or a camera; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The processor 1001 may include one or more processing cores. The processor 1001 connects to various parts within the server 1000 using various interfaces and lines, and performs various functions of the server 1000 and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 1005, and by calling data stored in the memory 1005. Optionally, the processor 1001 can be implemented using at least one of the following hardware forms: Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). The processor 1001 can integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem can also be implemented as a separate chip without being integrated into the processor 1001.

[0132] The memory 1005 may include random access memory (RAM) or read-only memory. Optionally, the memory 1005 may include a non-transitory computer-readable storage medium. The memory 1005 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 1005 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 1005 may also be at least one storage device located remotely from the aforementioned processor 1001. Figure 11 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and application programs.

[0133] exist Figure 11 In the illustrated electronic device 1000, the user interface 1003 is mainly used to provide an input interface for the user and to obtain user input data; while the processor 1001 can be used to call the application program stored in the memory 1005 and specifically perform the following operations: Divide the privacy transaction scenario into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, and train the initial scenario cluster privacy processing model based on the multimodal data to obtain the scenario cluster privacy processing model for the transaction scenario cluster. Privacy transaction processing is performed on at least one of the target privacy transaction scenarios based on the scenario cluster privacy processing model.

[0134] In one embodiment, when the processor 1001 performs the step of dividing the privacy transaction scenario into at least one transaction scenario cluster, it specifically executes the following steps: Obtain reference multimodal data corresponding to each privacy transaction scenario, and extract multimodal features from the reference multimodal data corresponding to the privacy transaction scenario; Based on the multimodal features corresponding to the privacy transaction scenarios, each privacy transaction scenario is divided into at least one transaction scenario cluster.

[0135] In one embodiment, when the processor 1001 divides each privacy transaction scenario into at least one transaction scenario cluster based on the multimodal features corresponding to the privacy transaction scenario, it specifically performs the following steps: Feature clustering is performed on the multimodal features corresponding to all the privacy transaction scenarios to obtain scenario modality labels for each privacy transaction scenario; Based on the scenario modal labels, each privacy transaction scenario is divided into at least one transaction scenario cluster.

[0136] In one embodiment, the reference multimodal data includes reference text data and reference image data, and the multimodal features include at least text features and image features. When the processor 1001 extracts the multimodal features from the reference multimodal data corresponding to the privacy transaction scenario, it specifically performs the following steps: The reference text data and reference image data are input into a multimodal feature network for feature extraction to obtain text features and image features.

[0137] In one embodiment, the processor 1001 further performs the following steps when executing the model processing method: Acquire at least one multimodal sample data under a reference privacy transaction scenario, wherein the multimodal sample data includes sample text data and sample image data; Based on the sample text data and the sample image data, the initial multimodal feature network is trained by feature extraction to obtain the trained multimodal feature network; The first loss function satisfies the following formula:

[0138] Wherein, Loss A is the first loss, f text To extract features from the text, f img Extract features from the image.

[0139] In one embodiment, when the processor 1001 performs model training on the initial scene cluster privacy processing model using the multimodal data to obtain a scene cluster privacy processing model for the transaction scene cluster, it specifically performs the following steps: The multimodal data is input into the initial scene cluster privacy processing model, and the modal privacy processing data corresponding to the multimodal data and the modal restoration data corresponding to the modal privacy processing data are determined through the initial scene cluster privacy processing model. The initial scene cluster privacy processing model is trained based on the multimodal data, the modal privacy processing data, and the modal restoration data to obtain the trained scene cluster privacy processing model for the transaction scene cluster.

[0140] In one embodiment, the initial scene cluster privacy processing model includes an initial privacy desensitization network and an initial anti-privacy desensitization network. When the processor 1001 executes the step of inputting the multimodal data into the initial scene cluster privacy processing model, and determining the modality privacy processing data corresponding to the multimodal data and the modality restoration data corresponding to the modality privacy processing data through the initial scene cluster privacy processing model, the processor 1001 specifically performs the following steps: The multimodal data is input into the initial scene cluster privacy processing model. Based on the multimodal data, the initial privacy desensitization network is used to determine the modal privacy processing data, and based on the modal privacy processing data, the initial anti-privacy desensitization network is used to determine the modal restoration data.

[0141] In one embodiment, when the processor 1001 performs model training on the initial scene cluster privacy processing model based on the multimodal data, the modal privacy processing data, and the modal restoration data to obtain a trained scene cluster privacy processing model for the transaction scene cluster, it specifically performs the following steps: Based on the multimodal data and the modal privacy processing data, determine the privacy processing loss for the initial scene cluster privacy processing model; Based on the multimodal data and the modal reconstruction data, determine the privacy reconstruction loss for the initial scene cluster privacy processing model; The initial scenario cluster privacy processing model is trained based on the privacy processing loss and the privacy reconstruction loss to obtain the trained scenario cluster privacy processing model for the transaction scenario cluster.

[0142] In one embodiment, when executing the model processing method, the processor 1001 further performs the following steps: Obtain the average predicted probability value for the transaction scenario cluster; based on the multimodal data, use the initial scenario cluster privacy processing model to output the scenario predicted probability value for the transaction scenario cluster; The scene classification loss for the initial scene cluster privacy processing model is determined based on the average predicted probability value and the scene predicted probability value. When training the initial scene cluster privacy processing model based on the privacy processing loss and the privacy reconstruction loss, the following steps are specifically performed: The initial scene cluster privacy processing model is trained based on the scene classification loss, the privacy processing loss, and the privacy reconstruction loss.

[0143] In one embodiment, when the processor 1001 performs model training on the initial scene cluster privacy processing model based on the scene classification loss, the privacy processing loss, and the privacy reconstruction loss, it specifically executes the following steps: The scene classification loss, the privacy processing loss, and the privacy reconstruction loss are input into the second loss calculation formula to obtain the total model loss; The initial scene cluster privacy processing model is trained based on the total model loss. The second loss calculation formula satisfies the following formula:

[0144] Among them, Loss privacy Let I be the total loss of the model, and let I be the multimodal data. privacy For the modal privacy processing data, I reconstruction For the modal reconstruction data, P pred P is the predicted probability value for the scenario. avg The average predicted probability value is denoted as .

[0145] In one embodiment, the scene cluster privacy processing model includes a privacy desensitization network and an anti-privacy desensitization network. When the processor 1001 performs privacy transaction processing on at least one of the target privacy transaction scenarios based on the scene cluster privacy processing model, it specifically executes the following steps: Identify at least one client corresponding to the target privacy transaction scenario, and send the privacy de-identification network to the client so that the client performs privacy de-identification processing based on the privacy de-identification network; The anti-privacy desensitization network is used to perform anti-privacy desensitization processing on the client in the target privacy transaction scenario.

[0146] In one embodiment, when the processor 1001 performs anti-privacy desensitization processing on the client in the target privacy transaction scenario based on the anti-privacy desensitization network, it specifically executes the following steps: Obtain de-identified transaction data uploaded by the client. The de-identified transaction data is the de-identified data obtained after the multimodal transaction data in the target privacy transaction scenario has been processed by the privacy de-identification network. The de-identified transaction data is input into an anti-privacy de-identification network for anti-privacy de-identification processing, and reference multimodal transaction data is output.

[0147] In one or more embodiments of this specification, by dividing privacy-related transaction scenarios into at least one transaction scenario cluster, obtaining multimodal data corresponding to at least one target privacy-related transaction scenario within the transaction scenario cluster, obtaining initial scenario cluster privacy processing models corresponding to each transaction scenario cluster, training the initial scenario cluster privacy processing models based on the multimodal data, and deploying the trained scenario cluster privacy processing models to several target privacy-related transaction scenarios for privacy-related transaction processing. This avoids training a separate model for each privacy-related transaction scenario, saving model processing resources; it also optimizes the model processing process under multiple transaction scenarios, avoiding performance degradation after deploying the model to several target transaction scenarios, effectively enhancing model processing stability and scenario generalization ability; furthermore, it proposes a self-supervised multimodal model training and feature extraction method, resulting in better model processing performance and a wider range of applicable transaction scenarios; and the scenario cluster-based model processing method introduces scenario (adversarial) classification loss to resist the distribution differences of data in different scenarios, making the trained model have good robustness.

[0148] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory, or random access memory, etc.

[0149] The above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, any equivalent variations made in accordance with the claims of this application shall still fall within the scope of this application.

Claims

1. A model processing method, the method comprising: Divide the privacy transaction scenario into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster; Obtain the initial scenario cluster privacy processing model corresponding to the transaction scenario cluster, wherein the initial scenario cluster privacy processing model includes an initial privacy desensitization network and an initial anti-privacy desensitization network; The multimodal data is input into the initial scene cluster privacy processing model. Based on the multimodal data, the initial privacy desensitization network is used to determine the modal privacy processing data, and based on the modal privacy processing data, the initial anti-privacy desensitization network is used to determine the modal restoration data. Based on the multimodal data, the modal privacy processing data, and the modal restoration data, the privacy processing loss and the privacy reconstruction loss for the initial scene cluster privacy processing model are determined. Based on the privacy processing loss and the privacy reconstruction loss, the initial scene cluster privacy processing model is trained to obtain a scene cluster privacy processing model for the transaction scene cluster. The privacy desensitization network in the scenario cluster privacy processing model is sent to the client corresponding to the target privacy transaction scenario, so that the client performs privacy desensitization processing based on the privacy desensitization network, and performs anti-privacy desensitization processing on the client in the target privacy transaction scenario based on the anti-privacy desensitization network in the scenario cluster privacy processing model.

2. The method according to claim 1, wherein dividing the privacy transaction scenario into at least one transaction scenario cluster includes: Obtain reference multimodal data corresponding to each privacy transaction scenario, and extract multimodal features from the reference multimodal data corresponding to the privacy transaction scenario; Based on the multimodal features corresponding to the privacy transaction scenarios, each privacy transaction scenario is divided into at least one transaction scenario cluster.

3. The method according to claim 2, wherein dividing each privacy transaction scenario into at least one transaction scenario cluster based on the multimodal features corresponding to the privacy transaction scenario includes: Feature clustering is performed on the multimodal features corresponding to all the privacy transaction scenarios to obtain scenario modality labels for each privacy transaction scenario; Based on the scenario modal labels, each privacy transaction scenario is divided into at least one transaction scenario cluster.

4. The method according to claim 2, wherein the reference multimodal data includes reference text data and reference image data, and the multimodal features include at least text features and image features. The extraction of multimodal features from the reference multimodal data corresponding to the privacy transaction scenario includes: The reference text data and reference image data are input into a multimodal feature network for feature extraction to obtain text features and image features.

5. The method according to claim 4, further comprising: Acquire at least one multimodal sample data under a reference privacy transaction scenario, wherein the multimodal sample data includes sample text data and sample image data; Based on the sample text data and the sample image data, the initial multimodal feature network is trained using the first loss function to extract features, thereby obtaining the trained multimodal feature network; The first loss function satisfies the following formula: Wherein, Loss A is the first loss, f text To extract features from the text, f img Extract features from the image.

6. The method according to claim 1, wherein training the initial scene cluster privacy processing model based on the multimodal data, the modal privacy processing data, and the modal reconstruction data to obtain a trained scene cluster privacy processing model for the transaction scene cluster includes: Based on the multimodal data and the modal privacy processing data, determine the privacy processing loss for the initial scene cluster privacy processing model; Based on the multimodal data and the modal reconstruction data, determine the privacy reconstruction loss for the initial scene cluster privacy processing model; The initial scenario cluster privacy processing model is trained based on the privacy processing loss and the privacy reconstruction loss to obtain the trained scenario cluster privacy processing model for the transaction scenario cluster.

7. The method according to claim 6, wherein determining the privacy processing loss for the initial scene cluster privacy processing model based on the multimodal data and the modal privacy processing data, and determining the privacy reconstruction loss for the initial scene cluster privacy processing model based on the modal privacy processing data and the modal reconstruction data, comprises: The negative feature distance between the multimodal data and the modal privacy processing data is used as the privacy processing loss, and the modal privacy processing data and modal reconstruction number are calculated. The feature distance is used as the privacy reconstruction loss. The total model loss is obtained based on the privacy processing loss and the privacy reconstruction loss. The total model loss is used to train the initial scene cluster privacy processing model to obtain the trained scene cluster privacy processing model for the transaction scene cluster. The total loss of the model satisfies the following formula: Among them, Loss privacy Let I be the total loss of the model, and let I be the multimodal data. privacy For the modal privacy processing data, I reconstruction This is the modal reconstruction data.

8. The method according to claim 7, wherein the feature distance is a Euclidean distance.

9. The method according to claim 6, further comprising: Obtain the average predicted probability value for the aforementioned transaction scenario cluster; Based on the multimodal data, the initial scene cluster privacy processing model is used to output the scene prediction probability value for the transaction scene cluster; The scene classification loss for the initial scene cluster privacy processing model is determined based on the average predicted probability value and the scene predicted probability value. The process of training the initial scene cluster privacy processing model based on the privacy processing loss and the privacy reconstruction loss includes... The initial scene cluster privacy processing model is trained based on the scene classification loss, the privacy processing loss, and the privacy reconstruction loss.

10. The method according to claim 9, wherein training the initial scene cluster privacy processing model based on the scene classification loss, the privacy processing loss, and the privacy reconstruction loss comprises: The scene classification loss, the privacy processing loss, and the privacy reconstruction loss are input into the second loss calculation formula to obtain the total model loss; The initial scene cluster privacy processing model is trained based on the total model loss. The second loss calculation formula satisfies the following formula: Among them, Loss privacy Let I be the total loss of the model, and let I be the multimodal data. privacy For the modal privacy processing data, I reconstruction For the modal reconstruction data, P pred P is the predicted probability value for the scenario. avg The average predicted probability value is denoted as .

11. The method according to claim 1, wherein the scene cluster privacy processing model includes a privacy desensitization network and an anti-privacy desensitization network. The step of sending the privacy de-identification network in the scenario cluster privacy processing model to the client corresponding to the target privacy transaction scenario, so that the client performs privacy de-identification processing based on the privacy de-identification network, includes: Identify at least one client corresponding to the target privacy transaction scenario, and send the privacy de-identification network to the client so that the client can perform privacy de-identification processing based on the privacy de-identification network to obtain de-identified transaction data.

12. The method according to claim 1 or 11, wherein the anti-privacy desensitization processing of the client in the target privacy transaction scenario based on the anti-privacy desensitization network in the scenario cluster privacy processing model includes: Obtain de-identified transaction data uploaded by the client. The de-identified transaction data is the de-identified data obtained after the multimodal transaction data in the target privacy transaction scenario has been processed by the privacy de-identification network. The de-identified transaction data is input into an anti-privacy de-identification network for anti-privacy de-identification processing, and reference multimodal transaction data is output.

13. A model processing apparatus, the apparatus comprising: The scenario segmentation module is used to segment privacy transaction scenarios into at least one transaction scenario cluster, and obtain multimodal data corresponding to at least one target privacy transaction scenario in the transaction scenario cluster. The model training module is used to obtain an initial scenario cluster privacy processing model corresponding to the transaction scenario cluster. The initial scenario cluster privacy processing model includes an initial privacy desensitization network and an initial anti-privacy desensitization network. The multimodal data is input into the initial scenario cluster privacy processing model. Based on the multimodal data, the initial privacy desensitization network is used to determine modal privacy processing data. Based on the modal privacy processing data, the initial anti-privacy desensitization network is used to determine modal restoration data. Based on the multimodal data, the modal privacy processing data, and the modal restoration data, the privacy processing loss and the privacy reconstruction loss for the initial scenario cluster privacy processing model are determined. The initial scenario cluster privacy processing model is trained based on the privacy processing loss and the privacy reconstruction loss to obtain a scenario cluster privacy processing model for the transaction scenario cluster. The transaction processing module is used to send the privacy desensitization network in the scenario cluster privacy processing model to the client corresponding to the target privacy transaction scenario, so that the client performs privacy desensitization processing based on the privacy desensitization network, and performs anti-privacy desensitization processing on the client in the target privacy transaction scenario based on the anti-privacy desensitization network in the scenario cluster privacy processing model.

14. A computer storage medium storing a plurality of instructions adapted for loading by a processor and executing the method steps of any one of claims 1 to 12.

15. A computer program product storing at least one instruction, said at least one instruction being loaded by a processor and executing the method steps of any one of claims 1 to 12.

16. An electronic device comprising: A processor and a memory; wherein the memory stores a computer program adapted to be loaded by the processor and executed the method steps as claimed in any one of claims 1 to 12.

Images