Data processing method, device, equipment and medium

Abstract

The embodiment of the present application provides a data processing method, apparatus, device and medium, the method comprising: obtaining local model parameters corresponding to N local recognition models, N being the number of clients; obtaining M weight combinations associated with the local model parameter set, performing parameter fusion on the local model parameter set according to each weight combination, and obtaining M candidate global models, M being a positive integer; obtaining evaluation indicators of the M candidate global models in a multimedia verification data set, determining a target global model in the M candidate global models according to the evaluation indicators, and transmitting the target global model to N clients, so that the N clients update the parameters of the associated local recognition models according to the target global model, and obtain an object recognition model. By adopting the embodiment of the present application, the effectiveness of parameter fusion between federated training models can be improved, thereby improving the generalization effect of the federated recognition model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a data processing method, device, equipment and medium. Background Art

[0002] Federated learning has become a new training method to solve the problem of data tampering across departments and even platforms. Model training can be performed to obtain model parameters without providing one's own data, that is, joint training can be performed while ensuring data privacy. Since the federated learning process requires a large amount of data to support it, and the data is distributed among different data holders, each data holder is required to build the model. When jointly building a model with various data holders, it is necessary to fuse the model parameters trained by each data holder.

[0003] In the prior art, each data holder can use its own data to train a local model, and all data holders can periodically upload the trained local model parameters to the server for parameter averaging to obtain the total model. After the parameter averaging is completed, the total model is sent to each data holder to continue local training until the training convergence condition is reached. However, if the local model parameters of each data holder are averaged, the differences between the data held by each data holder are ignored. When the local model parameters are updated based on the total model, the optimization direction of the local models of each data holder will be inconsistent, thereby affecting the generalization effect of the federated model. Summary of the invention

[0004] The embodiments of the present application provide a data processing method, apparatus, device and medium, which can improve the effectiveness of parameter fusion between federated training models, thereby improving the generalization effect of the federated recognition model.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Obtain local model parameters corresponding to N local recognition models respectively; the N local recognition models are obtained by independently training different clients, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes an object with a target object type, N is a positive integer greater than 1, and N is the number of clients;

[0007] Obtain M weight combinations associated with the local model parameter set, perform parameter fusion on the local model parameter set according to each weight combination, and obtain M candidate global models; the local model parameter set includes local model parameters corresponding to N local recognition models, each weight combination includes training influence weights corresponding to N local model parameters, and M is a positive integer;

[0008] Obtain evaluation indicators of M candidate global models in a multimedia verification data set, determine a target global model among the M candidate global models according to the evaluation indicators, and transmit the target global model to N clients, so that the N clients update parameters of the associated local recognition models according to the target global model to obtain an object recognition model; the object recognition model is used to identify objects of the target object type contained in the multimedia data.

[0009] On the one hand, an embodiment of the present application provides a data processing method, including:

[0010] When the number of training times of the target local recognition model meets the synchronization cycle, the local model parameters corresponding to the target local recognition model are uploaded to the service device, so that the service device performs parameter fusion on the local model parameters uploaded by N clients respectively to obtain the target global model; the local model parameters uploaded by N clients respectively include the local model parameters corresponding to the target local recognition model, the target global model is determined by the evaluation indicators of M candidate global models in the multimedia verification data set respectively, the M candidate global models are determined by M weight combinations associated with the local model parameter set and the local model parameter set, the local model parameter set includes the local model parameters uploaded by N clients respectively, each weight combination includes the training influence weights associated with N clients respectively, N is a positive integer greater than 1, and M is a positive integer;

[0011] The target global model returned by the receiving service device is updated with parameters of the target local recognition model according to the target global model, and the target local recognition model after parameter update is determined as the object recognition model; the object recognition model is used to recognize objects of the target object type contained in the multimedia data.

[0012] An embodiment of the present application provides a data processing device, including:

[0013] A parameter acquisition module, used to obtain local model parameters corresponding to N local recognition models; the N local recognition models are independently trained by different clients, and the multimedia sample data held by each client for training the associated local recognition model contains objects of the target object type, N is a positive integer greater than 1, and N is the number of clients;

[0014] A parameter fusion module is used to obtain M weight combinations associated with a local model parameter set, and perform parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models; the local model parameter set includes local model parameters corresponding to N local recognition models, each weight combination includes training influence weights corresponding to N local model parameters, and M is a positive integer;

[0015] The model determination module is used to obtain evaluation indicators of M candidate global models in the multimedia verification data set, determine the target global model among the M candidate global models according to the evaluation indicators, and transmit the target global model to N clients so that the N clients can update the parameters of the associated local recognition models according to the target global model to obtain an object recognition model; the object recognition model is used to identify objects of the target object type contained in the multimedia data.

[0016] Among them, the parameter fusion module includes:

[0017] A weight combination acquisition unit, used to acquire M weight combinations associated with the local model parameter set; the M weight combinations include a weight combination i, where i is a positive integer less than or equal to M;

[0018] A weighted averaging unit is used to perform weighted averaging on the training influence weights contained in the weight combination i and the N local model parameters contained in the local model parameter set to obtain fusion model parameters, and determine the recognition model carrying the fusion model parameters as the candidate global model i associated with the weight combination i;

[0019] The model determination module is specifically used for:

[0020] The evaluation indicators of M candidate global models in the multimedia verification data set are obtained respectively, and among the M candidate global models, the candidate global model corresponding to the largest evaluation indicator is determined as the target global model.

[0021] Wherein, the weight combination acquisition unit includes:

[0022] A norm value determination subunit is used to sample N values ​​within a target value range and determine the sum of the absolute values ​​corresponding to the N values ​​as the norm value;

[0023] The weight determination subunit is used to determine the ratios between the N values ​​and the norm values ​​as a weight combination i associated with the local model parameter set.

[0024] Among them, the model determination module includes:

[0025] A verification data set acquisition unit is used to acquire a multimedia verification data set including positive sample pairs and negative sample pairs; a positive sample pair refers to a multimedia sample data pair including the same object, and a negative sample pair refers to a multimedia sample data pair including different objects;

[0026] A first prediction unit is used to obtain an alternative global model i from the M alternative global models, input the positive sample pair into the alternative global model i, and output a first object prediction result of the positive sample pair through the alternative global model i; i is a positive integer less than or equal to M;

[0027] A second prediction unit, used for inputting the negative sample pair into the candidate global model i, and outputting a second object prediction result of the negative sample pair through the candidate global model i;

[0028] The first evaluation index determination unit is used to determine the evaluation index of the candidate global model i in the multimedia verification data set according to the first object prediction result and the second object prediction result.

[0029] Wherein, the first evaluation index determination unit includes:

[0030] A prediction result statistics subunit, used to count the number of first correct predictions of the candidate global model i in the positive sample pair according to the first object prediction result, and count the number of second correct predictions of the candidate global model i in the negative sample pair according to the second object prediction result;

[0031] A total number of correct sample pairs statistics subunit, used to determine the sum of the first correct prediction number and the second correct prediction result as the total number of correct sample pairs predicted by the candidate global model i in the multimedia verification data set;

[0032] The evaluation index calculation subunit is used to obtain the total number of sample pairs corresponding to the positive sample pairs and the negative sample pairs in the multimedia verification data set, and determine the evaluation index corresponding to the alternative global model i according to the ratio between the total number of predicted correct sample pairs and the total number of sample pairs.

[0033] The number of multimedia verification data sets is P, the P multimedia verification data sets include multimedia verification data set j, P is a positive integer, and j is a positive integer less than or equal to P;

[0034] The evaluation index calculation subunit is specifically used for:

[0035] The ratio between the total number of correctly predicted sample pairs of the candidate global model i in the multimedia verification data set j and the total number of sample pairs corresponding to the multimedia verification data set j is determined as the prediction accuracy of the candidate global model i in the multimedia verification data set j;

[0036] Obtain the prediction accuracy of the candidate global model i in P multimedia verification data sets, and calculate the average accuracy corresponding to the P prediction accuracy rates and the standard deviation values ​​corresponding to the P prediction accuracy rates;

[0037] According to the average accuracy and standard deviation, the evaluation index corresponding to the candidate global model i is determined.

[0038] Among them, the model determination module includes:

[0039] A verification data set acquisition unit is used to acquire a multimedia verification data set including positive sample pairs and negative sample pairs; a positive sample pair refers to a multimedia sample data pair including the same object, and a negative sample pair refers to a multimedia sample data pair including different objects;

[0040] A similarity threshold determination unit is used to obtain the false acceptance rate of the candidate global model i included in the M candidate global models in the multimedia verification data set, and determine the similarity threshold in the similarity corresponding to the negative sample pair; the similarity threshold is determined by the number of negative sample pairs and the false acceptance rate, and i is a positive integer less than or equal to M;

[0041] The second evaluation index determination unit is used to obtain the similarity corresponding to the positive sample pairs, obtain the first sample pairs whose similarity is greater than the similarity threshold in the positive sample pairs, and determine the ratio between the number of first sample pairs and the number of positive sample pairs as the evaluation index corresponding to the alternative global model i.

[0042] Wherein, the similarity threshold determination unit includes:

[0043] An error prediction number obtaining subunit is used to obtain the error prediction number of the candidate global model i included in the M candidate global models in the negative sample pair;

[0044] The false acceptance rate determination subunit is used to determine the ratio between the number of false predictions and the number of negative sample pairs as the false acceptance rate of the candidate global model i in the multimedia verification data set.

[0045] The target global model is generated based on the N local model parameters corresponding to the rth synchronization period, where r is a positive integer;

[0046] The device also includes:

[0047] The historical global model acquisition module is used to determine the target global model corresponding to the rth synchronization period from the candidate global models according to the evaluation index, and obtain the historical global model corresponding to the (r-1)th synchronization period; the historical global model is generated based on the local model parameters uploaded by N clients in the (r-1)th synchronization period;

[0048] A model parameter difference acquisition module is used to obtain the training learning rate of N local recognition models in the rth synchronization cycle and obtain the model parameter difference between the target global model and the historical global model;

[0049] A federated momentum determination module is used to determine the ratio between the model parameter difference and the training learning rate as the federated momentum, and send the federated momentum to N clients; the federated momentum together with the target global model is used to instruct the N clients to update the parameters of the associated local recognition models, and the federated momentum is used to indicate the training directions of the N local recognition models in their respective clients.

[0050] An embodiment of the present application provides a data processing device, including:

[0051] A model parameter uploading module is used to upload the local model parameters corresponding to the target local recognition model to the service device when the number of training times of the target local recognition model meets the synchronization period, so that the service device performs parameter fusion on the local model parameters uploaded by N clients to obtain the target global model; the local model parameters uploaded by N clients include the local model parameters corresponding to the target local recognition model, the target global model is determined by the evaluation indicators of M candidate global models in the multimedia verification data set, the M candidate global models are determined by M weight combinations associated with the local model parameter set and the local model parameter set, the local model parameter set includes the local model parameters uploaded by N clients, each weight combination includes the training influence weights associated with N clients, N is a positive integer greater than 1, and M is a positive integer;

[0052] The target global model receiving module is used to receive the target global model returned by the service device, update the parameters of the target local recognition model according to the target global model, and determine the target local recognition model after the parameter update as the object recognition model; the object recognition model is used to identify objects with the target object type contained in the multimedia data.

[0053] The device further comprises:

[0054] A feature extraction module is used to obtain multimedia sample data, input the multimedia sample data into a target local recognition model, and output object space features corresponding to the multimedia sample data through the target local recognition model;

[0055] A loss function determination module is used to determine a training loss function corresponding to a target local recognition model according to the object spatial features and label information corresponding to the multimedia sample data;

[0056] The training times statistics module is used to determine the training gradient of the target local recognition model according to the training loss function, update the parameters of the target local recognition model according to the training gradient and the training learning rate corresponding to the target local recognition model, and count the training times corresponding to the target local recognition model.

[0057] The multimedia data includes a face image to be recognized, and the target object type includes a face type;

[0058] The device also includes:

[0059] A face feature extraction module is used to obtain a face image to be identified, input the face image to be identified into an object recognition model, and obtain face spatial features corresponding to the face image to be identified in the object recognition model;

[0060] The face classification module is used to determine the face classification result corresponding to the face image to be identified based on the face spatial features; the face classification result is used to characterize the identity authentication result of the object with face type contained in the face image to be identified.

[0061] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory is connected to the processor, the memory is used to store a computer program, and the processor is used to call the computer program so that the computer device executes the method provided in the above aspect of the embodiment of the present application.

[0062] On one hand, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded and executed by a processor, so that a computer device with a processor executes the method provided in the above aspect of the embodiment of the present application.

[0063] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method provided in the above aspect.

[0064] In an embodiment of the present application, local model parameters of local recognition models uploaded by N clients are obtained, wherein the N local recognition models are independently trained by different clients, and each client includes multimedia sample data for training the associated local training model, the multimedia sample data includes an object having a target object type, and N is a positive integer greater than 1; and M weight combinations (M is a positive integer) for the N local model parameters are obtained, and parameters of the N local model parameters are fused through each weight combination to obtain M candidate global models, and then the optimal target global model is selected from the M candidate global models through the evaluation indicators of the M candidate global models in the multimedia verification data set, that is, the optimal target global model is selected through the evaluation indicators on the multimedia verification data set, which can improve the fusion effectiveness between the N local model parameters, and the N clients continue to update the parameters of their respective local recognition models based on the target global model, which can improve the generalization effect of the object recognition model. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0066] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0067] Figure 2a and Figure 2b It is a schematic diagram of a federated training scenario of a recognition model provided in an embodiment of the present application;

[0068] Figure 3 It is a timing diagram of a data processing method provided in an embodiment of the present application;

[0069] Figure 4 is a schematic diagram of determining a target global model provided by an embodiment of the present application;

[0070] Figure 5 It is a flow chart of a federated model training method provided in an embodiment of the present application;

[0071] Figure 6 It is a schematic diagram of a weight combination in a multimedia verification data set provided by an embodiment of the present application;

[0072] Figure 7 This is a schematic diagram of a user identity authentication scenario provided by an embodiment of the present application;

[0073] Figure 8 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0074] Fig. 9 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0075] Fig.10 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0076] Fig.11 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0077] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0078] This application relates to artificial intelligence (AI) technology, blockchain (BlockChain) technology and cloud technology. Artificial intelligence is the theory, method, technology and application system of using digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making. Artificial intelligence technology is a comprehensive discipline that involves a wide range of fields, including both hardware-level technology and software-level technology. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. This application specifically relates to computer vision technology (Computer Vision, CV) under artificial intelligence technology.

[0079] Among them, computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify, locate and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition. The present application specifically relates to face recognition under computer vision. Through federated learning, joint training is performed while ensuring the privacy of each client data. The object recognition model obtained by the joint training can be applied to each client. The object recognition model can be used to recognize the face image in the client to obtain the face recognition result, which can be used as the basis for user identity verification.

[0080] The present application relates to cloud storage under cloud technology. Cloud storage is a new concept extended and developed from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to bring together a large number of storage devices of various types (storage devices are also called storage nodes) in the network to work together through application software or application interfaces to provide data storage and business access functions to the outside world. At present, the storage method of the storage system is: create a logical volume. When creating a logical volume, a physical storage space is allocated to each logical volume. The physical storage space may be composed of disks of a storage device or several storage devices. The client stores data on a logical volume, that is, the data is stored on the file system. The file system divides the data into many parts, each of which is an object. The object not only contains data but also contains additional information such as data identification (ID entity, ID). The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object, so that when the client requests to access the data, the file system can allow the client to access the data according to the storage location information of each object.

[0081] The process of allocating physical storage space for a logical volume by a storage system is as follows: according to the estimated capacity of the object stored in the logical volume (the estimated capacity often has a large margin relative to the actual capacity of the object to be stored) and the group of the Redundant Array of Independent Disk (RAID), the physical storage space is pre-divided into stripes, and a logical volume can be understood as a stripe, thereby allocating physical storage space for the logical volume. In this application, the multimedia sample data held by each client can be stored on different logical volumes, that is, the multimedia sample data held by each client can be stored on the file system; for the multimedia sample data held by any client, the file system can divide it into many parts, each part is an object, and the object can contain not only the multimedia sample data, but also the data identifier of the multimedia sample data. The file system writes each object into the physical storage space of the logical volume, and the file system records the storage location information of each object; when the federated training recognition model is used, the client can request to access the multimedia sample data in the file system, and the file system can allow the client to access the multimedia sample data according to the storage location information of the object.

[0082] Blockchain is a new application model of computer technologies such as distributed data storage, point-to-point transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks (also called blocks) generated by cryptography, and each data block is linked by random hashing (also called hash algorithm). The latter block contains the encrypted hash of the previous block, the corresponding timestamp and transaction data (usually represented by the hash value calculated by the Merkle tree algorithm). This design makes the block content difficult to tamper with. The distributed ledger connected in series with blockchain technology allows both parties to effectively record transactions and can permanently check the transaction. All client and service devices involved in this application can be blockchain nodes belonging to the same blockchain system. In the process of federal training recognition model, the uploaded local model parameters and the target global model can be stored on the blockchain to ensure the traceability of the parameter fusion method in the federal training process.

[0083] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 10d and a user terminal cluster, and the user terminal cluster may include one or more user terminals, and the number of user terminals is not limited here. Figure 1As shown, the user terminal cluster may specifically include user terminal 10a, user terminal 10b and user terminal 10c, etc. Among them, server 10d may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. User terminal 10a, user terminal 10b and user terminal 10c, etc. may include: smart phones, tablet computers, laptop computers, PDAs, mobile Internet devices (mobile internet devices, MID), wearable devices (such as smart watches, smart bracelets, etc.) and smart TVs, etc., which have multimedia data recognition functions. As Figure 1 As shown, the user terminal 10a, the user terminal 10b and the user terminal 10c can respectively establish a network connection with the server 10d, so that each user terminal can exchange data with the server 10d through the network connection.

[0084] like Figure 1As shown, each user terminal in the user terminal cluster can be integrated with a client, and the number of clients integrated in each user terminal can be one or more, such as different clients can be integrated in one user terminal, different clients can hold different multimedia data, and the multimedia data held by different clients can all be used to train the recognition model (it is assumed here that the multimedia data held by the clients involved in this application are of the same type of data, such as the multimedia data held by different clients are all face image data). Since training the recognition model requires a large amount of sample data, and the multimedia data held by different clients may involve privacy information or confidential information, that is, the multimedia data held by each client cannot be made public, the recognition model can be trained by federal training. In other words, each client can use the multimedia data it holds as multimedia sample data for training the recognition model, and train on the multimedia sample data it holds independently. Different clients can periodically synchronize model parameters (the synchronized model parameters can be called local model parameters at this time), that is, each client can periodically upload the trained model parameters to the server 10d, and the server 10d can collect the local model parameters uploaded by each client, and perform parameter fusion on the local model parameters uploaded by each client to obtain the target global model in each cycle, and then send the target global model to each client, and each client can continue to train the local model parameters according to the target global model until the convergence condition is reached or the number of training iterations reaches the preset maximum number of iterations, and obtain the trained object recognition model, which can be used to identify objects with target object types contained in multimedia data, and can improve the generalization recognition effect of the object recognition model. Among them, the target object types can include but are not limited to: faces, plants, commodities, pedestrians, various animals, and various scenes.

[0085] Please also see Figure 2a and Figure 2b , Figure 2a and Figure 2b Schematic diagram of a federated training scenario of a recognition model provided in an embodiment of the present application. Figure 2a and Figure 2b The client 1 shown can be the above Figure 1 The client with the federated training recognition model authority integrated in the user terminal 10a shown in the figure, the client 2 can be the above Figure 1 The client with the federated training recognition model authority integrated in the user terminal 10b shown in the figure, the client N can be the above Figure 1 The client with the federated training recognition model authority integrated in the user terminal 10c shown in the figure, the parameter service device can be the above Figure 1The server 10d shown. Figure 2a As shown, the number of clients participating in the federated training of the recognition model is N, and the value of N can be a positive integer greater than 1, such as N can be 2, 3, ...; each client can hold face sample data for training the recognition model, and the face sample data held by each client is independent of each other. For example, in order to ensure the privacy of the data, client 1 will not provide its own face sample data to other devices (for example, client 2, client N, parameter server device, etc.). Therefore, each client can use the face sample data it holds to perform local training of the recognition model locally (the recognition model trained locally by the client can be called a local recognition model, and the model parameters obtained by the client in local training can be called local model parameters).

[0086] Since the face sample data used by each client is different, each client needs to periodically upload local model parameters to the parameter service device so that the parameter service device can synchronize the local model parameters trained by N clients, that is, perform parameter fusion on the local model parameters trained by N clients to obtain a global model. For example, if every 100 training iterations (also called the number of training times, or the number of training steps) is set as a synchronization cycle, each client needs to upload local model parameters to the parameter service device once every 100 training iterations. Figure 2a As shown, when the number of training iterations of the local recognition model of client 1 reaches 100, the client 1 can send the model parameter 1 obtained by the 100th training iteration (i.e., the local model parameter obtained by client 1 in the 100th training iteration) to the parameter service device; similarly, when the number of training iterations of the local recognition model of client 2 reaches 100, the client 2 can send the model parameter 2 obtained by the 100th training iteration to the parameter service device; client N can send its own model parameter N obtained by the 110th training iteration to the parameter service device. After the parameter service device receives the local model parameters (including model parameter 1, model parameter 2, ..., model parameter N) obtained by the 100th training iteration respectively sent by N clients, different model parameter fusion schemes (for example, different weight combinations) can be obtained through the search module in the parameter service device, and the local model parameters sent by N clients are fused through the above-mentioned model parameter fusion scheme to obtain the candidate global model. At this time, the candidate global model can also be understood as a model using different model parameter fusion schemes.

[0087] Furthermore, the candidate global model can be transmitted to an evaluation unit (Arbiter), which can be a component integrated in the parameter service device or an external component that has a communication connection relationship with the parameter service device. In the evaluation unit, the evaluation index corresponding to the candidate global model can be obtained through the verification data set, and the evaluation index corresponding to the candidate global model can be returned to the parameter service device. Among them, the verification data set may include face sample data carrying label information. After the face sample data in the verification data set is input into the alternative global model, the alternative global model can output the face recognition result for the face sample data, and then the output face recognition result can be compared with the label information carried by the face sample data. If the face recognition result is the same as the label information, it means that the alternative global model predicts correctly. If the face recognition result is different from the label information, it means that the alternative global model predicts incorrectly. According to the face recognition result output by the alternative global model, the evaluation index of the alternative global model in the verification data set can be determined; the evaluation index may include but is not limited to: accuracy (accuracy, the proportion of sample data predicted correctly by the model among all sample data), recall (Recall, the proportion of sample data predicted by the model as "true" among all true sample data), precision (Precision, the proportion of sample data predicted by the model as "true" that is actually true), F1 value (an indicator designed by comprehensively considering precision and recall). The parameter service device can select the best candidate global model from the candidate global models as the target global model corresponding to the current synchronization cycle according to the evaluation index corresponding to the candidate global model, and return the target global model to each client. Each client can update the local local model parameters according to the target global model returned by the parameter service device, and continue training. When the number of training iterations of each client for the local local recognition model reaches 200, it is necessary to repeat the above operation to obtain the target global model corresponding to the next synchronization cycle for continued training until the training of the local recognition model reaches the convergence condition, or the number of training iterations reaches the set maximum number of iterations. The local model parameters at this time are saved, and the local recognition model containing the current local model parameters can be determined as a trained model. The present application can determine the trained local recognition model as an object recognition model.

[0088] like Figure 2bAs shown, the face sample data held by client 1 constitutes a data set 20a, the face sample data held by client 2 constitutes a data set 20b, ..., and the face sample data held by client N constitutes a data set 20c; the client 1 can use the data set 20a locally to train the local recognition model 20d, the client 2 can use the data set 20b locally to train the local recognition model 20e, ..., and the client N can use the data set 20c locally to train the local recognition model 20f. When the number of training iterations of the associated local recognition model of each client reaches 100 times, the local model parameters obtained by the 100th training iteration need to be sent to the parameter service device. Therefore, the parameter service device can obtain a local model parameter set 20g, which can include the local model parameters sent by the above-mentioned N clients, such as model parameter 1 sent by client 1, local model 2 sent by client 2, ..., model parameter N sent by client N.

[0089] Furthermore, the parameter service device can obtain M weight combinations for the local model parameter set 20g through the search unit (the value of M can be a positive integer, such as M can be 1, 2, 3, ...), and the M weight combinations at this time can refer to the M model parameter fusion methods selected by the search unit for the local model parameter set 20g. Each weight combination can include the training influence weights corresponding to the local model parameters contained in the local model parameter set 20g. That is to say, each weight combination can be understood as an N-dimensional vector. For example, the above-mentioned M weight combinations can include: {a1, a2, a3, ..., aN}, {b1, b2, b3, ..., bN}, and then the local model parameter set 20g can be subjected to parameter fusion according to each weight combination to obtain M alternative global models. In the evaluation unit, the evaluation indicators corresponding to the M candidate global models are obtained through the verification data set, such as the evaluation indicator of the candidate global model 1 in the verification data set is: evaluation indicator 1, the evaluation indicator of the candidate global model 2 in the verification data set is: evaluation indicator 2, ..., the evaluation indicator of the candidate global model M in the verification data set is: evaluation indicator M. The candidate global model corresponding to the largest evaluation indicator is selected from the M candidate global models as the target global model corresponding to the current synchronization cycle, and then the target global model is sent to N clients. After receiving the target global model sent by the parameter service device, any client can update the local local model parameters according to the target global model, and then continue training based on the updated local model parameters; when the number of training iterations reaches the next synchronization cycle (for example, the 200th training iteration), the above operation can be repeated until the training process of the local recognition model is completed. It can be understood that the N clients are conducting federated training on the same recognition model, that is, the initialization recognition model used before the federated training is the same, and the maximum number of iterations, synchronization cycle, training method used and other information set during the federated training process are the same; after the federated training is completed, the object recognition model obtained by each client may be the same, or there may be some differences, which is related to the differences between the face sample data held by each client. For example, after the parameter service device determines the target global model 50 in the 50th synchronization cycle, it sends it to each client, and each client can update the parameters of the local recognition model associated with each client according to the target global model 50. At this time, the local recognition models of each client are the same; each client can continue to train the local recognition model based on the face sample data held by each client. If convergence is achieved before the 51st synchronization cycle or the number of training iterations reaches the set maximum number of iterations, the training is completed to obtain the object recognition model, that is, the local model parameters obtained after the 50th synchronization cycle have not been parameter fused, so the object recognition models finally obtained by each client may have some differences.The embodiments of the present application can improve the effectiveness of parameter fusion between local model parameters during the federated training process, thereby improving the generalization effect of the object recognition model.

[0090] See also Figure 3 , Figure 3 1 is a timing diagram of a data processing method provided in an embodiment of the present application. It can be understood that the data processing method can be interactively executed by the client and the service device, and the client can be the above Figure 1 The client integrated in any user terminal in the user terminal cluster shown in the figure may be a separate server (for example, the above Figure 1 The server 10d) shown in the figure may be a server cluster composed of multiple servers, or a user terminal, etc. Figure 3 As shown, the data processing method may include the following steps:

[0091] Step S101, when the number of training times of the target local recognition model meets the synchronization period, the local model parameters corresponding to the target local recognition model are uploaded.

[0092] Specifically, when the multimedia data held by N (the value of N can be a positive integer greater than 1) clients are of the same type of data, and the multimedia data held by the N clients involve data privacy and data security, it means that the multimedia data held by the N clients cannot be aggregated; if it is necessary to use the multimedia data held by the N clients to train the recognition model, then under the premise of ensuring the data security and privacy of each client, the recognition model can be trained by federal training. In the federal training, the multimedia data held by the N clients can be used as multimedia sample data. Among them, the multimedia sample data can include face image data, user financial data, video data, user product data, etc., and each multimedia sample data can contain an object with a target object type, and the target object type can include object types such as face, pedestrian, and product.

[0093] Each of the N clients can perform independent training locally using the multimedia data it holds. The recognition model independently trained by each client can be called a local recognition model. Each client can periodically upload the parameters of the local recognition model trained independently for synchronization. In the embodiment of the present application, the synchronization period can be set according to actual needs. For example, the synchronization period can be set to K training times (also called training steps), indicating that every K steps of training of the local recognition model in the client requires uploading the local model parameters corresponding to the local recognition model to the service device (such as the above Figure 2aThe parameter service device in the corresponding embodiment) is synchronized, and the value of K is a positive integer greater than 1, such as K can be 100, 400, 1600, etc. Among them, the training process of the local recognition model for each of the N clients is similar, except that the multimedia sample data used are different. The following selects any one of the N clients as the target client, and takes the target client as an example to describe the training process of the local recognition model.

[0094] The target client can obtain the multimedia sample data it holds and input the multimedia sample data into the target local recognition model (the target local model here refers to the local recognition model independently trained locally by the target client). The target local recognition model can output the object spatial features corresponding to the multimedia sample data. Specifically, in the process of training the target local recognition model, the target client can read the multimedia sample data it holds, and organize the read multimedia sample data into a batch. The multimedia sample data contained in the batch can be input into the target local recognition model. The target local recognition model can be a convolutional neural network. At this time, the target local recognition model can include network layers such as a convolution layer (Convolution Layer), a non-linear activation layer (Relu Layer) and a pooling layer (Pooling Layer); after the multimedia sample data is input into the target local recognition model, convolution calculation (convolution calculation is performed through the convolution layer), non-linear activation function calculation (non-linear activation function calculation is performed through the non-linear activation layer), pooling calculation (pooling calculation is performed through the pooling layer) and other operations can be performed on the multimedia sample data in the target local recognition model to output the object space features corresponding to the multimedia sample data, that is, the object space features in the multimedia sample data can be extracted through the target local recognition model.

[0095] In the kth training process, the multimedia sample data contained in the batch can be expressed as X k ,The target client can use gradient descent (GD) for iterative training. Gradient descent is an iterative learning algorithm that can use multimedia sample data to update the local model parameters of the target local recognition model. The batch size (also called batch size) is a hyperparameter of gradient descent, which controls the number of training samples before the internal parameters of the target local recognition model are updated.

[0096] Furthermore, the target client can determine the training loss function corresponding to the target local recognition model based on the object spatial features and the label information corresponding to the multimedia sample data, and then determine the training gradient of the target local recognition model based on the training loss function, update the parameters of the target local recognition model based on the training gradient and the training learning rate corresponding to the target local recognition model, and count the number of training times corresponding to the target local recognition model. In other words, the target client can calculate the training loss corresponding to the training loss function based on the object spatial features extracted by the target local recognition model and the label information carried by the multimedia sample data. After the training loss is calculated, the training gradient can be calculated according to the chain rule. Where L is the training loss function, Represents gradient calculation, θ can be used to represent the local model parameters trained by the target client. The training loss function can be a classification function (for example, a softmax function), or a CosFace function (a loss function that maximizes the difference between classes and minimizes the difference within a class by normalizing and maximizing the cosine decision boundary) and an ArcFace function (a loss function that optimizes the distance between classes from the inverse cosine space, and makes the cosine value smaller in the monotonic interval by adding an m to the angle); and then the training learning rate corresponding to the target local recognition model can be obtained, and the local model parameters of the target local recognition model are updated according to the training learning rate and the training gradient. The updating method can be expressed as: where θ i,r,k It can be expressed as the local model parameters obtained by the k-th training of the target local recognition model (i.e., the a-th client among N clients, a is a positive integer less than or equal to N) in the r-th synchronization cycle, x a,r,k It can be expressed as the multimedia sample data used by the target local recognition model for the kth training in the rth synchronization cycle, η r It can be expressed as the training learning rate of the target local recognition model in the rth synchronization cycle, It can be expressed as the training gradient of the target local recognition model at the kth training in the rth synchronization cycle, θ a,r,k+1 It can be expressed as the local model parameters of the target local recognition model obtained by the k+1th training in the rth synchronization cycle, and the local model parameters θ of the k+1th training a,r,k+1 is the local model parameter θ based on the k-th training a,r,k , and the training gradient and training learning rate η r The training can be terminated when the number of training iterations reaches the set maximum number of iterations, which means that the training process of the target local recognition model is completed. Each time the local model parameters are updated, the number of training times of the target local recognition model can be increased once, that is, the target client can count the number of training times of the target local recognition model in real time. When the number of training times of the target local recognition model meets the synchronization cycle, that is, the number of training times of the target local recognition model is a multiple of the above-mentioned synchronization cycle K, the current local model parameters of the target local recognition model can be sent to the service device. For example, assuming that the synchronization cycle K is 100, when the number of training times of the target local recognition model is 100, the local model parameters obtained from the 100th training need to be sent to the service device for synchronization; when the number of training times of the target local recognition model is 200, the local model parameters obtained from the 200th training can be sent to the service device for synchronization; and so on, until the number of training times of the target local recognition model reaches the set maximum number of iterations, the training of the target local recognition model is terminated.

[0097] It can be understood that the above-mentioned N clients can all perform the above-mentioned operations, and when the number of training times of the local recognition model associated therewith meets the synchronization period, the local model parameters of the associated local recognition model can all be sent to the service device.

[0098] Step S102, obtaining local model parameters corresponding to N local recognition models respectively; the N local recognition models are obtained by independent training of different clients, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes objects with target object types, N is a positive integer greater than 1, and N is the number of clients.

[0099] Specifically, after N clients respectively send the local model parameters of their associated local recognition models to the service device, the service device can obtain the local model parameters corresponding to the N local recognition models. Among them, one local recognition model can correspond to one client, and N local recognition models can be independently trained in different clients. The multimedia sample data held by each client for training the local recognition model is not public. In the embodiment of the present application, considering the actual physical delay between different clients and the overall training efficiency, the synchronization period K (also referred to as the synchronization interval) can be set to hundreds or thousands of values. What is synchronized between the client and the service device is the local model parameters of the local recognition model, rather than the gradient at each training iteration, thereby improving the efficiency of federated training.

[0100] Step S103, obtain M weight combinations associated with the local model parameter set, and perform parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models; the local model parameter set includes local model parameters corresponding to N local recognition models, each weight combination includes training influence weights corresponding to N local model parameters, and M is a positive integer.

[0101] Specifically, after obtaining the local model parameters respectively sent by N clients, the service device can form a local model parameter set (such as the above) by the local model parameters uploaded by the N clients. Figure 2b In the corresponding embodiment, the local model parameter set 20g), the local model parameters sent by N clients can be expressed as: θ j , j∈{1, 2, …, N}, the above local model parameter set may include local model parameters corresponding to N clients respectively, or may include local model parameters corresponding to L clients among the N clients, where L is a positive integer less than N. The service device may use different parameter fusion methods to perform parameter fusion on the N local model parameters included in the local model parameter set. For example, a global weighted average scheme may be used to perform parameter fusion on the local model parameter set, or a voting scheme or an average scheme may be used to perform parameter fusion on the local model parameter set. The following takes the global weighted average scheme as an example to perform parameter fusion on the local model parameter set.

[0102] The service device can search for the optimal weight combination for the local model parameter set in the search space, and perform weighted averaging of the optimal weight combination and the local model parameter set to obtain the optimal global model. The service device can obtain M weight combinations associated with the local recognition model parameter set. If the local model parameter set includes local model parameters corresponding to N clients, then one weight combination may include training influence weights corresponding to the N local model parameters. For any weight combination i among the M weight combinations, the training influence weight contained in the weight combination i and the N local model parameters contained in the local model parameter set can be weighted averaged to obtain the fusion model parameters, and the recognition model carrying the fusion model parameters is determined as the alternative global model i associated with the above-mentioned weight combination i, where i is a positive integer less than or equal to M. In other words, the service device can randomly generate M weight combinations associated with the local recognition model parameter set during each synchronization process. The M weight combinations can be expressed as: At this time, any weight combination among the M weight combinations (i.e., the above weight combination i) can include the training influence weights corresponding to N local model parameters, and each weight combination The sum of all training influence weights contained in is 1, and the weight combination The training influence weight in can be expressed as w a , a∈{1, 2, …, N}, and the weight combination All training influence weights in satisfy the condition The weights can then be combined The N training influence weights contained in the local recognition model parameter set are weighted averaged to obtain the fusion model parameters. At this time, the recognition model carrying the fusion model parameters is determined as the candidate global model. (i.e. the alternative global model i mentioned above), that is Based on the above operation process, M candidate global models can be obtained, and the M candidate global models can be expressed as:

[0103] Among them, for any weight combination i among the above M weight combinations, its acquisition process may include: the service device may sample N values ​​within the target value range, determine the sum of the absolute values ​​corresponding to the N values ​​as the norm value, and determine the ratio between the N values ​​and the norm value as the weight combination i associated with the local model parameter set. Specifically, N values ​​may be sampled on a uniform distribution of [0, 1], and these N values ​​may be combined into an N-dimensional vector, and then each of the N values ​​may be divided by the L1 norm of the vector to ensure that the N values ​​contained in the vector are 1, and the weight combination i may be obtained. This process may be called a normalization operation; and so on, repeating the above operation M times may obtain M weight combinations.

[0104] Optionally, the model parameter fusion method can also adopt a partial local fusion solution. In the federated training scenario of N clients, each synchronization can randomly select the local model parameters of L clients for fusion, increasing the randomness in the model parameter fusion process, that is, each synchronization can randomly select the local model parameters corresponding to L clients from N clients, and the local model parameter set at this time may include the local model parameters corresponding to the L clients. If the local model parameter set includes L local model parameters, M weight combinations associated with the local recognition model parameter set can be obtained. A weight combination can include the training influence weights corresponding to the L local model parameters, that is, each weight combination at this time can refer to an L-dimensional vector, and the sum of the L training influence weights contained in each weight combination is 1. Among them, the method of obtaining the M weight combinations and the parameter fusion process of the L local model parameters are the same as the above operation, and will not be repeated here.

[0105] Step S104, obtaining evaluation indicators of the M candidate global models in the multimedia verification data set, and determining a target global model from the M candidate global models according to the evaluation indicators.

[0106] Specifically, the service device can obtain a multimedia verification data set containing positive sample pairs and negative sample pairs in the evaluation unit, wherein the positive sample pair refers to a multimedia sample data pair containing the same object (for example, a same-person sample pair), and the negative sample pair refers to a multimedia sample data pair containing different objects (for example, a non-same-person sample pair); obtain an alternative global model i (any alternative global model among the M alternative global models), input the positive sample pair into the alternative global model i, through which the alternative global model i can output the first object prediction result of the positive sample pair, input the negative sample pair into the alternative global model i, through which the alternative global model i can output the second object prediction result of the negative sample pair, and then determine the evaluation index of the alternative global model i in the multimedia verification data set according to the first object prediction result and the second object prediction result, and then determine the alternative global model corresponding to the largest evaluation index among the M alternative global models as the target global model. In other words, the service device can input each sample pair (positive sample pair and negative sample pair) contained in the multimedia verification data set into the alternative global model i in sequence, and the alternative global model i can output the prediction results corresponding to each positive sample pair, and the evaluation index of the alternative global model i in the multimedia verification data set can be determined according to the prediction results; for the above-mentioned M alternative global models, the above-mentioned method can be used to determine the evaluation index of each alternative global model in the multimedia verification data set, and the alternative global model corresponding to the largest evaluation index is determined as the target global model.

[0107] Among them, when the evaluation indicator is accuracy, the process of determining the evaluation indicator may include: the service device can count the first correct prediction number of the alternative global model i in the positive sample pair according to the prediction result of the first object, and count the second correct prediction number of the alternative global model i in the negative sample pair according to the prediction result of the second object; determine the sum of the first correct prediction number and the second correct prediction result as the total number of predicted correct sample pairs of the alternative global model i in the multimedia verification data set; obtain the total number of sample pairs corresponding to the positive sample pairs and the negative sample pairs in the multimedia verification data set, and determine the evaluation indicator corresponding to the alternative global model i according to the ratio between the total number of predicted correct sample pairs and the total number of sample pairs. In other words, according to the prediction results of the alternative global model i in the multimedia verification data set, the first correct prediction number of the alternative global model i in the positive sample pairs (which may refer to the number of correct predictions and positive sample pairs themselves, also known as true positive, TP), the first incorrect prediction number in the positive sample pairs (which may refer to the number of incorrect predictions and positive sample pairs themselves, also known as false negative, FN), the second correct prediction number in the negative sample pairs (which may refer to the number of correct predictions and negative sample pairs themselves, also known as true negative, TN), and the second incorrect prediction number in the negative sample pairs (which may refer to the number of incorrect predictions and negative sample pairs themselves, also known as false positive, FP) can be counted. The evaluation index (accuracy) of the alternative global model i can be expressed as: acc = (TP+TN) / (TP+FN+TN+FP), where TP+FN+TN+FP can be expressed as the total number of sample pairs contained in the multimedia verification data set, and TP+TN can be expressed as the number of sample pairs predicted correctly by the alternative global model i in the multimedia verification data set.

[0108] Optionally, when the number of multimedia verification data sets is P, the above-mentioned P multimedia verification data sets may include multimedia verification data set j, P is a positive integer, and j is a positive integer less than or equal to P. At this time, the process of determining the evaluation index may include: the service device may determine the ratio between the total number of predicted correct sample pairs of the alternative global model i in the multimedia verification data set j (TP+TN) and the total number of sample pairs corresponding to the multimedia verification data set j (TP+FN+TN+FP) as the prediction accuracy of the alternative global model i in the multimedia verification data set j, that is, acc=(TP+TN) / (TP+FN+TN+FP); and then obtain the prediction accuracy of the alternative global model i in the P multimedia verification data sets respectively, and calculate the average accuracy mean corresponding to the P prediction accuracy rates, and the standard deviation value std corresponding to the P prediction accuracy rates; according to the average accuracy rate and the standard deviation value, the evaluation index corresponding to the alternative global model i can be determined Among them, the evaluation indicators The calculation method can be expressed as:

[0109]

[0110] Among them, in the above formula (1), It can be expressed as the unified prediction accuracy (i.e., the above evaluation index) obtained by normalizing the prediction results of the candidate global model i in P multimedia verification data sets; acc j It can be expressed as the prediction accuracy of the candidate global model i in the multimedia verification data set j; mean can be expressed as the average accuracy corresponding to the prediction accuracy in P multimedia verification data sets; std can be expressed as the standard deviation value corresponding to the prediction accuracy in P multimedia verification data sets. It should be noted that different normalization methods can be used to process the above-mentioned M weight combinations and the above-mentioned evaluation indicators. The normalization method may include but is not limited to: L norm, M norm; of course, the embodiment of the present application may not perform normalization operations.

[0111] Optional, alternative global model The evaluation value (such as the above prediction accuracy) in the multimedia verification dataset j can be expressed as S ,j , j∈{1, 2, …, P}, and then the initial evaluation index S ,j Normalization is performed to eliminate the influence of differences between multimedia verification datasets. For example, the initial evaluation index S ,j Perform local normalization (LocalNorm):

[0112]

[0113] Among them, S in the above formula (2) j ′ It can be expressed as an evaluation indicator after local normalization, σ can be expressed as an activation function, and ε can be a hyperparameter in the training process, which can be set according to actual needs.

[0114] Optionally, the initial evaluation index S j ′ Perform moving normalization (Moving Norm):

[0115]

[0116] Among them, S in the above formulas (3) to (5) is j' can be expressed as the evaluation index after moving normalization, γ can be expressed as the normalization parameter, μ can be expressed as the moving average, v can be expressed as the moving variance, ε can be a hyperparameter in the training process, μ last can be the moving average corresponding to the latest training, v last It can be the moving variance corresponding to the latest training. Here, the hyperparameter ε in formula (5) can be the same as or different from the hyperparameter ε in the above formula (2).

[0117] Furthermore, after obtaining the evaluation indicators corresponding to the M candidate global models respectively through the above formula (2), or the above formulas (3) to (5), the weight combination corresponding to the largest evaluation indicator among the M evaluation indicators can be used as the optimal weight combination, and the optimal weight combination is applied to the local recognition model corresponding to each client:

[0118]

[0119] Among them, S in the above formula (6) and formula (7) j ′ ,a It can be expressed as the evaluation index of the local recognition model corresponding to the a-th client in the multimedia verification dataset j, It can be expressed as applying the optimal weight combination to the local recognition model of the a-th client, a∈{1, 2, …, N}, It can be expressed as the weight combination of the ath local recognition model (the local recognition model corresponding to the ath client) determined based on the above optimal weight combination; w last It can be the weight corresponding to the ath local recognition model in the latest training, It can be a hyperparameter in the training process. It can be set according to actual needs.

[0120] Optionally, when the evaluation indicator is the recall rate (TPR) corresponding to a fixed false acceptance rate (False Acceptance Rate, FAR), the service device can obtain the false acceptance rate of the alternative global model i included in the M alternative global models in the multimedia verification data set, and determine a similarity threshold in the similarity corresponding to the negative sample pairs, wherein the similarity threshold is determined by the number of negative sample pairs and the false acceptance rate; and then the similarity corresponding to the positive sample pairs can be obtained, and the first sample pair whose similarity is greater than the similarity threshold is obtained in the positive sample pairs, and the ratio between the number of the first sample pairs and the number of positive sample pairs is determined as the evaluation indicator corresponding to the alternative global model i. Among them, the process of determining the false acceptance rate may include: obtaining the number of false predictions of the alternative global model i included in the M alternative global models in the negative sample pairs (that is, the second number of false predictions mentioned above, FP); determining the ratio between the number of false predictions and the number of negative sample pairs (the sum of the second number of false predictions and the second number of correct predictions, that is, FP+TN) as the false acceptance rate of the alternative global model i in the multimedia verification data set, and the false acceptance rate can be expressed as: FAR=FP / (FP+TN).

[0121] Among them, if the multimedia verification data set includes N1 positive sample pairs and N2 negative sample pairs, the recall rate TPR calculation method under FAR=1e-3 is: obtain the similarity between the N2 negative sample pairs (for example, cosine similarity) and the similarity between the N1 positive sample pairs, and then the similarities of the N2 negative sample pairs can be sorted in descending order, and the topx=int(N2*FAR)th similarity is determined as the similarity threshold, and the positive sample pair with a similarity greater than the similarity threshold among the N1 positive sample pairs is determined as the first sample pair, and the ratio between the number of first sample pairs and the number of positive sample pairs N1 is determined as the recall rate TPR (that is, the above-mentioned evaluation index), wherein int() is a rounding function, and FAR=1e-3 can refer to a value preset based on actual needs.

[0122] It is understandable that the service device can determine the candidate global model corresponding to the largest evaluation index as the target global model based on the evaluation indexes of the M candidate global models in the multimedia verification data set, and determine the weight combination corresponding to the target global model as the optimal weight combination among the M weight combinations. In different synchronization processes, the optimal weight combination is different, such as the optimal weight combination determined in the first synchronization process and the second synchronization process is different. Please refer to Figure 4 , Figure 4 Schematic diagram of determining a target global model provided by an embodiment of the present application. Figure 4As shown, when the number of clients N=3 (the multimedia sample data held by client 1, the multimedia sample data held by client 2, and the multimedia sample data held by client 3 are needed in the federated training process), the process of determining the weight combination is described by taking the accuracy as an evaluation indicator. When the local model parameters received by the service device are the local model parameters of the 12800th training iteration, the color depth in area 30a is used to represent the accuracy value of the alternative global model corresponding to the 12800th training iteration on the multimedia verification data set, and the bar graph area 30b can be used to explain the relationship between the color and the accuracy value in area 30a, and each position in area 30a can represent a weight combination; when the local model parameters received by the service device are the local model parameters of the 256000th training iteration, the color depth in area 30c is used to represent the accuracy value of the alternative global model corresponding to the 256000th training iteration on the multimedia verification data set, and the bar graph area 30d can be used to explain the relationship between the color and the accuracy value in area 30c, and each position in area 30c can also represent a weight combination. Areas 30a and 30c indicate that at different training stages, the weight combination of the best result on the multimedia verification data set is at different positions and changes dynamically. Figure 4 As shown, the optimal weight combination at the 12800th training iteration is: optimal weight combination 1, and the optimal weight combination at the 256000th training iteration is: optimal weight combination 2.

[0123] Optionally, if the above-mentioned target global model is generated based on the N local model parameters corresponding to the r-th synchronization period, r is a positive integer, the service device can determine the target global model corresponding to the r-th synchronization period in the alternative global models according to the evaluation index, and obtain the historical global model corresponding to the (r-1)-th synchronization period, wherein the historical global model is generated based on the local model parameters uploaded by N clients in the (r-1)-th synchronization period; and then the training learning rate of the N local recognition models in the r-th synchronization period can be obtained, and the model parameter difference between the target global model and the historical global model can be obtained; the ratio between the model parameter difference and the training learning rate is determined as the federal momentum, and the federal momentum is sent to the N clients; the federal momentum together with the target global model is used to instruct the N clients to update the parameters of the associated local recognition models, and the federal momentum is used to indicate the training direction of the N local recognition models in their respective clients. For example, if the target global model corresponding to the r-th synchronization period is expressed as The historical global model corresponding to the (r-1)th synchronization period The training learning rate in the rth synchronization cycle can be expressed as η r , the federation momentum at this time can be expressed as ( It can be expressed as the federation momentum corresponding to the rth synchronization period, It can be expressed as the above model parameter difference), where the training learning rate η r It can be a fixed value, or it can be adaptively changed. For example, the training learning rate of all multimedia sample data held by any client during the first complete training can be set to 0.1, and the training learning rate of all multimedia sample data held during the 10th complete training can be set to 0.02, etc. It should be noted that the federation momentum in the first synchronization cycle can be expressed as:

[0124] Step S105, returning the target global model.

[0125] Specifically, the service device can return the above-mentioned target global model to N clients. After receiving the target global model returned by the service device, any client can update the parameters of the local recognition model according to the target global model and continue training based on the updated local model parameters.

[0126] Optionally, when the service device generates federation momentum When the service device can combine the target global model and the federated momentum Returned to N clients together, any client receives the target global model and federated momentum returned by the service device After that, we can use the target global model and federated momentum The parameters of the local recognition model are updated, and training is continued based on the updated local model parameters.

[0127] Step S106, updating the parameters of the target local recognition model according to the target global model, and determining the target local recognition model after parameter update as the object recognition model; the object recognition model is used to recognize objects of the target object type contained in the multimedia data.

[0128] Specifically, after receiving the target global model returned by the service device, the target client can update the parameters of its own target local recognition model according to the target global model, and continue to perform local training on the target local recognition model until the number of training times of the target local recognition model reaches the training termination conditions (including training convergence conditions, maximum number of iterations, etc.), thereby completing the training process of the target local recognition model and obtaining a trained object recognition model, which is used to identify objects of the target object type contained in multimedia data.

[0129] Optionally, the target client receives the target global model and federated momentum returned by the service device. After that, the training gradient and federated momentum in the local training of the target client can be Combined, the parameters of the target local recognition model are updated, such as θ i It can be expressed as the local model parameter obtained by the target client i in the rth synchronization cycle, g can be expressed as the training gradient corresponding to the rth synchronization cycle, and K can be expressed as the number of training times corresponding to a synchronization cycle.

[0130] Please also see Figure 5 , Figure 5 This is a flow chart of a federated model training method provided in an embodiment of the present application. Figure 5 As shown, taking the multimedia sample data as a face image as an example, the implementation process of the federated model training method is specifically described. The federated model training method can be implemented through the following steps S11 to S22.

[0131] In step S11, the client can read the local training data, that is, it can obtain the face sample data held by the client (that is, the above-mentioned multimedia sample data), and then continue to execute step S12 to obtain the face recognition model for initialization processing (the above-mentioned local recognition model), and perform local training on the face recognition model through the face sample data, that is, execute step S13, that is, calculate the training loss and training gradient of the face recognition model, and count the number of training times of the face recognition model in real time. The client can continue to execute step S14 to determine whether the number of training times reaches the synchronization cycle (the above-mentioned synchronization cycle K). If the number of training times of the face recognition model reaches the synchronization cycle, continue to execute step S15 to upload the client model parameters (the current model parameters of the face recognition model, that is, the above-mentioned local model parameters) to the service device; if the number of training times of the face recognition model does not reach the synchronization cycle, continue to execute step S21 to determine whether the face recognition model meets the training termination conditions. If the face recognition model meets the training termination conditions, it means that the training of the face recognition model is completed; if the face recognition model does not meet the training termination conditions, continue to execute step S22 to update the local model parameters of the face recognition model.

[0132] When the number of trainings corresponding to the face recognition model in the local training of all clients (i.e. the N clients mentioned above) reaches the synchronization period, the local model parameters can be uploaded to the service device. The service device can receive the local model parameters uploaded by all clients, and continue to execute step S16 to generate models corresponding to different fusion schemes in the search space, such as obtaining M weight combinations, and weighted averaging each weight combination with the received local model parameters to obtain M candidate global models. The specific implementation process of the M weight combinations can refer to the above step S103; then, steps S17 and S18 can be continued to be executed to read the verification set data (i.e. the above multimedia verification data set), and obtain the evaluation indicators corresponding to each candidate global model in the verification set, and then continue to execute steps S19-S20, select the fusion scheme corresponding to the optimal evaluation indicator for fusion (i.e. the alternative global model corresponding to the optimal evaluation indicator) to obtain the target global model, and send the target global model to each client. When the client receives the target global model and the face recognition model does not meet the training termination condition, it continues to execute step S22 to update the parameters of the face recognition model according to the target global model. It is understandable that the above steps S12 to 22 can be repeatedly executed until the face recognition model meets the training termination condition and the training of the face recognition model is completed.

[0133] Please also see Figure 6 , Figure 6 Schematic diagram of a weight combination in a multimedia verification data set provided by an embodiment of the present application. Figure 6 As shown in FIG. 1 , when the number of clients N=3 (client 1, client 2, and client 3), the multimedia sample data used in the federated training is the face data held by the above-mentioned client 1, client 2, and client 3, respectively. In the process of the above-mentioned three clients performing local training based on their own face data, it is assumed that the epochs in the federated training=26, where epoch is used to represent the number of complete trainings performed by each client on the face data held; Figure 6 The horizontal axis in the coordinate graph shown is epochs, and the vertical axis can be the training influence weights corresponding to different epochs. Obviously, with the increase of epochs, the training influence weights corresponding to the local model parameters trained by the three clients are more concentrated, that is, the later the training stage, the closer the influence training weights corresponding to each client are.

[0134] Optionally, for each object recognition model trained locally by each client, when the multimedia data includes a face image to be recognized and the target object type includes a face type, the target client can obtain the face image to be recognized, input the face image to be recognized into the object recognition model, and obtain the face spatial features corresponding to the face image to be recognized in the object recognition model; then the face classification result corresponding to the face image to be recognized can be determined according to the face spatial features; the face classification result is used to characterize the identity authentication result of the object with face type contained in the face image to be recognized. In other words, the object recognition model can be used in any scene for face recognition, such as user identity authentication scene, missing person tracking scene, business processing scene, etc.; wherein, in the user identity authentication scene and business processing scene, the object recognition model can be used to recognize the user face image provided by the user in the identity authentication scene to confirm the identity authenticity of the user face image; in the missing person tracking scene, the photo of the missing person before the missing person can be recognized and compared with the existing household registration photo to obtain the suspected user of the missing person.

[0135] Please also see Figure 7 , Figure 7 Schematic diagram of a user identity authentication scenario provided by an embodiment of the present application. Figure 7 As shown, when user A wants to handle business in the client 1 installed in the user terminal 40a, user A needs to perform identity authentication in the client 1. When user A initiates an identity authentication request in the client 1 installed in the user terminal 40a, a face verification box 40b can be displayed in the client 1. User A can align his face with the face verification box 40b in the user terminal 40a, and follow the instructions to perform corresponding actions (for example, shaking his head, nodding, blinking, etc.), and the user terminal 40a can collect the face image 40c to be recognized in the face verification box 40b in real time, and input the face image 40c to be recognized collected in real time into the object recognition model 40d, and perform feature extraction on the face image 40c to be recognized in the object recognition model 40d to obtain the face recognition result corresponding to the face image 40c to be recognized. At the same time, the client 1 can obtain the ID image 40e uploaded in advance by user A from the existing face image database, and compare the ID image 40e with the face recognition result output by the object recognition model 40d. If the ID image 40e is the same as the face recognition result, it can be determined that the identity authentication of user A has passed, and the identity authentication result is returned to the client 1 of the user terminal 40a; if the ID image 40e is different from the face recognition result, it can be determined that the identity authentication of user A has failed, and the identity authentication result is returned to the client 1 of the user terminal 40a, reminding user A to re-authenticate.

[0136] In an embodiment of the present application, local model parameters of local recognition models uploaded by N clients can be obtained, wherein the N local recognition models are independently trained by different clients, and the multimedia sample data held by each client for training the associated local training model all contain objects of the target object type, and N is a positive integer greater than 1; and M weight combinations (M is a positive integer) for the N local model parameters are obtained, and the parameters of the N local model parameters are fused through each weight combination to obtain M candidate global models, and then the optimal target global model is selected from the M candidate global models through the evaluation indicators of the M candidate global models in the multimedia verification data set, that is, the optimal target global model is selected through the evaluation indicators on the multimedia verification data set, which can improve the fusion effectiveness between the N local model parameters, and the N clients continue to update the parameters of their respective local recognition models based on the target global model, which can improve the generalization effect of the object recognition model; the embodiment of the present application can be applied to cross-departmental, cross-enterprise, and even cross-regional business data, and can improve the recognition effect of the object recognition model while ensuring data privacy and security.

[0137] See also Figure 8 , Figure 8 Schematic diagram of a data processing device provided in an embodiment of the present application. Figure 8 As shown, the above data processing device can be applied to the above Figure 1 The client on any user terminal in the user terminal cluster shown may be a computer program (including program code) in a computer device. The data processing device 1 may include: a parameter acquisition module 11, a parameter fusion module 12, and a model determination module 13;

[0138] A parameter acquisition module 11 is used to obtain local model parameters corresponding to N local recognition models respectively; the N local recognition models are obtained by independent training of different clients, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes objects with target object types, N is a positive integer greater than 1, and N is the number of clients;

[0139] The parameter fusion module 12 is used to obtain M weight combinations associated with the local model parameter set, and perform parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models; the local model parameter set includes local model parameters corresponding to N local recognition models, each weight combination includes training influence weights corresponding to N local model parameters, and M is a positive integer;

[0140] The model determination module 13 is used to obtain evaluation indicators of M candidate global models in the multimedia verification data set, determine the target global model among the M candidate global models according to the evaluation indicators, and transmit the target global model to N clients, so that the N clients update the parameters of the associated local recognition models according to the target global model to obtain an object recognition model; the object recognition model is used to identify objects of the target object type contained in the multimedia data.

[0141] The specific functional implementation of the parameter acquisition module 11, the parameter fusion module 12, and the model determination module 13 can be found in the above Figure 3 Steps S102 to S105 in the corresponding embodiment will not be described in detail here.

[0142] In some feasible implementations, the parameter fusion module 12 may include: a weight combination acquisition unit 121, a weighted average unit 122;

[0143] The weight combination acquisition unit 121 is used to acquire M weight combinations associated with the local model parameter set; the M weight combinations include a weight combination i, where i is a positive integer less than or equal to M;

[0144] The weighted averaging unit 122 is used to perform weighted averaging on the training influence weights included in the weight combination i and the N local model parameters included in the local model parameter set to obtain fusion model parameters, and determine the recognition model carrying the fusion model parameters as the candidate global model i associated with the weight combination i;

[0145] The model determination module 13 is specifically used for:

[0146] The evaluation indicators of M candidate global models in the multimedia verification data set are obtained respectively, and among the M candidate global models, the candidate global model corresponding to the largest evaluation indicator is determined as the target global model.

[0147] The specific functional implementation of the weight combination acquisition unit 121 and the weighted average unit 122 can be found in the above Figure 3 Step S103 in the corresponding embodiment will not be described in detail here.

[0148] In some feasible implementations, the weight combination acquisition unit 121 may include: a norm value determination subunit 1211, a weight determination subunit 1212;

[0149] The norm value determination subunit 1211 is used to sample N values ​​within the target value range and determine the sum of the absolute values ​​corresponding to the N values ​​as the norm value;

[0150] The weight determination subunit 1212 is used to determine the ratios between the N values ​​and the norm values ​​as a weight combination i associated with the local model parameter set.

[0151] The specific functional implementation of the norm value determination subunit 1211 and the weight determination subunit 1212 can be found in the above Figure 3 Step S103 in the corresponding embodiment will not be described in detail here.

[0152] In some feasible implementations, the model determination module 13 may include: a verification data set acquisition unit 131, a first prediction unit 132, a second prediction unit 133, and a first evaluation index determination unit 134;

[0153] The verification data set acquisition unit 131 is used to acquire a multimedia verification data set including positive sample pairs and negative sample pairs; the positive sample pairs refer to multimedia sample data pairs including the same object, and the negative sample pairs refer to multimedia sample data pairs including different objects;

[0154] The first prediction unit 132 is used to obtain an alternative global model i from the M alternative global models, input the positive sample pair into the alternative global model i, and output the first object prediction result of the positive sample pair through the alternative global model i; i is a positive integer less than or equal to M;

[0155] The second prediction unit 133 is used to input the negative sample pair into the candidate global model i, and output the second object prediction result of the negative sample pair through the candidate global model i;

[0156] The first evaluation index determination unit 134 is used to determine the evaluation index of the candidate global model i in the multimedia verification data set according to the first object prediction result and the second object prediction result.

[0157] The first evaluation index determination unit 134 may include: a prediction result statistics subunit 1341, a correct sample pair total statistics subunit 1342, and an evaluation index calculation subunit 1343;

[0158] The prediction result statistics subunit 1341 is used to count the number of first correct predictions of the candidate global model i in the positive sample pair according to the first object prediction result, and count the number of second correct predictions of the candidate global model i in the negative sample pair according to the second object prediction result;

[0159] A total number of correct sample pairs statistics subunit 1342 is used to determine the sum of the first correct prediction number and the second correct prediction result as the total number of correct sample pairs predicted by the candidate global model i in the multimedia verification data set;

[0160] The evaluation index calculation subunit 1343 is used to obtain the total number of sample pairs corresponding to the positive sample pairs and the negative sample pairs in the multimedia verification data set, and determine the evaluation index corresponding to the alternative global model i according to the ratio between the total number of predicted correct sample pairs and the total number of sample pairs.

[0161] The number of multimedia verification data sets is P, the P multimedia verification data sets include multimedia verification data set j, P is a positive integer, and j is a positive integer less than or equal to P;

[0162] The evaluation index calculation subunit 1343 is specifically used for:

[0163] The ratio between the total number of correctly predicted sample pairs of the candidate global model i in the multimedia verification data set j and the total number of sample pairs corresponding to the multimedia verification data set j is determined as the prediction accuracy of the candidate global model i in the multimedia verification data set j;

[0164] Obtain the prediction accuracy of the candidate global model i in P multimedia verification data sets, and calculate the average accuracy corresponding to the P prediction accuracy rates and the standard deviation values ​​corresponding to the P prediction accuracy rates;

[0165] According to the average accuracy and standard deviation, the evaluation index corresponding to the candidate global model i is determined.

[0166] The specific functional implementation of the verification data set acquisition unit 131, the first prediction unit 132, the second prediction unit 133, and the first evaluation index determination unit 134 can be found in the above Figure 3 Step S104 in the corresponding embodiment will not be described in detail here.

[0167] In some feasible implementations, the model determination module 13 may include: a verification data set acquisition unit 131, a similarity threshold determination unit 135, and a second evaluation index determination unit 136;

[0168] The verification data set acquisition unit 131 is used to acquire a multimedia verification data set including positive sample pairs and negative sample pairs; the positive sample pairs refer to multimedia sample data pairs including the same object, and the negative sample pairs refer to multimedia sample data pairs including different objects;

[0169] A similarity threshold determination unit 135 is used to obtain the false acceptance rate of the candidate global model i included in the M candidate global models in the multimedia verification data set, and determine the similarity threshold in the similarity corresponding to the negative sample pair; the similarity threshold is determined by the number of negative sample pairs and the false acceptance rate, and i is a positive integer less than or equal to M;

[0170] The second evaluation index determination unit 136 is used to obtain the similarity corresponding to the positive sample pairs, obtain the first sample pairs whose similarity is greater than the similarity threshold in the positive sample pairs, and determine the ratio between the number of the first sample pairs and the number of the positive sample pairs as the evaluation index corresponding to the alternative global model i.

[0171] The similarity threshold determination unit 135 may include: an error prediction number acquisition subunit 1351, an error acceptance rate determination subunit 1352;

[0172] The wrong prediction number obtaining subunit 1351 is used to obtain the wrong prediction number of the candidate global model i included in the M candidate global models in the negative sample pair;

[0173] The false acceptance rate determination subunit 1352 is used to determine the ratio between the number of false predictions and the number of negative sample pairs as the false acceptance rate of the candidate global model i in the multimedia verification data set.

[0174] The specific functional implementation of the verification data set acquisition unit 131, the similarity threshold determination unit 135, and the second evaluation index determination unit 136 can be found in the above Figure 3 Step S104 in the corresponding embodiment is not described here. When the first prediction unit 132, the second prediction unit 133, and the first evaluation index determination unit 134 are executing the corresponding steps, the similarity threshold determination unit 135 and the second evaluation index determination unit 136 are suspended from executing the operation; when the similarity threshold determination unit 135 and the second evaluation index determination unit 136 are executing the corresponding operation, the first prediction unit 132, the second prediction unit 133, and the first evaluation index determination unit 134 are suspended from executing the corresponding operation.

[0175] In some feasible implementations, the target global model is generated based on N local model parameters corresponding to the rth synchronization period, where r is a positive integer;

[0176] The data processing device 1 may further include: a historical global model acquisition module 14, a model parameter difference acquisition module 15, and a federated momentum determination module 16;

[0177] The historical global model acquisition module 14 is used to determine the target global model corresponding to the rth synchronization period from the candidate global models according to the evaluation index, and to acquire the historical global model corresponding to the (r-1)th synchronization period; the historical global model is generated based on the local model parameters uploaded by N clients in the (r-1)th synchronization period respectively;

[0178] A model parameter difference acquisition module 15 is used to obtain the training learning rates of N local recognition models in the rth synchronization cycle, and obtain the model parameter difference between the target global model and the historical global model;

[0179] The federated momentum determination module 16 is used to determine the ratio between the model parameter difference and the training learning rate as the federated momentum, and send the federated momentum to N clients; the federated momentum together with the target global model is used to instruct the N clients to update the parameters of the associated local recognition models, and the federated momentum is used to indicate the training directions of the N local recognition models in their respective clients.

[0180] The specific functional implementation of the historical global model acquisition module 14, the model parameter difference acquisition module 15, and the federated momentum determination module 16 can be found in the above Figure 3 Step S104 in the corresponding embodiment will not be described in detail here.

[0181] In an embodiment of the present application, local model parameters of local recognition models uploaded by N clients can be obtained, wherein the N local recognition models are independently trained by different clients, and the multimedia sample data held by each client for training the associated local training model all contain objects of the target object type, and N is a positive integer greater than 1; and M weight combinations (M is a positive integer) for the N local model parameters are obtained, and the parameters of the N local model parameters are fused through each weight combination to obtain M candidate global models, and then the optimal target global model is selected from the M candidate global models through the evaluation indicators of the M candidate global models in the multimedia verification data set, that is, the optimal target global model is selected through the evaluation indicators on the multimedia verification data set, which can improve the fusion effectiveness between the N local model parameters, and the N clients continue to update the parameters of their respective local recognition models based on the target global model, which can improve the generalization effect of the object recognition model; the embodiment of the present application can be applied to cross-departmental, cross-enterprise, and even cross-regional business data, and can improve the recognition effect of the object recognition model while ensuring data privacy and security.

[0182] See also Fig. 9 , Fig. 9 Schematic diagram of a data processing device provided in an embodiment of the present application. Fig. 9 As shown, the above data processing device can be applied to the above Figure 1 The corresponding service device in the embodiment (for example, the server 10d). The data processing device 2 may include: a model parameter uploading module 21, a target global model receiving module 22;

[0183] The model parameter uploading module 21 is used to upload the local model parameters corresponding to the target local recognition model to the service device when the number of training times of the target local recognition model meets the synchronization period, so that the service device performs parameter fusion on the local model parameters uploaded by N clients to obtain the target global model; the local model parameters uploaded by N clients include the local model parameters corresponding to the target local recognition model, the target global model is determined by the evaluation indicators of M candidate global models in the multimedia verification data set, the M candidate global models are determined by M weight combinations associated with the local model parameter set and the local model parameter set, the local model parameter set includes the local model parameters uploaded by N clients, each weight combination includes the training influence weights associated with N clients, N is a positive integer greater than 1, and M is a positive integer;

[0184] The target global model receiving module 22 is used to receive the target global model returned by the service device, update the parameters of the target local recognition model according to the target global model, and determine the target local recognition model after the parameter update as the object recognition model; the object recognition model is used to identify objects with the target object type contained in the multimedia data.

[0185] The specific functional implementation of the model parameter uploading module 21 and the target global model receiving module 22 can be found in the above Figure 3 Step S101, step S105-step S106 in the corresponding embodiment will not be described in detail here.

[0186] In some feasible implementations, the data processing device 2 may further include: a feature extraction module 23, a loss function determination module 24, and a training times statistics module 25;

[0187] The feature extraction module 23 is used to obtain multimedia sample data, input the multimedia sample data into the target local recognition model, and output the object space features corresponding to the multimedia sample data through the target local recognition model;

[0188] A loss function determination module 24, used to determine a training loss function corresponding to a target local recognition model according to the object spatial features and label information corresponding to the multimedia sample data;

[0189] The training times statistics module 25 is used to determine the training gradient of the target local recognition model according to the training loss function, update the parameters of the target local recognition model according to the training gradient and the training learning rate corresponding to the target local recognition model, and count the training times corresponding to the target local recognition model.

[0190] The specific functional implementation of the feature extraction module 23, the loss function determination module 24, and the training times statistics module 25 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0191] In some feasible implementations, the multimedia data includes a face image to be recognized, and the target object type includes a face type;

[0192] The data processing device 2 may further include: a face feature extraction module 26, a face classification module 27;

[0193] The face feature extraction module 26 is used to obtain a face image to be identified, input the face image to be identified into an object recognition model, and obtain face spatial features corresponding to the face image to be identified in the object recognition model;

[0194] The face classification module 27 is used to determine the face classification result corresponding to the face image to be identified according to the face spatial features; the face classification result is used to represent the identity authentication result of the object with face type contained in the face image to be identified.

[0195] The specific functional implementation of the face feature extraction module 26 and the face classification module 27 can be found in the above Figure 3 Step S106 in the corresponding embodiment will not be described in detail here.

[0196] In an embodiment of the present application, local model parameters of local recognition models uploaded by N clients can be obtained, wherein the N local recognition models are independently trained by different clients, and the multimedia sample data held by each client for training the associated local training model all contain objects of the target object type, and N is a positive integer greater than 1; and M weight combinations (M is a positive integer) for the N local model parameters are obtained, and the parameters of the N local model parameters are fused through each weight combination to obtain M candidate global models, and then the optimal target global model is selected from the M candidate global models through the evaluation indicators of the M candidate global models in the multimedia verification data set, that is, the optimal target global model is selected through the evaluation indicators on the multimedia verification data set, which can improve the fusion effectiveness between the N local model parameters, and the N clients continue to update the parameters of their respective local recognition models based on the target global model, which can improve the generalization effect of the object recognition model; the embodiment of the present application can be applied to cross-departmental, cross-enterprise, and even cross-regional business data, and can improve the recognition effect of the object recognition model while ensuring data privacy and security.

[0197] See also Fig.10 , Fig.10 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.10As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Fig.10 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0198] In such Fig.10 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0199] Obtain local model parameters corresponding to N local recognition models respectively; the N local recognition models are obtained by independently training different clients, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes an object with a target object type, N is a positive integer greater than 1, and N is the number of clients;

[0200] Obtain M weight combinations associated with the local model parameter set, perform parameter fusion on the local model parameter set according to each weight combination, and obtain M candidate global models; the local model parameter set includes local model parameters corresponding to N local recognition models, each weight combination includes training influence weights corresponding to N local model parameters, and M is a positive integer;

[0201] Obtain evaluation indicators of M candidate global models in a multimedia verification data set, determine a target global model among the M candidate global models according to the evaluation indicators, and transmit the target global model to N clients, so that the N clients update parameters of the associated local recognition models according to the target global model to obtain an object recognition model; the object recognition model is used to identify objects of the target object type contained in the multimedia data.

[0202] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 The description of the data processing method in the corresponding embodiment can also be performed as described above. Figure 8 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0203] See also Fig.11 , Fig.11 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.11 As shown, the computer device 2000 may include: a processor 2001, a network interface 2004 and a memory 2005. In addition, the above-mentioned computer device 2000 may also include: a user interface 2003, and at least one communication bus 2002. Among them, the communication bus 2002 is used to realize the connection and communication between these components. Among them, the user interface 2003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 2003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 2004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 2005 may also be at least one storage device located away from the aforementioned processor 2001. As Fig.11 As shown, the memory 2005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0204] In such Fig.11 In the computer device 2000 shown, the network interface 2004 can provide a network communication function; the user interface 2003 is mainly used to provide an input interface for the user; and the processor 2001 can be used to call the device control application stored in the memory 2005 to achieve:

[0205] When the number of training times of the target local recognition model meets the synchronization cycle, the local model parameters corresponding to the target local recognition model are uploaded to the service device, so that the service device performs parameter fusion on the local model parameters uploaded by N clients respectively to obtain the target global model; the local model parameters uploaded by N clients respectively include the local model parameters corresponding to the target local recognition model, the target global model is determined by the evaluation indicators of M candidate global models in the multimedia verification data set respectively, the M candidate global models are determined by M weight combinations associated with the local model parameter set and the local model parameter set, the local model parameter set includes the local model parameters uploaded by N clients respectively, each weight combination includes the training influence weights associated with N clients respectively, N is a positive integer greater than 1, and M is a positive integer;

[0206] The target global model returned by the receiving service device is updated with parameters of the target local recognition model according to the target global model, and the target local recognition model after parameter update is determined as the object recognition model; the object recognition model is used to recognize objects of the target object type contained in the multimedia data.

[0207] It should be understood that the computer device 2000 described in the embodiment of the present application can execute the above Figure 3 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig. 9 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0208] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores the computer program executed by the data processing device 1 mentioned above and the computer program executed by the data processing device 2, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 The description of the data processing method in the corresponding embodiment will therefore not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions may be deployed on a computing device for execution, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0209] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 The description of the data processing method in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0210] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0211] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0212] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0213] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0214] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Obtain local model parameters corresponding to N local recognition models, respectively; the N local recognition models are obtained by independent training of different clients, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes an object with a target object type, N is a positive integer greater than 1, and N is the number of clients; Acquire M weight combinations associated with a local model parameter set, and perform parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models; the local model parameter set includes local model parameters corresponding to the N local recognition models, each weight combination includes training influence weights corresponding to the N local model parameters, and M is a positive integer; Obtaining evaluation indicators of the M candidate global models in the multimedia verification data set, respectively, determining a target global model from the M candidate global models according to the evaluation indicators, and transmitting the target global model to N clients, so that the N clients respectively update parameters of the associated local recognition models according to the target global model to obtain an object recognition model; the object recognition model is used to identify an object of the target object type contained in the multimedia data; The step of obtaining evaluation indicators of the M candidate global models in the multimedia verification dataset includes: Acquire a multimedia verification data set including positive sample pairs and negative sample pairs; the positive sample pairs refer to multimedia sample data pairs including the same object, and the negative sample pairs refer to multimedia sample data pairs including different objects; Obtain an alternative global model i from the M alternative global models, input the positive sample pair into the alternative global model i, and output a first object prediction result of the positive sample pair through the alternative global model i; i is a positive integer less than or equal to M; Inputting the negative sample pair into the candidate global model i, and outputting the second object prediction result of the negative sample pair through the candidate global model i; According to the first object prediction result and the second object prediction result, an evaluation index of the candidate global model i in the multimedia verification data set is determined.

2. The method according to claim 1, characterized in that The acquiring of M weight combinations associated with the local model parameter set, and performing parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models, includes: Acquire M weight combinations associated with the local model parameter set; the M weight combinations include a weight combination i, where i is a positive integer less than or equal to M; Performing weighted averaging on the training influence weights included in the weight combination i and the N local model parameters included in the local model parameter set to obtain fusion model parameters, and determining the recognition model carrying the fusion model parameters as the candidate global model i associated with the weight combination i; Then, obtaining evaluation indicators of the M candidate global models in the multimedia verification data set respectively, and determining the target global model among the M candidate global models according to the evaluation indicators, includes: The evaluation indicators of the M candidate global models in the multimedia verification data set are obtained respectively, and among the M candidate global models, the candidate global model corresponding to the maximum evaluation indicator is determined as the target global model.

3. The method according to claim 2, characterized in that The obtaining of M weight combinations associated with the local model parameter set includes: Sample N values ​​within the target value range, and determine the sum of the absolute values ​​corresponding to the N values ​​as the norm value; The ratios between the N values ​​and the norm value are determined as a weight combination i associated with the local model parameter set.

4. The method according to claim 1, characterized in that: The step of determining, according to the first object prediction result and the second object prediction result, an evaluation index of the candidate global model i in the multimedia verification data set comprises: According to the first object prediction result, the number of first correct predictions of the candidate global model i in the positive sample pair is counted, and according to the second object prediction result, the number of second correct predictions of the candidate global model i in the negative sample pair is counted; Determine the sum of the first correct prediction number and the second correct prediction result as the total number of predicted correct sample pairs of the candidate global model i in the multimedia verification data set; The total number of sample pairs corresponding to the positive sample pairs and the negative sample pairs is obtained in the multimedia verification data set, and the evaluation index corresponding to the alternative global model i is determined according to the ratio between the total number of predicted correct sample pairs and the total number of sample pairs.

5. The method according to claim 4, characterized in that The number of the multimedia verification data sets is P, the P multimedia verification data sets include a multimedia verification data set j, P is a positive integer, and j is a positive integer less than or equal to P; Determining the evaluation index corresponding to the candidate global model i according to the ratio between the total number of predicted correct sample pairs and the total number of sample pairs includes: The ratio between the total number of correctly predicted sample pairs of the candidate global model i in the multimedia verification data set j and the total number of sample pairs corresponding to the multimedia verification data set j is determined as the prediction accuracy of the candidate global model i in the multimedia verification data set j; Obtain the prediction accuracy of the candidate global model i in P multimedia verification data sets, and calculate the average accuracy corresponding to the P prediction accuracy rates and the standard deviation values ​​corresponding to the P prediction accuracy rates; An evaluation index corresponding to the candidate global model i is determined according to the average accuracy and the standard deviation value.

6. The method according to claim 1, characterized in that The obtaining of evaluation indicators of the M candidate global models in the multimedia verification datasets respectively includes: Acquire a multimedia verification data set including positive sample pairs and negative sample pairs; the positive sample pairs refer to multimedia sample data pairs including the same object, and the negative sample pairs refer to multimedia sample data pairs including different objects; Obtaining the false acceptance rate of the candidate global model i included in the M candidate global models in the multimedia verification data set, and determining a similarity threshold in the similarity corresponding to the negative sample pair; the similarity threshold is determined by the number of the negative sample pairs and the false acceptance rate, and i is a positive integer less than or equal to M; The similarity corresponding to the positive sample pairs is obtained, and a first sample pair whose similarity is greater than the similarity threshold is obtained from the positive sample pairs, and a ratio between the number of the first sample pairs and the number of the positive sample pairs is determined as an evaluation indicator corresponding to the alternative global model i.

7. The method according to claim 6, characterized in that The obtaining of the false acceptance rate of the candidate global model i included in the M candidate global models in the multimedia verification data set includes: Obtaining the number of incorrect predictions of the candidate global model i included in the M candidate global models in the negative sample pair; The ratio between the number of incorrect predictions and the number of negative sample pairs is determined as the false acceptance rate of the candidate global model i in the multimedia verification dataset.

8. The method according to claim 1, characterized in that The target global model is generated based on N local model parameters corresponding to the rth synchronization period, where r is a positive integer; The method further comprises: Determine the target global model corresponding to the r-th synchronization period in the candidate global models according to the evaluation index, and obtain the historical global model corresponding to the (r-1)-th synchronization period; the historical global model is generated based on the local model parameters uploaded by the N clients in the (r-1)-th synchronization period; Obtaining the training learning rates of the N local recognition models in the rth synchronization period, and obtaining the model parameter difference between the target global model and the historical global model; The ratio of the model parameter difference to the training learning rate is determined as the federal momentum, and the federal momentum is sent to the N clients; the federal momentum together with the target global model is used to instruct the N clients to update the parameters of the associated local recognition models, and the federal momentum is used to indicate the training directions of the N local recognition models in their respective clients.

9. A data processing method, characterized in that: include: When the number of training times of the target local recognition model meets the synchronization cycle, the local model parameters corresponding to the target local recognition model are uploaded to the service device, so that the service device performs parameter fusion on the local model parameters uploaded by N clients respectively to obtain the target global model; the local model parameters uploaded by the N clients respectively include the local model parameters corresponding to the target local recognition model, the target global model is determined by the evaluation indicators of M candidate global models in the multimedia verification data set respectively, the M candidate global models are determined by M weight combinations associated with the local model parameter set and the local model parameter set, the local model parameter set includes the local model parameters uploaded by the N clients respectively, each weight combination includes the training influence weights associated with the N clients respectively, N is a positive integer greater than 1, and M is a positive integer; receiving the target global model returned by the service device, updating the parameters of the target local recognition model according to the target global model, and determining the target local recognition model after the parameter update as an object recognition model; the object recognition model is used to recognize objects of the target object type contained in the multimedia data; Among them, the M alternative global models include an alternative global model i, and the evaluation index of the alternative global model i in the multimedia verification data set is determined based on a first object prediction result and a second object prediction result, the first object prediction result is the output result of the alternative global model i for the positive sample pairs in the multimedia verification data set, and the second object prediction result is the output result of the alternative global model i for the negative sample pairs in the multimedia verification data set; the positive sample pair refers to a multimedia sample data pair containing the same object, and the negative sample pair refers to a multimedia sample data pair containing different objects, and i is a positive integer less than or equal to M.

10. The method according to claim 9, characterized in that Also includes: Acquire multimedia sample data, input the multimedia sample data into the target local recognition model, and output object spatial features corresponding to the multimedia sample data through the target local recognition model; Determining a training loss function corresponding to the target local recognition model according to the object spatial features and label information corresponding to the multimedia sample data; The training gradient of the target local recognition model is determined according to the training loss function, the parameters of the target local recognition model are updated according to the training gradient and the training learning rate corresponding to the target local recognition model, and the number of training times corresponding to the target local recognition model is counted.

11. The method according to claim 9, characterized in that The multimedia data includes a face image to be identified, and the target object type includes a face type; The method further comprises: Acquire the face image to be recognized, input the face image to be recognized into an object recognition model, and obtain the face spatial features corresponding to the face image to be recognized in the object recognition model; A face classification result corresponding to the face image to be identified is determined according to the face spatial feature; the face classification result is used to characterize the identity authentication result of an object with a face type contained in the face image to be identified.

12. A data processing device, characterized in that: include: A parameter acquisition module, used to acquire local model parameters corresponding to N local recognition models respectively; the N local recognition models are obtained by independent training of different clients respectively, each client includes multimedia sample data for training the associated local recognition model, the multimedia sample data includes objects with target object types, N is a positive integer greater than 1, and N is the number of clients; A parameter fusion module, used to obtain M weight combinations associated with a local model parameter set, and perform parameter fusion on the local model parameter set according to each weight combination to obtain M candidate global models; the local model parameter set includes local model parameters corresponding to the N local recognition models, each weight combination includes training influence weights corresponding to the N local model parameters, and M is a positive integer; A model determination module is used to obtain evaluation indicators of the M candidate global models in the multimedia verification data set, determine a target global model from the M candidate global models according to the evaluation indicators, and transmit the target global model to N clients, so that the N clients respectively update parameters of the associated local recognition models according to the target global model to obtain an object recognition model; The object recognition model is used to recognize objects of the target object type contained in the multimedia data; Wherein, the model determination module includes: A verification data set acquisition unit, used to acquire a multimedia verification data set including positive sample pairs and negative sample pairs; the positive sample pairs refer to multimedia sample data pairs including the same object, and the negative sample pairs refer to multimedia sample data pairs including different objects; A first prediction unit is configured to obtain an alternative global model i from the M alternative global models, input the positive sample pair into the alternative global model i, and output a first object prediction result of the positive sample pair through the alternative global model i; i is a positive integer less than or equal to M; A second prediction unit, configured to input the negative sample pair into the candidate global model i, and output a second object prediction result of the negative sample pair through the candidate global model i; The first evaluation index determination unit is used to determine the evaluation index of the candidate global model i in the multimedia verification data set according to the first object prediction result and the second object prediction result.

13. A computer device, characterized in that: including memory and processor; The memory is connected to the processor, the memory is used to store the computer program, and the processor is used to call the computer program so that the computer device executes any one of the methods of claims 1-11.

14. A computer-readable storage medium, characterized in that: A computer program is stored in the computer-readable storage medium. The computer program is suitable for being loaded and executed by a processor, so that a computer device having a processor executes the method according to any one of claims 1 to 11.

15. A computer program product, characterized in that The computer program product comprises computer instructions, which are stored in a computer-readable storage medium. When the computer instructions are executed by a processor, the method of any one of claims 1 to 11 is implemented.

Images