Computer vision model training method and apparatus, electronic device, and storage medium

Abstract

The present disclosure relates to a computer vision model training method and device, an electronic device and a storage medium. The method comprises obtaining multiple sets of model parameters from multiple electronic devices respectively, determining a visual model corresponding to each set of model parameters according to a preset model framework, and generating multiple synthetic images according to the visual models. Further, the visual models are distilled according to the synthetic images to obtain a target visual model. The present disclosure can obtain multiple visual models through communication of model parameters with multiple electronic devices, and distill the model through synthetic images without receiving the training set of each electronic device to obtain a target visual model that integrates the characteristics of multiple visual models. The training method of the present disclosure can integrate the characteristics of multiple visual models through model distillation, and effectively protect the privacy of each electronic device while integrating the model characteristics.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer technology, and particularly relates to a computer vision model training method and device, an electronic device and a storage medium. BACKGROUND

[0002] At present, in the technical field of computer technology, in order to avoid data privacy leakage, the data generation end is usually bound to the model training end. However, this training mode protects the data privacy at the same time, and forms a data island for each terminal, so that the data of different terminals cannot be aggregated together for training, and it is difficult to exert the greatest potential. SUMMARY

[0003] The present disclosure provides a computer vision model training method and device, an electronic device and a storage medium technical solution, aiming to protect privacy while integrating the characteristics of models in each terminal.

[0004] According to a first aspect of the present disclosure, a computer vision model training method is provided, comprising:

[0005] obtaining a plurality of vision models;

[0006] generating a plurality of synthetic images according to each of the vision models respectively;

[0007] performing model distillation on each of the synthetic images as input of each of the vision models respectively to obtain a target vision model.

[0008] In a possible implementation, the generating a plurality of synthetic images according to each of the vision models respectively comprises:

[0009] determining a plurality of initial noise images;

[0010] inputting each of the initial noise images into each of the vision models for iterative updating until a value of a first loss function meets a convergence condition to obtain a synthetic image corresponding to each of the noise images,

[0011] wherein the value of the first loss function is determined according to a first loss, a second loss and a third loss.

[0012] In a possible implementation, the first loss is used to represent a loss generated by the corresponding vision model in the process of detecting the noise image;

[0013] the second loss is used to represent a similarity degree between the noise image and a real image;

[0014] the third loss is used to represent a loss generated by the noise image in the process of transmission of each batch processing layer of the corresponding vision model.

[0015] In a possible implementation, the determination of the first loss comprises:

[0016] determining an initial first annotation result corresponding to each of the noise images;

[0017] inputting each of the initial noise images into each of the visual models to obtain a corresponding second annotation result;

[0018] determining a first loss according to the first annotation result and the second annotation result of each of the noise images.

[0019] In a possible implementation, the second loss is determined according to pixel values in the noise images.

[0020] In a possible implementation, the third loss is determined according to first feature maps output by each batch processing layer after the noise images are input into the corresponding visual model, and second feature maps input into each of the batch processing layers.

[0021] In a possible implementation, the model distillation of each of the synthetic images as input of each of the visual models to obtain the target visual model comprises:

[0022] determining a first visual model for transferring detection performance of other visual models in each of the visual models, and taking other models except the first visual model as second visual models;

[0023] performing model distillation of each of the synthetic images as input of the first visual model and each of the second visual models to obtain the target visual model.

[0024] In a possible implementation, the model distillation of each of the synthetic images as input of the first visual model and each of the second visual models to obtain the target visual model comprises:

[0025] inputting each of the synthetic images into the first visual model to obtain a first detection result;

[0026] inputting each of the synthetic images into each of the second visual models to obtain a second detection result;

[0027] iteratively training the first visual model through a second loss function to obtain the target visual model, the second loss function being determined according to the first detection result and each of the second detection results.

[0028] In a possible implementation, the method further comprises:

[0029] in response to receiving a model update request, returning a target model parameter of the target visual model.

[0030] In a possible implementation, the second loss function is a sum of L2 norms of the first detection result and each of the second detection results.

[0031] In a possible implementation, the method further includes:

[0032] obtaining real images stored in the plurality of electronic devices;

[0033] The model distillation, by taking each of the synthetic images as an input of the first visual model and each of the second visual models respectively, to obtain the target visual model includes:

[0034] The model distillation, by taking each of the real images and each of the synthetic images as an input of the first visual model and each of the second visual models respectively, to obtain the target visual model.

[0035] According to a second aspect of the present disclosure, a computer vision model training apparatus is provided, including:

[0036] A model determination module is configured to obtain a plurality of visual models.

[0037] An image generation module is configured to generate a plurality of synthetic images according to each of the visual models respectively.

[0038] A model distillation module is configured to perform model distillation by taking each of the synthetic images as an input of each of the visual models respectively, to obtain a target visual model.

[0039] In a possible implementation, the image generation module includes:

[0040] An image initialization submodule is configured to determine a plurality of initial noise images.

[0041] An image iteration submodule is configured to input each of the initial noise images into each of the visual models respectively for iterative updating until a value of a first loss function meets a convergence condition, to obtain a synthetic image corresponding to each of the noise images.

[0042] The value of the first loss function is determined according to a first loss, a second loss, and a third loss.

[0043] In a possible implementation, the first loss is used to represent a loss generated by a corresponding visual model in detecting the noise image.

[0044] The second loss is used to represent a similarity degree between the noise image and a real image.

[0045] The third loss is used to represent a loss generated by the noise image in a transmission process of each batch processing layer of the corresponding visual model.

[0046] In a possible implementation, the determination of the first loss comprises:

[0047] determining an initial first annotation result corresponding to each of the noise images;

[0048] inputting each of the initial noise images into each of the visual models to obtain a corresponding second annotation result;

[0049] determining a first loss according to the first annotation result and the second annotation result of each of the noise images.

[0050] In a possible implementation, the second loss is determined according to pixel values in the noise images.

[0051] In a possible implementation, the third loss is determined according to first feature maps output by each batch processing layer after the noise image is input into the corresponding visual model, and second feature maps input into each batch processing layer.

[0052] In a possible implementation, the model distillation module comprises:

[0053] a model selection sub-module, configured to determine, from the visual models, a first visual model used to transfer detection performance of other visual models, and to determine, as second visual models, models other than the first visual model;

[0054] a model distillation sub-module, configured to perform model distillation on each of the synthetic images as input of the first visual model and each of the second visual models to obtain a target visual model.

[0055] In a possible implementation, the model distillation sub-module comprises:

[0056] a first detection unit, configured to input each of the synthetic images into the first visual model to obtain a first detection result;

[0057] a second detection unit, configured to input each of the synthetic images into the second visual model to obtain a second detection result;

[0058] an iterative training unit, configured to iteratively train the first visual model by using a second loss function to obtain the target visual model, the second loss function being determined according to the first detection result and each of the second detection results.

[0059] In a possible implementation, the apparatus further comprises:

[0060] a parameter sending module, configured to return target model parameters of the target visual model in response to receiving a model update request.

[0061] In a possible implementation, the second loss function is a sum of L2 norms of the first detection result and each of the second detection results.

[0062] In a possible implementation, the apparatus further includes:

[0063] a real image acquisition module, configured to acquire real images stored in the plurality of electronic devices;

[0064] The model distillation sub-module includes:

[0065] a model distillation unit, configured to perform model distillation on each of the real images and each of the synthetic images as inputs of the first visual model and each of the second visual model respectively, to obtain a target visual model.

[0066] According to a third aspect of the present disclosure, an electronic device is provided, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the above method.

[0067] According to a fourth aspect of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions, and the computer program instructions are executed by a processor to implement the above method.

[0068] In the embodiments of the present disclosure, the characteristics of multiple visual models can be fused by model distillation, and the privacy of each electronic device is effectively protected while the characteristics of the models are fused.

[0069] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, but not limiting the present disclosure. Other features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0070] The accompanying drawings, which are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure and, together with the specification, serve to explain the technical solutions of the present disclosure.

[0071] Figure 1 A system schematic diagram of a computer vision model training method according to an embodiment of the present disclosure is shown;

[0072] Figure 2 A flowchart of a computer vision model training method according to an embodiment of the present disclosure is shown;

[0073] Figure 3 A flowchart of a process of determining a synthetic image according to an embodiment of the present disclosure is shown;

[0074] Figure 4is a schematic diagram illustrating determination of a composite image according to an example embodiment;

[0075] Figure 5 is another schematic diagram illustrating determination of a composite image according to an example embodiment;

[0076] Figure 6 is a schematic diagram illustrating generation of a composite image in an iterative manner according to an embodiment of the present disclosure;

[0077] Figure 7 is a schematic diagram illustrating a model distillation process according to an embodiment of the present disclosure;

[0078] Figure 8 is a schematic diagram illustrating a parameter communication process according to an example embodiment;

[0079] Figure 9 is another schematic diagram illustrating a parameter communication process according to an example embodiment;

[0080] Figure 10 is a schematic diagram illustrating a computer vision model training apparatus according to an embodiment of the present disclosure;

[0081] Figure 11 is a block diagram of an electronic device according to an example embodiment;

[0082] Figure 12 is another block diagram of an electronic device according to an example embodiment. DETAILED DESCRIPTION

[0083] Various example embodiments, features and aspects of the present disclosure will be explained below in detail with reference to the accompanying drawings. The same reference numerals are used throughout the drawings and the same elements are represented by the same characters without redundant explanation if necessary. Although various aspects of embodiments are illustrated in the drawings, the drawings are not necessarily drawn to scale unless specifically indicated.

[0084] The term "exemplary" is used herein to mean "serving as an example, instance, or illustration." Any implementation described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other implementations.

[0085] The term "and / or" used herein only means an association relationship of the associated objects, and means that there can be three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone. In addition, the term "at least one" herein means any one of a plurality or any combination of at least two of a plurality, for example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0086] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0087] Figure 1 A system schematic diagram illustrating a computer vision model training method applying embodiments of the present disclosure is shown. Figure 1 As shown, in one possible implementation, the system for implementing the computer vision model training method of this disclosure includes a first electronic device 10 and a plurality of second electronic devices 11 connected to the first electronic device 10 via a network. Each second electronic device 11 deploys a visual model to train the visual model using image data it acquires. Optionally, the visual models in each second electronic device 11 have the same framework, differing only in model parameters due to different training sets. The first electronic device 10 receives the model parameters trained by each second electronic device 11 using a corresponding training set, determines the visual model in each second electronic device 11 within the first electronic device 10, and fuses the visual models in each second electronic device 11 by generating a synthetic image and performing model distillation to obtain a target visual model possessing the characteristics of each visual model.

[0088] Optionally, the training set in each of the second electronic devices 11 can be determined by receiving images acquired by a specific electronic device, or by directly acquiring images by the second electronic device 11.

[0089] Furthermore, both the first electronic device 10 executing the computer vision model training method and the second electronic device 11 providing model parameters in this system can be terminal devices or servers, etc. The terminal device can be a user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, in-vehicle device, wearable device, or any other device capable of data processing. The detection method can be implemented by a processor calling computer-readable instructions stored in memory. Alternatively, when the electronic device is a server, the aforementioned computer vision model training method can also be executed through the server. Optionally, the server can be a single server or a cluster of multiple servers.

[0090] The embodiments of the present disclosure can be applied to any scene of fusing multiple computer vision models, for example, fusing features of face recognition models in different smartphones to improve the universality of the face recognition models. Or fusing features of image classification models in different terminal devices to improve the accuracy of the classification results of the image classification models.

[0091] Figure 2 A flowchart of a computer vision model training method according to an embodiment of the present disclosure is shown. As shown in the figure, the embodiment of the present disclosure can include the following steps when training a computer vision model: Figure 2

[0092] Step S10, obtaining multiple vision models.

[0093] In a possible implementation, multiple vision models are deployed in one or more electronic devices, and each vision model is trained based on a training set in the corresponding electronic device. The training set in each electronic device can be determined by receiving multiple images collected by a specific image collection device, or directly by multiple images collected by an image collection unit built-in the electronic device. The electronic device performing the computer vision model training method of the present disclosure can obtain multiple sets of model parameters from multiple electronic devices respectively, and determine the vision model corresponding to each set of model parameters according to a preset model framework to obtain multiple vision models.

[0094] Since the training sets in each electronic device are different and not intercommunicable, the model parameters of the vision models trained are also different. Further, the multiple vision models can be determined by obtaining multiple sets of model parameters corresponding to the vision models in the multiple electronic devices to further fuse the characteristics of the vision models. Optionally, each set of model parameters includes all parameters of the vision model in the corresponding electronic device.

[0095] In a possible implementation, the way to obtain the model parameters transmitted by the electronic device can be to send a parameter obtaining request to each electronic device, and the electronic device returns the corresponding model parameters after receiving the parameter obtaining request. Alternatively, each electronic device can also automatically upload the model parameters after the corresponding vision model is trained.

[0096] Further, the structure of the vision model in each electronic device can be the same or different. In the case where the structure of the vision model is the same, only the model parameters of the vision model in each electronic device need to be obtained. In the case where the structure of the vision model is different, the corresponding model framework can also be obtained at the same time as the model parameters of the vision model are obtained.

[0097] ​In a possible implementation, a model framework identical to the visual model structure of each electronic device is preset, and a corresponding visual model is generated in the electronic device receiving the model parameters according to each group of model parameters. In this way, the visual model trained by each electronic device is obtained only by obtaining the model parameters.

[0098] Further, when the visual model structures deployed in each electronic device are different, the electronic device receiving the model parameters receives, at the same time, the model framework corresponding to each group of model parameters when receiving each group of model parameters transmitted by each electronic device. The visual model trained by each electronic device is obtained according to each group of model parameters and the corresponding model framework.

[0099] Step S20, generate a plurality of synthetic images according to each of the visual models respectively.

[0100] In a possible implementation, after determining the plurality of visual models, a plurality of synthetic images are generated based on each visual model. The determination process of the synthetic images can further perform model distillation according to each synthetic image, and fuse the performance of each visual model without receiving the training set corresponding to each visual model.

[0101] Figure 3 A flowchart of a process of determining a synthetic image according to an embodiment of the present disclosure is shown. As shown in Figure 3 In a possible implementation, the process of determining a synthetic image according to a visual model according to an embodiment of the present disclosure can include the following steps:

[0102] Step S21, determine an initial plurality of noise images.

[0103] In a possible implementation, the initial plurality of noise images can be obtained by random initialization of an electronic device. For example, the plurality of noise images can be initialized by randomly adding noise to a preset blank image. Alternatively, the noise images can be generated by Gaussian noise, salt and pepper noise, etc. Alternatively, when the initial plurality of noise images are determined, the first annotation results corresponding to each noise image can also be randomly generated according to the output results of each visual model.

[0104] Further, the first annotation result can include one or more types of data. For example, when the output of the visual model is the detection box coordinates, the detection box coordinates corresponding to each noise image are randomly determined as the first annotation result. When the output of the visual model is the image category, the category corresponding to each noise image is randomly determined as the first annotation result. When the output result of the visual model is the detection box coordinates and the image category, the detection box coordinates and the category corresponding to each noise image are randomly determined as the first annotation result.

[0105] Step S22, input each initial noise image into each visual model respectively, and perform iterative updating until the value of the first loss function meets the convergence condition, to obtain a synthetic image corresponding to each noise image.

[0106] In a possible implementation, the plurality of initial noise images obtained are input into each visual model to output a predicted second annotation result. Each noise image can be input into each visual model to obtain a plurality of second annotation results output by each visual model. Alternatively, each noise image is input into only one visual model to obtain a second annotation result output by the corresponding visual model. For example, when the visual model is used for image classification, the output second annotation result can be a detection box coordinate obtained by detecting the noise image and / or an image category.

[0107] In a possible implementation, for each noise image, the visual model input thereto is iteratively updated until the value of the first loss function meets the convergence condition, to obtain a synthetic image corresponding to the noise image and the visual model. The value of the first loss function is determined according to a first loss, a second loss, and a third loss of each noise image. The first loss represents a loss generated by the corresponding visual model in detecting the noise image, the second loss represents a similarity between the noise image and a real image, and the third loss represents a loss generated by the noise image in a transmission process of each batch processing layer of the corresponding visual model.

[0108] In a possible implementation, the first loss function can include the first loss, the second loss, and the third loss. Alternatively, the first loss function can be a weighted sum of the first loss, the second loss, and the third loss. The first loss is used to represent a loss generated by the corresponding visual model in detecting the noise image. The second loss is used to represent a similarity between the noise image and a real image, and the third loss is used to represent a loss generated by the noise image in a transmission process of each batch processing layer of the corresponding visual model. Further, the determination process of the first loss includes determining a first annotation result corresponding to each initial noise image, inputting each initial noise image into each visual model to obtain a corresponding second annotation result, and determining the first loss according to the first annotation result and the second annotation result of each noise image. The first annotation result is obtained by initialization together with the initial noise image. The second loss is determined according to pixel values in the noise image. The third loss is determined according to the first feature map output by each batch processing layer after the noise image is input into the corresponding visual model, and the second feature map input into each batch processing layer.

[0109] The embodiment of the present disclosure determines the first loss function based on the pixel degree of the noise image and the real image, the detection loss of the input visual model, and the difference between the input and output of each layer in the input visual model, and can iteratively process the noise image based on three different losses. The synthesized image obtained by the first loss function can represent the characteristics of the training set corresponding to the input visual model, thereby improving the accuracy of the subsequent model distillation result.

[0110] Optionally, the first loss can be determined according to the content of the output result of each visual model, representing the difference between the detection result of the visual model and the actual result, that is, the difference between the first label result corresponding to the noise image input into the visual model and the second label result output by the visual model. For example, when the first label result and the second label result include image categories, the first loss can be determined by calculating the cross-entropy. When the first label result and the second label result include detection box coordinates, the first loss can be determined by calculating the L1 norm. Further, when the first label result and the second label result include multiple contents, the first loss can also be determined by calculating the weighted sum of each content.

[0111] It is easy to understand that the above-mentioned way of determining the difference between the model detection result and the actual result is an exemplary way in the present disclosure. In addition to the above-mentioned way, the first loss can also be determined by various loss calculation methods based on the difference between the output of the visual model in the application process, which will not be described here.

[0112] Further, the second loss represents the pixel degree of the noise image and the real image, which can be determined by the pixel value of each pixel position of the noise image. The lower the value of the second loss, the more similar the noise image and the real image. For example, for each noise image, the pixel value of each pixel position can be obtained first, and the L2 norm of each pixel value can be calculated to determine the second loss, or the total variation of each pixel value can be calculated to determine the second loss. Optionally, the second loss can also be determined by calculating the weighted sum of the L2 norm and the total variation of each pixel value. The total variation represents the sum of the degree of signal change, which can be determined by calculating the square of the difference between each pixel value and each adjacent pixel, and then calculating the sum.

[0113] In a possible implementation, each visual model includes a plurality of batch processing layers, and each noise image is sequentially processed by the plurality of batch processing layers in the input visual model to obtain a final output result. The input and output of each batch processing layer are a feature image, and the input and output of each batch processing layer are different. Optionally, the third loss can represent the difference between the input and output of each batch processing layer in the visual model. For example, for each visual model, the mean and standard deviation of the input feature image and the mean and standard deviation of the output feature image of each batch processing layer can be determined. First, the L2 norm sum between the mean of the input feature image and the mean of the output feature image of each batch processing layer is calculated, and the L2 norm sum between the standard deviation of the input feature image and the standard deviation of the output feature image is calculated. Then, the third loss is obtained by calculating the weighted sum of the two L2 norm sums.

[0114] In a possible implementation, after obtaining the first loss function of each noise image inputting the visual model, the noise image is iterated by the gradient descent method until the value of the first loss function meets the preset convergence condition, and the synthetic image is obtained when the iteration is stopped. The convergence condition can be that the value of the first loss function is less than the convergence threshold, or the value of the first loss function is the minimum value in the preset iteration number.

[0115] Figure 4 is a schematic diagram for determining a synthetic image according to an example embodiment. As shown in Figure 4 In an optional implementation, for each randomly generated noise image 40, different visual models 41 can be input respectively for iteration. Since the model parameters of each visual model 41 are different, the same noise image 40 can obtain different N synthetic images 42 after iteration by N visual models 41.

[0116] Figure 5 is another schematic diagram for determining a synthetic image according to an example embodiment. As shown in Figure 5 In an optional implementation, for each randomly generated noise image 50, each noise image can be input into only one visual model 51 for iteration to obtain a corresponding synthetic image 52. The generation mode of the synthetic image can increase the difference between different synthetic images, and further improve the accuracy of the model distillation result.

[0117] Figure 6 is a schematic diagram for generating a synthetic image in an iterative manner according to an embodiment of the present disclosure. As shown in Figure 6As shown, in a possible implementation, the synthetic image generation manner of the embodiments of the present disclosure is to first determine a noise image 60, then input the noise image into a visual model 61, and calculate a first loss function 62 according to a second label result output by the visual model 61, pixel values of each pixel position of the noise image 60, and input and output of each batch processing layer in the visual model 61. Further, it is judged whether the value of the first loss function 62 satisfies a preset convergence condition 63, and in the case of satisfaction, a synthetic image 64 corresponding to the noise image 60 is obtained, and in the case of non-satisfaction, the noise image 61 is adjusted, and the next iteration process is performed.

[0118] The synthetic images generated by the embodiments of the present disclosure are generated based on iteration of each visual model, and carry the characteristics of each visual model, so that each model can still be fused through model distillation without real images.

[0119] In step S30, each synthetic image is input into each visual model for model distillation to obtain a target visual model.

[0120] In a possible implementation, a first visual model for migrating detection performance of other visual models can be determined in each visual model, and other models except the first visual model are taken as second visual models. Each synthetic image is input into the first visual model and each second visual model for model distillation to obtain a target visual model. The distillation process can be that each synthetic image is input into the first visual model to obtain a first detection result, each synthetic image is input into each second visual model to obtain a second detection result, the first visual model is iteratively trained through a second loss function to obtain a target visual model, and the second loss function is determined according to the first detection result and each second detection result.

[0121] Optionally, the first detection result and the second detection result are also determined according to the output of the visual model, and one or more kinds of data are included. The second loss function can be the L2 norm sum of the first detection result and each second detection result. That is, the second loss function can be determined by calculating the L2 norm sum of the same data in the first detection result and each second detection result, respectively. For example, when the first detection result and the second detection result both include detection box coordinates, the L2 norm sum of the detection box coordinates in the first detection result and the detection box coordinates in each second detection result is calculated to obtain the second loss function. Or, when the first detection result and the second detection result both include detection box coordinates and image types, the L2 norm sum of the detection box coordinates in the first detection result and the detection box coordinates in each second detection result is calculated, and the L2 norm sum of the image types in the first detection result and the image types in each second detection result is calculated, and a weighted sum of the two L2 norm sums is calculated to obtain the second loss function.

[0122] Further, the model distillation process through the second loss function can be adjusting the second loss function through gradient descent method until the value of the second loss function meets the preset condition, and obtaining the target visual model.

[0123] Figure 7 A schematic diagram of a model distillation process according to an embodiment of the present disclosure is shown. As shown in Figure 7 one possible implementation, each synthetic image 70 is input into a first visual model 71 and each second visual model 72, the first visual model 71 obtains a first detection result 73 after detecting the input synthetic image 70, and each second visual model 72 obtains a second detection result 74 after detecting the input synthetic image 70. Further, the second loss function 75 is calculated through the first detection result 73 and each second detection result 74, and the model distillation is performed by iterating the second loss function 75, the model characteristics of each second visual model 72 are fused into the first visual model 71, and the target visual model with all visual model characteristics is obtained.

[0124] In an optional implementation of an embodiment of the present disclosure, to improve the authenticity of the final distillation result, the receiving device of each group of model parameters can also obtain part of real images to construct a training set together with the synthetic images for model distillation training. That is, the electronic device can also obtain real images stored in multiple electronic devices, and each real image and each synthetic image are input into the first visual model and each second visual model respectively for model distillation to obtain the target visual model.

[0125] Figure 8 is a schematic diagram of a parameter communication process according to an exemplary embodiment. As shown in Figure 8 one possible implementation, the model parameters sent by the second electronic device 80 are received by the first electronic device 81 to perform model distillation on the visual model in each second electronic device 80, and the target visual model with all visual model characteristics is obtained. Further, to improve the performance of the visual model in each second electronic device 80, the first electronic device 81 returns each target model parameter in the target visual model to each second electronic device 80 after processing to obtain the target visual model, and the second electronic device 80 updates the visual model according to the target model parameter.

[0126] Figure 9 is another schematic diagram of a parameter communication process according to an exemplary embodiment. As shown in Figure 9 one possible implementation, the target model parameters of the target visual model can be returned in response to receiving a model update request.

[0127] The model parameters sent by the plurality of second electronic devices 90 are received by the first electronic device 91 to perform model distillation on the visual models in the second electronic devices 90, so as to obtain a target visual model with all the characteristics of the visual models. Further, when the second electronic device 90 needs to update the deployed visual model, the second electronic device 90 sends a model update request to the first electronic device 91, and the first electronic device 91 returns the target model parameters of the target visual model to the second electronic device 90 to update the model.

[0128] The embodiments of the present disclosure can obtain the plurality of visual models trained on the plurality of electronic devices through only one communication of model parameters with the plurality of electronic devices. Meanwhile, the synthetic images are generated by the visual models themselves without receiving the training set of each electronic device, a training set for model distillation is constructed, and a target visual model with the characteristics of the plurality of visual models is obtained. The training method of the embodiments of the present disclosure can fuse the characteristics of the plurality of visual models through model distillation, and effectively protect the privacy of each electronic device while fusing the characteristics of the models.

[0129] It can be understood that the above-mentioned various method embodiments of the present disclosure can be combined with each other to form combined embodiments without deviating from the principle logic. Limited by the length, the present disclosure will not be repeated. Those skilled in the art can understand that the specific execution order of each step in the above-mentioned method should be determined according to its function and possible internal logic.

[0130] In addition, the present disclosure also provides a computer vision model training apparatus, an electronic device, a computer readable storage medium, and a program, which can be used to implement any one of the computer vision model training methods provided by the present disclosure. The corresponding technical solutions and descriptions are described in the method part and will not be repeated.

[0131] Figure 10 A schematic diagram of a computer vision model training apparatus according to an embodiment of the present disclosure is shown as follows. Figure 10 As shown, the apparatus includes a model determination module 100, an image generation module 101, and a model distillation module 102.

[0132] The model determination module 100 is configured to obtain a plurality of visual models.

[0133] The image generation module 101 is configured to generate a plurality of synthetic images according to each of the visual models respectively.

[0134] The model distillation module 102 is configured to perform model distillation on each of the synthetic images as the input of each of the visual models, so as to obtain a target visual model.

[0135] In a possible implementation, the image generation module 101 includes:

[0136] an image initialization submodule, configured to determine a plurality of initial noise images;

[0137] an image iteration submodule, configured to input each of the initial noise images into each of the visual models for iterative updating until a value of a first loss function meets a convergence condition, to obtain a synthetic image corresponding to each of the noise images,

[0138] wherein the value of the first loss function is determined according to a first loss, a second loss, and a third loss.

[0139] In a possible implementation, the first loss is used to represent a loss generated by a corresponding visual model in detecting the noise image.

[0140] The second loss is used to represent a similarity between the noise image and a real image.

[0141] The third loss is used to represent a loss generated by the noise image in a transmission process of each batch processing layer of the corresponding visual model.

[0142] In a possible implementation, the determination process of the first loss includes:

[0143] determining a first annotation result corresponding to each of the initial noise images;

[0144] inputting each of the initial noise images into each of the visual models to obtain a corresponding second annotation result;

[0145] determining the first loss according to the first annotation result and the second annotation result of each of the noise images.

[0146] In a possible implementation, the second loss is determined according to each pixel value in the noise image.

[0147] In a possible implementation, the third loss is determined according to a first feature map output by each batch processing layer after the noise image is input into the corresponding visual model, and a second feature map input into each of the batch processing layers.

[0148] In a possible implementation, the model distillation module 102 includes:

[0149] a model selection submodule, configured to determine, from the visual models, a first visual model used to transfer detection performance of other visual models, and to determine, as a second visual model, a model other than the first visual model;

[0150] The model distillation submodule is configured to perform model distillation on each of the synthetic images as input of the first visual model and each of the second visual models, to obtain a target visual model.

[0151] In a possible implementation, the model distillation submodule comprises:

[0152] The first detection unit is configured to input each of the synthetic images into the first visual model to obtain a first detection result.

[0153] The second detection unit is configured to input each of the synthetic images into the second visual model to obtain a second detection result.

[0154] The iterative training unit is configured to iteratively train the first visual model by using a second loss function to obtain a target visual model, where the second loss function is determined according to the first detection result and each of the second detection results.

[0155] In a possible implementation, the apparatus further comprises:

[0156] The parameter sending module is configured to return target model parameters of the target visual model in response to receiving a model update request.

[0157] In a possible implementation, the second loss function is L2 norm sum of the first detection result and each of the second detection results.

[0158] In a possible implementation, the apparatus further comprises:

[0159] The real image acquisition module is configured to acquire real images stored in the plurality of electronic devices.

[0160] The model distillation submodule comprises:

[0161] The model distillation unit is configured to perform model distillation on each of the real images and each of the synthetic images as input of the first visual model and each of the second visual models, to obtain a target visual model.

[0162] In some embodiments, the apparatus provided by the embodiments of the present disclosure has functions or comprises modules that can be used to perform the methods described in the above method embodiments, and the specific implementation can refer to the description of the above method embodiments. For brevity, details are not repeated here.

[0163] The embodiments of the present disclosure also propose a computer-readable storage medium having computer program instructions stored thereon, and the computer program instructions are executed by a processor to implement the above method. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0164] The embodiments of the present disclosure also provide an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above method.

[0165] The embodiments of the present disclosure also provide a computer program product, comprising computer readable code, or a non-volatile computer readable storage medium carrying computer readable code, when the computer readable code is run in the processor of an electronic device, the processor in the electronic device executes the above method.

[0166] The electronic device can be provided as a terminal, a server or other forms of devices.

[0167] Figure 11 is a block diagram of an electronic device 1100 according to an exemplary embodiment. For example, the electronic device 1100 can be a terminal such as a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0168] Referring to Figure 11 , the electronic device 1100 can include one or more of the following components: a processing component 1102, a memory 1104, a power supply component 1106, a multimedia component 1108, an audio component 1110, an input / output (I / O) interface 1112, a sensor component 1114, and a communication component 1116.

[0169] The processing component 1102 usually controls the overall operations of the electronic device 1100, such as operations associated with display, telephone calling, data communication, camera operation and recording operation. The processing component 1102 can include one or more processors 1120 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 1102 can include one or more modules to facilitate interaction between the processing component 1102 and other components. For example, the processing component 1102 can include a multimedia module to facilitate interaction between the multimedia component 1108 and the processing component 1102.

[0170] The memory 1104 is configured to store various types of data to support the operations of the electronic device 1100. Examples of such data include instructions for any application or method operating on the electronic device 1100, contact data, phonebook data, messages, pictures, videos, and the like. The memory 1104 can be implemented by any type of volatile or nonvolatile memory, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disc, or optical disc.

[0171] The power component 1106 supplies power for various components of the electronic device 1100. The power component 1106 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the electronic device 1100.

[0172] The multimedia component 1108 includes a screen providing an output interface between the electronic device 1100 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes the touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensors can not only sense a boundary of a touching or sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 1108 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the electronic device 1100 is in an operation mode, such as a photographing mode or a video mode. Each of the front camera and the rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0173] The audio component 1110 is configured to output and / or input an audio signal. For example, the audio component 1110 includes a microphone (MIC) configured to receive an external audio signal when the electronic device 1100 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 1104 or transmitted via the communication component 1116. In some embodiments, the audio component 1110 also includes a speaker for outputting an audio signal.

[0174] The I / O interface 1112 provides an interface between the processing component 1102 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0175] The sensor component 1114 includes one or more sensors for providing status assessments for various aspects of the electronic device 1100. For example, the sensor component 1114 can detect an open / closed position of the electronic device 1100, relative positioning of components of the electronic device 1100, such as a display and a keypad of the electronic device 1100, a change in position of the electronic device 1100 or a component of the electronic device 1100, the presence or absence of user contact with the electronic device 1100, the orientation or acceleration / deceleration / g-force and a temperature change of the electronic device 1100. The sensor component 1114 can include a proximity sensor configured to detect presence of a nearby object without any physical touch. The sensor component 1114 can also include a light sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, utilized in an imaging application. In some embodiments, the sensor component 1114 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0176] The communication component 1116 is configured to facilitate wired or wireless communication between the electronic device 1100 and another device. The electronic device 1100 can access a wireless network based on a communication standard, such as Wireless Fidelity (WiFi), 2nd-Generation (2G) or 3rd-Generation (3G) cellular telecommunication standards, or a combination thereof. In an example embodiment, the communication component 1116 receives broadcast signals or broadcast-related information from an external broadcast management system using a broadcast channel. In an example embodiment, the communication component 1116 can further include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, Ultra-Wide Band (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0177] In an example embodiment, the electronic device 1100 can be implemented using one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices (PLDs), Field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, or other electronic units to perform the above-described methods.

[0178] In an example embodiment, a non-transitory computer-readable storage medium, such as the memory 1104 including computer program instructions, is also provided, which can be executed by the processor 1120 of the electronic device 1100 to perform the above-described methods.

[0179] Figure 12is a block diagram of another electronic device according to exemplary embodiments. For example, the electronic device 1200 can be provided as a server. Referring to FIG. 12, the electronic device 1200 includes a processing component 1222, a memory 1232, a communication interface 1242, a user interface 1252, and a power supply component 1226. Figure 12 The processing component 1222 includes one or more processors, and the memory 1232 includes a volatile memory and / or a non-volatile memory. For example, the memory 1232 can include a buffer, a

[0180] The electronic device 1200 can further include a power supply component 1226 configured to perform power management of the electronic device 1200, a wired or wireless network interface 1250 configured to connect the electronic device 1200 to a network, and an input / output (I / O) interface 1258. The electronic device 1200 can operate based on an operating system stored in the memory 1232, such as a Microsoft Windows Server operating system (Windows Server TM ), an Apple Inc. operating system based on a graphical user interface (Mac OSX TM ), a multi-user multi-processing computer operating system (Unix TM ), a free and open-source Unix-like operating system (Linux TM ), an open-source Unix-like operating system (FreeBSD TM ), or the like.

[0181] In exemplary embodiments, a non-transitory computer readable storage medium, such as the memory 1232 including computer program instructions, is also provided, which can be executed by the processing component 1222 of the electronic device 1200 to complete the above-described methods.

[0182] The present disclosure can be a system, a method, and / or a computer program product. The computer program product can include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0183] Computer readable storage media can be tangible storage media which can retain and store instructions for use by an instruction execution device. Computer readable storage media can be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer readable storage media include the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0184] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network can comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0185] Computer readable program instructions for carrying out operations of the present disclosure can be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The computer readable program instructions can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate array (FPGA), or programmable logic array (PLA) can execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure.

[0186] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0187] These computer readable program instructions can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions can also be stored in a computer readable storage medium that can include random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other data storage device. When the computer readable program instructions are loaded into the computer and other programmable data processing apparatus, a series of operational steps are implemented that provide processes such that the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0188] The computer readable program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus or other device to produce a computer implemented process such that the instructions which execute on the computer, other programmable data processing apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0189] The flow diagrams and the block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams and the block diagrams can represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logic functions. In some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flow diagrams, and combinations thereof, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and

[0190] The computer program product can be embodied in a tangible medium of

[0191] The above description of the various embodiments is intended to be illustrative in all respects, rather than being restrictive, as it is readily possible to make modifications of the various embodiments, as those skilled in the art would realize upon reading the above description. The scope of the application should be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

[0192] Those skilled in the art can understand that, in the above-described method of the specific embodiments, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process, and the specific execution order of each step should be determined according to its function and possible inherent logic.

[0193] If the technical solutions of the present application involve personal information, the product applying the technical solutions of the present application has been explicitly informed of the personal information processing rules before processing the personal information, and has obtained the personal independent consent. If the technical solutions of the present application involve sensitive personal information, the product applying the technical solutions of the present application has obtained the personal independent consent before processing the sensitive personal information, and at the same time meets the requirement of "explicit consent". For example, at the personal information collection device such as camera, a clear and prominent mark is set to inform that it has entered the personal information collection range and will collect personal information. If the individual voluntarily enters the collection range, it is considered to agree to collect personal information. Or on the device for processing personal information, through the pop-up information or by asking the individual to upload his personal information, the individual's authorization is obtained under the condition that the device uses obvious mark / information to inform the individual of the personal information processing rules. The personal information processing rules can include personal information processor, personal information processing purpose, processing method and personal information type, etc.

[0194] The above has described various embodiments of the present disclosure, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes are obvious to those skilled in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles, practical application or improvement of technology in the market of the embodiments, or to enable other ordinary skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer vision model training method, characterized in that, The method comprises: obtaining a plurality of visual models; generating a plurality of synthetic images according to each of the visual models respectively; performing model distillation on each of the synthetic images as input of each of the visual models to obtain a target visual model; the model distillation on each of the synthetic images as input of each of the visual models to obtain a target visual model comprises: determining a first visual model for migrating detection performance of other visual models in each of the visual models, and taking other models except the first visual model as second visual models; inputting each of the synthetic images into the first visual model to obtain a first detection result; inputting each of the synthetic images into the second visual models respectively to obtain a second detection result; iteratively training the first visual model through a second loss function to obtain a target visual model, the second loss function being determined according to the first detection result and each of the second detection results.

2. The method of claim 1, wherein, the generating a plurality of synthetic images according to each of the visual models respectively comprises: determining a plurality of initial noise images; inputting each of the initial noise images into each of the visual models for iterative updating until a value of a first loss function meets a convergence condition to obtain a synthetic image corresponding to each of the noise images, wherein the value of the first loss function is determined according to a first loss, a second loss and a third loss.

3. The method of claim 2, wherein, the first loss is used to represent a loss generated by a corresponding visual model in detecting the noise image; the second loss is used to represent a similarity between the noise image and a real image; the third loss is used to represent a loss generated by the noise image in a transmission process of each batch processing layer of the corresponding visual model.

4. The method of claim 3, wherein, the determination process of the first loss comprises: determining a first annotation result corresponding to each of the initial noise images; inputting each of the initial noise images into each of the visual models to obtain a corresponding second annotation result; determining the first loss according to the first annotation result and the second annotation result of each of the noise images.

5. The method according to claim 3 or 4, characterized in that, the second loss is determined according to each pixel value in the noise image.

6. The method of any one of claim 5, wherein, the third loss is determined according to a first feature map output by each batch processing layer after inputting the noise image into the corresponding visual model, and a second feature map input into each of the batch processing layers.

7. The method of any one of claim 6, wherein, The method further comprises: in response to receiving a model update request, returning target model parameters of the target visual model.

8. The method of claim 7, wherein, the second loss function is L2 norm sum of the first detection result and each of the second detection results.

9. The method of any one of claim 7, wherein, The method further comprises: obtaining real images stored in a plurality of electronic devices; the model distillation on each of the synthetic images as input of the first visual model and each of the second visual models to obtain a target visual model comprises: performing model distillation on each of the real images and each of the synthetic images as input of the first visual model and each of the second visual models to obtain a target visual model.

10. A computer vision model training apparatus, comprising: The device comprises: a model determination module configured to obtain a plurality of visual models; an image generation module configured to generate a plurality of synthetic images according to each of the visual models respectively; The model distillation module is configured to perform model distillation on each of the synthetic images as input of each of the visual models to obtain a target visual model. The model distillation module comprises: a model selection sub-module configured to determine a first visual model for transferring detection performance of other visual models from among the visual models, and determine other models except the first visual model as second visual models; a first detection unit configured to input each of the synthetic images into the first visual model to obtain a first detection result; a second detection unit configured to input each of the synthetic images into the second visual models respectively to obtain second detection results; an iterative training unit configured to iteratively train the first visual model by a second loss function to obtain the target visual model, the second loss function being determined according to the first detection result and the second detection results.

11. An electronic device, comprising: comprise: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method of any one of claims 1 to 9.

12. A computer-readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 9. The computer program instructions, when executed by the processor, implement the method of any one of claims 1 to 9.

Images