Model training method, storage medium and electronic device

Abstract

The application discloses a model training method, a storage medium and an electronic device. The method comprises the following steps: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model by using the adversarial discrete image, wherein the image processing model is used for performing image processing on an input image to obtain a processing result of the input image. The application solves the technical problem of low recognition accuracy of a model in the related art.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, and in particular, to a model training method, a storage medium, and an electronic device. BACKGROUND

[0002] At present, when a neural network model is used to process visual tasks such as target recognition, target detection, and image representation learning, the texture features of an image are generally recognized, and it is difficult to recognize the actual meaning to be expressed in the image, resulting in low recognition accuracy of the model.

[0003] At present, no effective solution has been proposed for the above problems. SUMMARY

[0004] Embodiments of the present application provide a model training method, a storage medium, and an electronic device to at least solve the technical problem of low recognition accuracy of the model in related technologies.

[0005] According to an aspect of an embodiment of the present application, a model training method is provided, including: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0006] According to another aspect of an embodiment of the present application, an image processing method is provided, including: obtaining a target image; and performing image processing on the target image using an image processing model to obtain a target processing result of the target image, wherein the image processing model is obtained by training an adversarial discrete image, the adversarial discrete image is obtained by performing discretization processing on an adversarial image, and the adversarial image is generated by superimposing noise data on an original image.

[0007] According to another aspect of an embodiment of the present application, a model training method is provided, including: in response to an input instruction acting on an operation interface, displaying an original image on the operation interface; and in response to a model training instruction acting on the operation interface, displaying a training result of an image processing model on the operation interface, wherein the training result is used to represent that the image processing model is trained, the image processing model is obtained by training an adversarial discrete image, the adversarial discrete image is obtained by performing discretization processing on an adversarial image, and the adversarial image is generated by superimposing noise data on an original image.

[0008] According to another aspect of the embodiments of the present application, a model training method is provided, including: displaying an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model using the adversarial discrete image, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image; and driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is configured to represent that the image processing model is trained.

[0009] According to another aspect of the embodiments of the present application, a model training method is provided, including: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the model training request; obtaining an original image and an image processing model corresponding to the model training request, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training the image processing model using the adversarial discrete image to obtain a target processing model; and outputting the target processing model by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing model.

[0010] According to another aspect of the embodiments of the present application, a computer readable storage medium is provided, including a stored program, wherein the program, when executed, controls a device where the computer readable storage medium is located to perform any of the above methods.

[0011] According to another aspect of the embodiments of the present application, an electronic device is provided, including: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the model training request; obtaining an original image and an image processing model corresponding to the model training request, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training the image processing model using the adversarial discrete image to obtain a target processing model; and outputting the target processing model by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing model.

[0012] In the embodiment of the present application, first, an original image is acquired; noise data is superimposed on the original image to generate an adversarial image; the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image; and the adversarial discrete image is used to train an image processing model, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image, thereby improving the robustness of the image processing model. It is easy to note that by superimposing noise data on the original image, an adversarial image used for adversarial training can be obtained, the adversarial image can be discretized to obtain an adversarial discrete image containing semantic information, and the image processing model can be trained by the adversarial discrete image to achieve visual perception. Moreover, since the difference between the adversarial discrete image and the original sample distribution is close, the original classification effect will not be affected when the image processing model is trained by the generated adversarial discrete image, and the robustness of the model can be improved, thereby solving the technical problem of low recognition accuracy of the model in the related art. BRIEF DESCRIPTION OF DRAWINGS

[0013] The accompanying drawings, which are included to provide a further understanding of the present application, constitute a part of the present application and illustrate embodiments of the present application and the description thereof, which serve to explain the present application and do not constitute improper limitations on the present application. In the drawings:

[0014] Figure 1 FIG. 1 is a schematic diagram of a virtual reality device according to an embodiment of the present application;

[0015] Figure 2 FIG. 2 is a structural block diagram of a computing environment according to an embodiment of the present application;

[0016] Figure 3 FIG. 3 is a flowchart of a model training method according to an embodiment of the present application;

[0017] Figure 4 FIG. 4 is a schematic diagram of image discretization according to an embodiment of the present application;

[0018] Figure 5 FIG. 5 is a schematic diagram of a training discretization model according to an embodiment of the present application;

[0019] Figure 6 FIG. 6 is a flowchart of an image processing method according to an embodiment of the present application;

[0020] Figure 7 FIG. 7 is a flowchart of a model training method according to an embodiment of the present application;

[0021] Figure 8 FIG. 8 is a flowchart of a model training method according to an embodiment of the present application;

[0022] Figure 9 is a flowchart of a model training method according to an embodiment of the application;

[0023] Figure 10 is a schematic diagram of a model training apparatus according to an embodiment of the application;

[0024] Figure 11 is a schematic diagram of an image processing apparatus according to an embodiment of the application;

[0025] Figure 12 is a schematic diagram of a model training apparatus according to an embodiment of the application;

[0026] Figure 13 is a schematic diagram of a model training apparatus according to an embodiment of the application;

[0027] Figure 14 is a schematic diagram of a model training apparatus according to an embodiment of the application;

[0028] Figure 15 is a structural block diagram of a computer terminal according to an embodiment of the application. DETAILED DESCRIPTION

[0029] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the scope of protection of the present application.

[0030] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] First, some of the nouns or terms that appear in the description of the embodiments of the present application are applicable to the following explanations:

[0032] Vector Quantization GAN, VQGAN for short: a vector quantization GAN model;

[0033] Vision Transformer, ViT for short: a new type of vision transformer model;

[0034] ResNet: a deep convolutional network based on residual connection;

[0035] DAT: a discrete adversarial training algorithm based on a visual model;

[0036] AdvProp: a training method that uses adversarial samples for data augmentation.

[0037] At present, when AI (artificial intelligence) visual technology processes visual tasks, it is difficult to achieve robust visual perception, and even often makes errors that are difficult to understand. For example, an AI face recognition model can only recognize real faces, but when a cartoon face image is input, the model has difficulty outputting normal results. The following methods are generally used to improve the robustness of the model:

[0038] (1) Adversarial training optimizes adversarial perturbations and constructs adversarial samples that cause the model to make classification errors, so as to train the model to improve the recognition rate on these adversarial samples. However, the distribution of the adversarial samples generated by this method is quite different from that of actual real natural samples, which will affect the model's ability to recognize normal samples.

[0039] (2) Automatic data augmentation technology searches for image transformations that maximize the probability of incorrect classification by the current model, and applies it to the augmentation of training data to improve the model's recognition robustness. However, this method requires pre-defining some image transformation operations, and the diversity of the training data augmentation obtained is not strong enough. Although it does not affect the ability to recognize normal samples, the effect of improving robustness is generally not good.

[0040] In order to solve the above problems, the present application provides a model training method, which can change discrete images by adding noise data to images in an adversarial manner, so as to provide more complex samples to the model training, and finally obtain a model with strong visual perception robustness through continuous adversarial enhancement.

[0041] Embodiment 1

[0042] According to the embodiments of the present application, a model training method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0043] Figure 1 is a schematic diagram of a hardware environment of a virtual reality device according to a model training method embodiment of the present application. As shown in Figure 1 the virtual reality device 104 is connected to the terminal 106, and the terminal 106 is connected to the server 102 through a network. The virtual reality device 104 is not limited to a virtual reality headset, virtual reality glasses, virtual reality all-in-one machine, etc., and the terminal 104 is not limited to a PC, a mobile phone, a tablet computer, etc. The server 102 can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network, or a local area network.

[0044] Optionally, the virtual reality device 104 of the embodiment includes a memory, a processor, and a transmission device. The memory is used to store an application program, which can be used to perform the following steps: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0045] The terminal of the embodiment can be used to perform the following steps: displaying an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; obtaining the original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image; driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is used to represent that the training of the image processing model is completed; and sending the training result to the virtual reality device 104, and displaying the training result at a target delivery position after the virtual reality device 104 receives the training result.

[0046] Optionally, the virtual reality device 104 of the embodiment has an eye tracking HMD (Head Mount Display) head-mounted display and an eye tracking module, which have the same functions as those in the above embodiment, i.e., the screen in the HMD head-mounted display is used to display real-time images, and the eye tracking module in the HMD is used to obtain the real-time motion path of the user's eyeballs. The terminal of the embodiment obtains the position information and motion information of the user in the real three-dimensional space through a tracking system, and calculates the three-dimensional coordinates of the user's head in the virtual three-dimensional space and the direction of the user's field of view in the virtual three-dimensional space.

[0047] Figure 1 The illustrated hardware structural diagram can not only serve as an exemplary block diagram of the AR / VR device (or mobile device) described above, but also as an exemplary block diagram of the server described above, in an alternative embodiment, Figure 2 The use of the above-described Figure 1 The AR / VR device (or mobile device) shown is an embodiment of a computing node in a computing environment 201. Figure 2 is a structural block diagram of a computing environment according to a model training method of an embodiment of the present application, as described above. Figure 2 As shown, the computing environment 201 includes a plurality of computing nodes (e.g., servers) running on a distributed network (210-1, 210-2,..., are used in the figure to show). Each computing node contains local processing and memory resources, and end users 202 can remotely run applications or store data in the computing environment 201. Applications can be provided as a plurality of services 220-1, 220-2, 220-3 and 220-4 in the computing environment 301, representing services "A", "D", "E" and "H", respectively.

[0048] End users 202 can provide and access services through web browsers or other software applications on the client, and in some embodiments, the supply and / or requests of end users 202 can be provided to an entry gateway 230. The entry gateway 230 can include a corresponding agent to process the supply and / or requests for services 220 (one or more services provided in the computing environment 201).

[0049] Services 220 are provided or deployed according to various virtualization technologies supported by the computing environment 201. In some embodiments, services 220 can be provided according to virtual machine (VM) based virtualization, container-based virtualization, and / or the like. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, executing programs and applications without directly accessing any actual hardware resources. While the virtual machine virtualization machine, according to the container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0050] In an embodiment of container-based virtualization, several containers of services 220 can be assembled into a POD (e.g., a Kubernetes POD). For example, as shown in Figure 2As shown, service 220-2 can be equipped with one or more PODs 240-1, 240-2, …, 240-N (collectively, PODs 240). Each POD 240 can include an agent 245 and one or more containers 242-1, 242-2, …, 242-M (collectively, containers 242). The one or more containers 242 in a POD 240 handle requests related to one or more respective functions of the service, and the agent 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services 220 can also be equipped with PODs similar to PODs 240.

[0051] In operation, executing a user request from end user 202 can require invoking one or more services 220 in computing environment 201, executing one or more functions of a service 220, and invoking one or more functions of another service 220. As Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 can invoke service "D" 220-2, which can request service "E" 220-3 to execute one or more functions.

[0052] The computing environment described above can be a cloud computing environment, in which allocation of resources is managed by a cloud service provider, allowing development of functionality without regard to implementing, tuning, or scaling servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be partitioned into a set of functions that can automatically scale independently, rather than scaling a single hardware device to handle potential load.

[0053] In the above-described operating environment, the present disclosure provides a model training method as shown in Figure 3 It should be noted that the model training method of this embodiment can be executed by a mobile terminal as shown in Figure 1 The mobile terminal of the embodiment shown in Figure 3 is a flowchart of a model training method according to Embodiment 1 of the present disclosure. As shown in Figure 3 The method can include the following steps:

[0054] Step S302, obtaining an original image.

[0055] The original image described above can be an image used to train an image processing model, wherein the original image can be a face image, a human body image, a landscape image, an animal image, a building image, a remote sensing image, a cartoon image, but is not limited thereto, and can also be any image. Among them, the face image and the human body image are obtained from a public face image set and a public human body image set.

[0056] In an optional embodiment, the original image can be obtained by a shooting device, or obtained from a public image set, or obtained by any other means, and the manner of obtaining the original image is not limited herein. It should be noted that the face image and the human body image are obtained by a legal manner with authorization of the user.

[0057] In another optional embodiment, the original image corresponding to the actual use of the image processing model can be obtained for training, so as to improve the recognition accuracy of the image processing model. For example, when the image processing model is a model for processing face images, the original image obtained can be various types of face images obtained from a public image set; when the image processing model is a model for processing landscape images, the original image obtained can be various types of landscape images obtained from a public image set.

[0058] In yet another optional embodiment, when the image processing model is used to recognize various types of images, various types of original images can be obtained as training samples to train the image processing model, so as to improve the application range of the image processing model.

[0059] Step S304: superimposing noise data on the original image to generate an adversarial image.

[0060] The noise data refers to data for disturbing the original image, wherein the noise data can disturb the original pixels in the original image.

[0061] In an optional embodiment, noise data can be superimposed on the original image to obtain an adversarial image of the original image, and the adversarial training process can be realized by the original image and the adversarial image.

[0062] In another optional embodiment, the visual words in the original image can be disturbed to realize the process of superimposing noise data, so as to generate the adversarial image. Optionally, the visual words in the original image can be changed.

[0063] Step S306: discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image.

[0064] The adversarial discrete image contains semantic information.

[0065] In an optional embodiment, a discretization model can be used to discretize the adversarial image in a discrete space to obtain an adversarial discrete image. The discretization model can be VQGAN, but is not limited thereto.

[0066] In another optional embodiment, the adversarial image can be encoded to obtain a feature vector corresponding to the adversarial image, the feature vector can be discretized to obtain a discretized feature, and the discretized feature can be decoded to obtain an adversarial discrete image. By discretizing the adversarial image, an adversarial discrete image with semantic information can be obtained. Optionally, when the feature vector is discretized, the feature vector can be input into a vector quantizer and aligned with a discrete codebook to obtain a visual word corresponding to the feature vector, and the visual word can be decoded to reconstruct the adversarial discrete image. The adversarial discrete image contains visual semantic information.

[0067] In another optional embodiment, when the adversarial image is a face image, the face image can display eyes, eyebrows, nose, mouth, etc. The face image can be discretized to identify information such as eyes, eyebrows, nose, and mouth contained in the face image. The process of face recognition can also be realized for a cartoon face image. By identifying the visual information contained in the discrete image corresponding to the cartoon face image, it can be identified that the cartoon face image contains eyes, eyebrows, nose, mouth, etc.

[0068] In yet another optional embodiment, noise can also be added to the adversarial discrete image, and the visual word representing the eyes in the adversarial discrete image can be changed to a visual word representing glasses. The noise data can also be meaningless data, the noise data can also be a synonym of the original visual word, and the noise data can also be a similar word of the original visual word.

[0069] In step S308, the adversarial discrete image is used to train the image processing model.

[0070] The image processing model is used to process an input image to obtain a processing result of the input image.

[0071] The image processing model described above can be used for classification, detection, etc. The specific processing task of the image processing model is not limited here.

[0072] In an optional embodiment, the adversarial discrete image can be used to perform adversarial training on the image processing model, and the discriminator can be used to determine the adversarial loss of the adversarial discrete image during adversarial training. The adversarial loss can improve the quality of image generation of the image processing model.

[0073] In another optional embodiment, the DAT can utilize the VQGAN to discretize the original image into a series of visual words, and similar to the text task, a certain visual word can have semantic information of a certain part of the image, and the combined semantic information is used for overall recognition. Then, the DAT utilizes the idea of adversarial training to modify the input original image through gradient, change the visual words of the original image, construct an adversarial image that makes the model classification error, and train the robust image processing model by using the adversarial image. By changing the visual words, a new type of discretized adversarial image can be generated. Since the discretized adversarial image is close to the difference between the original sample distribution, this scheme will not affect the original classification effect, and on this basis, the model robustness can be significantly improved. On this basis, other robust training techniques can be combined to use, in order to further improve the robustness of the image processing model.

[0074] In yet another optional embodiment, Adversarial Examples Improve Image Recognition (AdVProp) can be used to change the batch normalization layer (BatchNorm) of the model to two paths, respectively learning the distribution of the original image and the adversarial image, which can effectively improve the recognition ability and robustness of the model.

[0075] Through the above steps, first, the original image is obtained; noise data is superimposed on the original image to generate an adversarial image; the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image; and the image processing model is trained using the adversarial discrete image, wherein the image processing model is used for image processing on the input image to obtain a processing result of the input image, thereby improving the robustness of the image processing model. It is easy to note that by superimposing noise data on the original image, an adversarial image for adversarial training can be obtained, and the adversarial image can be discretized to obtain an adversarial discrete image containing semantic information. By training the image processing model using the adversarial discrete image, the image processing model can achieve visual perception. Since the adversarial discrete image is close to the difference between the original sample distribution, training the image processing model using the generated adversarial discrete image will not affect the original classification effect, and the robustness of the model can be improved, thereby solving the technical problem of low recognition accuracy of the model in the related art.

[0076] In the above embodiments of the present application, the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image, including: utilizing a discretization model to discretize the adversarial image to obtain the adversarial discrete image, wherein the discretization model is used to encode the intermediate features of the discretization model using a preset discrete codebook.

[0077] The discretization model described above can be a VQGAN, which is mainly used to discretize an image into a series of visual words. Its principle is similar to a text task. A certain visual word can have semantic information of a certain part of an image. The combined semantic information is used for overall recognition. It should be noted that the discretization model can be an existing discretization model or a trained discretization model.

[0078] The preset discrete codebook described above is used to represent the correspondence between the intermediate features and the visual words.

[0079] In an optional embodiment, the intermediate features of the discretization model can be encoded by using the preset discrete codebook to obtain encoded features. The encoded features are recombined to obtain the adversarial discrete image, so as to realize the discretization processing of the adversarial image.

[0080] In the above embodiments of the present application, the discretization model comprises an encoding module, a vector quantization module and a decoding module connected in sequence. The discretization model is used to discretize the adversarial image to obtain an adversarial discrete image, comprising: using the encoding module to encode the adversarial image to obtain adversarial encoded features; using the vector quantization module to align the adversarial encoded features with the preset discrete codebook to obtain adversarial visual words; and using the decoding module to generate the adversarial discrete image based on the adversarial visual words.

[0081] The encoding module described above can be an encoder (Encoder, abbreviated as Enc), and can also be other network structures used for encoding.

[0082] The vector quantization module described above can be a vector quantizer, wherein the vector quantizer contains the preset discrete codebook.

[0083] The decoding module described above can be a decoder (Decoder, abbreviated as Dec), and can also be other network structures used for decoding.

[0084] In an optional embodiment, the adversarial image can be image encoded by using the encoding module to obtain adversarial encoded features, wherein the adversarial encoded features can represent visual information of the original image. The adversarial encoded features and the preset discrete codebook can be aligned by using the vector quantization module, so as to give the visual information of the adversarial encoded features semantic information, so as to obtain original visual words. The original visual words can be decoded by using the decoding module to obtain the adversarial discrete image, so that the adversarial discrete image can contain visual word information.

[0085] In the above embodiments of the present application, the method further includes: obtaining a training image; performing discretization processing on the training image by using the discretization model to obtain a training discrete image; performing discrimination on the training discrete image by using the discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image and the discrimination result; and adjusting parameters of the discretization model based on the first loss function.

[0086] The training image described above can be an image to be trained by the discretization model.

[0087] The discriminator described above is used to determine the difference between the training discrete image and the training image. If the difference is too large, it means that the training discrete image is not a real image. If the difference is too small, it means that the training discrete image is a real image.

[0088] In an optional embodiment, the training image can be discretized by using the discretization model to obtain a training discrete image. The training discrete image can be discriminated by using the discriminator to determine whether the training discrete image is a real image. If the discrete image is discriminated as a real image, it means that the discretization processing accuracy of the discretization model is high. If the discrete image is discriminated as not a real image, it means that the discretization processing accuracy of the discretization model is low. A first loss function can be constructed according to the training image, the training discrete image and the discrimination result, wherein the first loss function is used to represent whether the discretization model is accurate. The parameters of the discretization model can be adjusted according to the first loss function to obtain a discretization model with higher accuracy.

[0089] In the above embodiments of the present application, the method further includes: performing discretization processing on the original image to obtain an original discrete image corresponding to the original image; and generating noise data based on the original discrete image.

[0090] In an optional embodiment, the noise data described above needs to be determined according to the gradient of the image processing model for the adversarial discrete image, so as to avoid the noise data being too large to cause the image processing model to be difficult to accurately identify the adversarial discrete image.

[0091] In another optional embodiment, the original image can be encoded to obtain a feature vector corresponding to the original image, the feature vector can be discretized to obtain a discretized feature, and the discretized feature can be decoded to obtain an original discretized image corresponding to the original image. By discretizing the original image, an original discretized image with semantic information can be obtained. Optionally, when the feature vector is discretized, the feature vector can be input into a vector quantizer and aligned with a discrete codebook to obtain a visual word corresponding to the feature vector, and the visual word can be decoded and reconstructed to obtain the original discretized image. The original discretized image contains visual semantic information.

[0092] In the above embodiments of the present application, generating noise data based on the original discretized image includes processing the original discretized image using an image processing model to obtain a target gradient of the original discretized image, and generating the noise data based on the target gradient.

[0093] In an optional embodiment, the original discretized image can be processed using the image processing model to obtain a deviation between the original discretized image and a real image, i.e., the target gradient, and the noise data can be generated according to the target gradient.

[0094] The noise data is α*target gradient, where α is a parameter for controlling the size of the disturbance.

[0095] In the above embodiments of the present application, the original discretized image is processed using the image processing model to obtain a target gradient of the original discretized image, including obtaining an original processing result of the original image, processing the original discretized image using the image processing model to obtain a first processing result of the original image, constructing a second loss function of the image processing model based on the original processing result and the first processing result, and determining the target gradient based on the second loss function.

[0096] In an optional embodiment, the original image can be processed using the image processing model to obtain an original processing result, the original discretized image of the original image can be processed using the image processing model to obtain a first processing result of the original image, a second loss function can be constructed according to the original processing result and the first processing result, so that the target gradient is determined according to the second loss function, the target gradient is used to directly approximate the disturbance on the input original image, and the image is disturbed based on the target gradient, which can avoid a large degree of disturbance, resulting in a large difference between the original discretized image and the adversarial discretized image.

[0097] In the above embodiments of the present application, the image processing model is trained using the adversarial discrete image, including: obtaining an adversarial processing result of the adversarial image; processing the adversarial discrete image using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting the parameters of the image processing model based on the third loss function.

[0098] In an optional embodiment, an original processing result of the original image can be obtained, the image processing model can be used to process the adversarial image and the adversarial discrete image respectively to obtain an adversarial processing result of the adversarial image and a second processing result of the adversarial discrete image, a third loss function of the image processing model can be constructed according to the adversarial processing result and the second processing result, and the parameters of the image processing model can be adjusted using the third loss function to improve the processing capability and robustness of the image processing model.

[0099] In another optional embodiment, since the adversarial discrete image is close to the distribution of the adversarial image and the original image, adjusting the parameters of the image processing model according to the third loss function will not affect the classification effect of the image processing model on the original image, and on this basis, the robustness of the model can be significantly improved.

[0100] Figure 4 is a schematic diagram of image discretization according to an embodiment of the present application. DAT can first encode the original image using an encoder to obtain original encoding features, then align the original encoding features with a preset discrete codebook using a vector quantization module to obtain original visual words, and generate an original discrete image based on the original visual words using a decoding module. Noise data can be superimposed on the original image to obtain an adversarial image. Optionally, the adversarial image can be discretized to obtain an adversarial discrete image, such as Figure 4 In the process of discretizing the adversarial image, 0 and 3 in the original visual words can be changed to 3 and 0 to obtain changed visual words, and the changed visual words can be decoded using a decoder to obtain the adversarial discrete image, Figure 4 The cat in the original image has six whiskers. Since the discretization model may lose data during the discretization process, the original discrete image obtained after discretization has only five whiskers. The adversarial image can be an image obtained by adding noise to the original image. The noise can be to remove one whisker on the left cheek to obtain the adversarial image. The adversarial image can be discretized, and the visual words in the discretization process are changed from 0 and 3 to 3 and 0, and the final adversarial discrete image has only four whiskers. The image processing model can be adversarially trained using the adversarial discrete image to make the image processing model more robust.

[0101] Optionally, noise data can also be superimposed on the adversarial discrete image to obtain a perturbed adversarial discrete image.

[0102] For Figure 4 The discretization process in the formula (1) is represented by L(Q(x+δ), y, θ), where θ is a parameter of the discretization model, δ is an adversarial perturbation obtained based on the image input, Q(x+δ) represents an adversarial discrete image, y represents a discrimination result of the discriminator, and is used to represent whether the adversarial discrete image is a real image, that is, whether the difference between the adversarial discrete image and the original image is too large.

[0103] Figure 5 is a schematic diagram of training a discretization model according to an embodiment of the present application. As Figure 5 shown, the discretization model includes an image encoder Enc, a vector quantizer q, and an image decoder Dec. The original image first enters the encoder to obtain a feature, the feature is aligned through the vector quantizer and a discrete codebook to obtain a discrete feature (i.e., a visual word), and finally the discrete feature is reconstructed back to the original image through the decoder. The training target is to minimize the reconstruction error between the original image and the reconstructed image. In addition, the VQGAN introduces an adversarial loss of the GAN to improve the image generation quality. The whole process is called a discretization process of the image.

[0104] After the discretization model is trained, the input image can be input into the discretization process to obtain a discrete image Q(x), and the classification model is used to calculate a loss L. A target gradient of the discrete image is obtained through gradient back propagation, and the target gradient is directly approximated to a perturbation on the input image, where α is a parameter for controlling the perturbation size. According to α, an adversarial sample Q(x+σ) of the discretization can be obtained, the sample has a perturbed visual word, and can be directly used for training. The model parameters can be adjusted to minimize L until the model converges.

[0105] The present scheme adopts the image discretization technology VQGAN, constructs a discrete image and a corresponding visual semantic word, and migrates the traditional pixel-level adversarial training to the semantic word. Compared with the pixel-level adversarial training, the perturbation on the semantic word is more meaningful, and experiments show that the adversarial sample generated in this way is closer to the distribution of real samples in nature. After the adversarial training, the present scheme can finally achieve that the performance of the model on normal classification is not affected, and the model has robust recognition ability. The DAT in the present scheme does not need to change the network structure, and is suitable for a series of image processing models.

[0106] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all described as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the actions described, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0107] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and the necessary general hardware platform, and of course it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the method of each embodiment of the present application.

[0108] Embodiment 2

[0109] According to the embodiments of the present application, a model training method embodiment is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0110] Figure 6 is a flowchart of an image processing method according to Embodiment 2 of the present application, as shown in Figure 6 the method can include the following steps:

[0111] Step S602, obtaining a target image;

[0112] Step S604, using an image processing model to perform image processing on the target image to obtain a target processing result of the target image.

[0113] Wherein, the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by discretizing the adversarial image, and the adversarial image is generated by superimposing noise data on the original image.

[0114] In the above embodiments of the present application, the method further comprises: using a discretization model to discretize the adversarial image to obtain an adversarial discrete image, wherein the discretization model is used to encode the intermediate features of the discretization model using a preset discrete codebook.

[0115] In the above embodiments of the present application, the discretization model comprises an encoding module, a vector quantization module and a decoding module connected in sequence, wherein the discretization model is used to discretize the original image to obtain the adversarial discrete image, including: using the encoding module to encode the adversarial image to obtain adversarial encoding features; using the vector quantization module to align the adversarial encoding features with a preset discrete codebook to obtain adversarial visual words; and using the decoding module to generate the adversarial discrete image based on the adversarial visual words.

[0116] In the above embodiments of the present application, the method further comprises: obtaining a training image; discretizing the training image using the discretization model to obtain a training discrete image; using the discriminator to discriminate the training discrete image to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image and the discrimination result; and adjusting parameters of the discretization model based on the first loss function.

[0117] In the above embodiments of the present application, the method further comprises: discretizing the original image to obtain an original discrete image corresponding to the original image; and generating noise data based on the original discrete image.

[0118] In the above embodiments of the present application, generating the noise data based on the original discrete image comprises: using an image processing model to process the original discrete image to obtain a target gradient of the original discrete image; and generating the noise data based on the target gradient.

[0119] In the above embodiments of the present application, using the image processing model to process the original discrete image to obtain the target gradient of the original discrete image comprises: obtaining an original processing result of the original image; using the image processing model to process the original discrete image to obtain a first processing result of the original image; constructing a second loss function of the image processing model based on the original processing result and the first processing result; and determining the target gradient based on the second loss function.

[0120] In the above embodiments of the present application, training the image processing model using the adversarial discrete image comprises: obtaining an adversarial processing result of the adversarial image; using the image processing model to process the adversarial discrete image to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting parameters of the image processing model based on the third loss function.

[0121] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0122] Embodiment 3

[0123] According to the embodiments of the present application, a model training method embodiment is also provided. It should be noted that the steps shown in the flowchart of the drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in a different order.

[0124] Figure 7 is a flowchart of a model training method according to Embodiment 3 of the present application, as shown in Figure 7 the method can include the following steps:

[0125] Step S702, in response to the input instruction acting on the operation interface, displaying the original image on the operation interface;

[0126] Step S704, in response to the model training instruction acting on the operation interface, displaying the training result of the image processing model on the operation interface.

[0127] wherein the training result is used to represent that the image processing model is trained, the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by discretizing the adversarial image, and the adversarial image is generated by superimposing noise data on the original image.

[0128] In the above embodiments of the present application, the method further comprises: utilizing the discretization model to discretize the adversarial image to obtain the adversarial discrete image, wherein the discretization model is used to encode the intermediate feature of the discretization model using a preset discrete codebook.

[0129] In the above embodiments of the present application, the discretization model comprises an encoding module, a vector quantization module and a decoding module connected in sequence, wherein the discretization of the original image using the discretization model comprises: utilizing the encoding module to image encode the adversarial image to obtain adversarial encoding features; utilizing the vector quantization module to align the adversarial encoding features with the preset discrete codebook to obtain adversarial visual words; and utilizing the decoding module to generate the adversarial discrete image based on the adversarial visual words.

[0130] In the above embodiments of the present application, the method further comprises: obtaining a training image; utilizing the discretization model to discretize the training image to obtain a training discrete image; utilizing the discriminator to discriminate the training discrete image to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; based on the training image, the training discrete image and the discrimination result, constructing a first loss function of the discretization model; and adjusting the parameters of the discretization model based on the first loss function.

[0131] In the method in the above embodiments of the present application, the method further includes: performing discretization processing on the original image to obtain an original discrete image corresponding to the original image; and generating noise data based on the original discrete image.

[0132] In the method in the above embodiments of the present application, the generating of the noise data based on the original discrete image includes: processing the original discrete image by using an image processing model to obtain a target gradient of the original discrete image; and generating the noise data based on the target gradient.

[0133] In the method in the above embodiments of the present application, the processing of the original discrete image by using the image processing model to obtain the target gradient of the original discrete image includes: obtaining an original processing result of the original image; processing the original discrete image by using the image processing model to obtain a first processing result of the original image; constructing a second loss function of the image processing model based on the original processing result and the first processing result; and determining the target gradient based on the second loss function.

[0134] In the method in the above embodiments of the present application, the training of the image processing model by using the adversarial discrete image includes: obtaining an adversarial processing result of the adversarial image; processing the adversarial discrete image by using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting a parameter of the image processing model based on the third loss function.

[0135] It should be noted that the preferred implementation schemes involved in the above embodiments of the present application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0136] Embodiment 4

[0137] According to the embodiments of the present application, a model training method applied to a virtual reality (VR) device, an augmented reality (AR) device or the like in a virtual reality scene is further provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a group of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described herein can be executed in an order different from that shown herein.

[0138] Figure 8 is a flowchart of a model training method according to Embodiment 4 of the present application. As shown in Figure 8 the method can include the following steps:

[0139] Step S802: displaying an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device.

[0140] Step S804: superimposing noise data on the original image to generate an adversarial image.

[0141] In step S806, the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image.

[0142] In step S808, the adversarial discrete image is used to train an image processing model.

[0143] The image processing model is used to process an input image to obtain a processing result of the input image.

[0144] In step S810, the VR device or the AR device is driven to display the training result of the image processing model.

[0145] The training result is used to represent that the image processing model is trained.

[0146] Optionally, in the embodiment, the above-mentioned model training method can be applied to a hardware environment composed of a server and a virtual reality device. The original image is displayed on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device, the server can be a server corresponding to a media file operator, the network includes but is not limited to a wide area network, a metropolitan area network or a local area network, and the virtual reality device is not limited to a virtual reality helmet, a virtual reality glasses, a virtual reality all-in-one machine, etc.

[0147] Optionally, the virtual reality device includes a memory, a processor and a transmission device. The memory is used to store an application program, which can be used to execute the following steps: displaying an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; superimposing noise data on the original image to generate an adversarial image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model using the adversarial discrete image, wherein the image processing model is used to process an input image to obtain a processing result of the input image; and driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is used to represent that the image processing model is trained.

[0148] It should be noted that the above-mentioned model training method applied in the VR device or the AR device of the embodiment can include Figure 8 The method of the illustrated embodiment is used to achieve the purpose of driving the VR device or the AR device to display the training result.

[0149] Optionally, the processor of the embodiment can call the application program stored in the memory through the transmission device to execute the above-mentioned steps. The transmission device can receive a media file sent by a server through a network, and can also be used for data transmission between the processor and the memory.

[0150] Optionally, in the virtual reality device, a head-mounted display with eye tracking, a screen in the HMD head set for displaying the video picture of the show, an eye tracking module in the HMD for acquiring the real-time motion path of the user's eyeball, a tracking system for tracking the position information and motion information of the user in the real three-dimensional space, a calculation processing unit for acquiring the real-time position and motion information of the user from the tracking system and calculating the three-dimensional coordinates of the user's head in the virtual three-dimensional space and the field of view direction of the user in the virtual three-dimensional space.

[0151] In the embodiments of the present application, the virtual reality device can be connected with a terminal, the terminal is connected with a server through a network, the virtual reality device is not limited to a virtual reality helmet, virtual reality glasses, a virtual reality all-in-one machine, etc., the terminal is not limited to a PC, a mobile phone, a tablet computer, etc., and the server can be a server corresponding to a media file operator. The network includes but is not limited to a wide area network, a metropolitan area network or a local area network.

[0152] Embodiment 5

[0153] According to the embodiments of the present application, a model training method embodiment is also provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0154] Figure 9 is a flowchart of a model training method according to Embodiment 5 of the present application, as shown in Figure 9 The method can include the following steps:

[0155] Step S902, training the image processing model using the adversarial discrete image.

[0156] The image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0157] Step S904, obtaining the original image corresponding to the model training request and the image processing model.

[0158] The image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0159] Step S906, superimposing noise data on the original image to generate an adversarial image.

[0160] Step S908, performing discrete processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image.

[0161] In step S910, the image processing model is trained by using the adversarial discrete image, to obtain a target processing model.

[0162] In step S912, the target processing model is output by calling a second interface.

[0163] The second interface includes a second parameter, and a parameter value of the second parameter is the target processing model.

[0164] In the above embodiments of the present application, the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image, including: discretizing the adversarial image by using a discretization model to obtain the adversarial discrete image, wherein the discretization model is used to encode an intermediate feature of the discretization model by using a preset discrete codebook.

[0165] In the above embodiments of the present application, the discretization model includes: an encoding module, a vector quantization module and a decoding module connected in sequence, wherein the discretization model is used to discretize the original image to obtain the adversarial discrete image, including: encoding the adversarial image by using the encoding module to obtain adversarial encoding features; aligning the adversarial encoding features with the preset discrete codebook by using the vector quantization module to obtain adversarial visual words; and generating the adversarial discrete image based on the adversarial visual words by using the decoding module.

[0166] In the above embodiments of the present application, the method further includes: obtaining a training image; discretizing the training image by using the discretization model to obtain a training discrete image; discriminating the training discrete image by using a discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image and the discrimination result; and adjusting parameters of the discretization model based on the first loss function.

[0167] In the above embodiments of the present application, the method further includes: discretizing the original image to obtain an original discrete image corresponding to the original image; and generating noise data based on the original discrete image.

[0168] In the above embodiments of the present application, the noise data is generated based on the original discrete image, including: processing the original discrete image by using the image processing model to obtain a target gradient of the original discrete image; and generating the noise data based on the target gradient.

[0169] In the above embodiments of the present application, the original discrete image is processed by using the image processing model to obtain the target gradient of the original discrete image, including: obtaining an original processing result of the original image; processing the original discrete image by using the image processing model to obtain a first processing result of the original image; constructing a second loss function of the image processing model based on the original processing result and the first processing result; and determining the target gradient based on the second loss function.

[0170] In the above embodiment of the present application, the image processing model is trained using the adversarial discrete image, including: obtaining an adversarial processing result of the adversarial image; processing the adversarial discrete image using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting the parameters of the image processing model based on the third loss function.

[0171] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios and implementation processes as the schemes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0172] Embodiment 6

[0173] According to the embodiments of the present application, a model training device for implementing the above-mentioned model training method is also provided, Figure 10 is a schematic diagram of a model training device according to Embodiment 6 of the present application, as Figure 10 shown, the device 1000 includes an acquisition module 1002, a processing module 1004, a superposition module 1006, and a training module 1008.

[0174] The acquisition module is configured to acquire an original image; the processing module is configured to superimpose noise data on the original image to generate an adversarial image; the superposition module is configured to perform a discretization process on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and the training module is configured to train an image processing model using the adversarial discrete image, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image.

[0175] It should be noted that the acquisition module 1002, the processing module 1004, the superposition module 1006, and the training module 1008 correspond to steps S302 to S308 in Embodiment 1, and the four modules have the same instances and application scenarios as the corresponding steps, but are not limited to the contents disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in the computer terminal 10 provided in Embodiment 1.

[0176] In the above embodiments of the present application, the processing module is further configured to train the image processing model using the adversarial discrete image, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image.

[0177] In the above embodiments of the present application, the discretization model includes an encoding module, a vector quantization module, and a decoding module connected in sequence, wherein the processing module includes an encoding unit, an alignment unit, and a first generation unit.

[0178] The encoding unit is configured to encode the adversarial image to obtain adversarial encoded features by using an encoding module; the alignment unit is configured to align the adversarial encoded features with a preset discrete codebook to obtain adversarial visual words by using a vector quantization module; and the first generation unit is configured to generate an adversarial discrete image based on the adversarial visual words by using a decoding module.

[0179] In the above embodiments of the present application, the device further comprises a discrimination module, a construction module and an adjustment module.

[0180] The acquisition module is further configured to acquire a training image; the processing module is further configured to perform discretization processing on the training image by using a discretization model to obtain a training discrete image; the discrimination module is configured to discriminate the training discrete image by using a discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; the construction module is configured to construct a first loss function of the discretization model based on the training image, the training discrete image and the discrimination result; and the adjustment module is configured to adjust parameters of the discretization model based on the first loss function.

[0181] In the above embodiments of the present application, the device further comprises a generation module.

[0182] The processing module is further configured to perform discretization processing on the original image to obtain an original discrete image corresponding to the original image; and the generation module is further configured to generate noise data based on the original discrete image.

[0183] In the above embodiments of the present application, the generation module comprises a first processing unit and a second generation unit.

[0184] The first processing unit is configured to perform discretization processing on the original image to obtain an original discrete image corresponding to the original image; and the second generation unit is configured to generate noise data based on the original discrete image.

[0185] In the above embodiments of the present application, the processing unit comprises an acquisition subunit, a processing subunit, a construction subunit and a determination subunit.

[0186] The acquisition subunit is configured to acquire an original processing result of the original image; the processing subunit is configured to process the original discrete image by using an image processing model to obtain a first processing result of the original image; the construction subunit is configured to construct a second loss function of the image processing model based on the original processing result and the first processing result; and the determination subunit is configured to determine a target gradient based on the second loss function.

[0187] In the above embodiments of the present application, the training module comprises an acquisition unit, a second processing unit, a construction unit and an adjustment unit.

[0188] The acquisition unit is configured to acquire an adversarial processing result of the adversarial image; the second processing unit is configured to process the adversarial discrete image by using the image processing model to obtain a second processing result of the adversarial image; the construction unit is configured to construct a third loss function of the image processing model based on the adversarial processing result and the second processing result; and the adjustment unit is configured to adjust parameters of the image processing model based on the third loss function.

[0189] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario, and implementation process as provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0190] Embodiment 7

[0191] According to the embodiments of the present application, an image processing device for implementing the above image processing method is further provided, Figure 11 is a schematic diagram of an image processing device according to Embodiment 7 of the present application, as Figure 11 shown, the device 1100 includes an acquisition module 1102 and a processing module 1104.

[0192] The acquisition module is configured to acquire a target image; and the processing module is configured to perform image processing on the target image by using an image processing model to obtain a target processing result of the target image, wherein the image processing model is obtained by training an adversarial discrete image, the adversarial discrete image is obtained by performing discretization processing on an adversarial image, and the adversarial image is generated by superimposing noise data on an original image.

[0193] It should be noted that the acquisition module 1102 and the processing module 1104 correspond to steps S602 to S604 in Embodiment 2, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules as part of the device can run in the computer terminal 10 provided in Embodiment 1.

[0194] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario, and implementation process as provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0195] Embodiment 8

[0196] According to the embodiments of the present application, a model training device for implementing the above model training method is further provided, Figure 12 is a schematic diagram of a model training device according to Embodiment 8 of the present application, as Figure 12 shown, the device 1200 includes a first display module 1202 and a first display module 1204.

[0197] The first display module is configured to display the original image on the operation interface in response to an input instruction acting on the operation interface.

[0198] It should be noted that the first display module 1202 and the first display module 1204 correspond to steps S702 to S704 in Embodiment 3, and the two modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can run in the computer terminal 10 provided in Embodiment 1 as part of the device.

[0199] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same application scenarios, implementation processes as the scheme provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0200] Embodiment 9

[0201] According to the embodiments of the present application, a model training device for implementing the above model training method is further provided, Figure 13 is a schematic diagram of a model training device according to Embodiment 9 of the present application, as Figure 13 shown, the device 1300 includes a display module 1302, a superposition module 1304, a processing module 1306, a training module 1308, and a driving module 1310.

[0202] The display module is configured to display an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device. The superposition module is configured to superimpose noise data on the original image to generate an adversarial image. The processing module is configured to discretize the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image. The training module is configured to train an image processing model using the adversarial discrete image, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image. The driving module is configured to drive the VR device or the AR device to display a training result of the image processing model, wherein the training result is configured to indicate that the training of the image processing model is completed.

[0203] It should be noted that the display module 1302, the superimposition module 1304, the processing module 1306, the training module 1308, and the driving module 1310 correspond to steps S802 to S810 in Embodiment 4, and the five modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can run in the computer terminal 10 provided in Embodiment 1 as part of the device.

[0204] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario, and implementation process as Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0205] Embodiment 10

[0206] According to the embodiments of the present application, a model training device for implementing the above model training method is further provided, Figure 14 is a schematic diagram of a model training device according to Embodiment 10 of the present application, as Figure 14 shown, the device 1400 includes a calling module 1402, an obtaining module 1404, a superimposition module 1406, a processing module 1408, a training module 1410, and an output module 1412.

[0207] The calling module is configured to obtain a model training request by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the model training request. The obtaining module is configured to obtain an original image and an image processing model corresponding to the model training request, wherein the image processing model is configured to perform image processing on an input image to obtain a processing result of the input image. The superimposition module is configured to superimpose noise data on the original image to generate an adversarial image. The processing module is configured to perform discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image. The training module is configured to train the image processing model using the adversarial discrete image to obtain a target processing model. The output module is configured to output the target processing model by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing model.

[0208] It should be noted that the calling module 1402, the obtaining module 1404, the superimposition module 1406, the processing module 1408, the training module 1410, and the output module 1412 correspond to steps S902 to S912 in Embodiment 5, and the six modules have the same instances and application scenarios as the corresponding steps, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules can run in the computer terminal 10 provided in Embodiment 1 as part of the device.

[0209] It should be noted that the preferred embodiments involved in the above embodiments of the present application have the same scheme, application scenario and implementation process as provided in Embodiment 1, but are not limited to the scheme provided in Embodiment 1.

[0210] Embodiment 11

[0211] The embodiments of the present application can provide an electronic device, wherein the electronic device can be an AR / VR device, which can be any AR / VR device in a group of AR / VR devices. Alternatively, in the present embodiment, the AR / VR device can be replaced by a terminal device such as a mobile terminal.

[0212] Alternatively, in the present embodiment, the AR / VR device can be located in at least one network device of a plurality of network devices of a computer network.

[0213] In the present embodiment, the AR / VR device can execute program codes of the following steps in the model training method: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0214] Alternatively, Figure 15 is a structural block diagram of a computer terminal according to an embodiment of the present application. As shown in Figure 15 The computer terminal A can include one or more (only one is shown in the figure) processors, a memory.

[0215] The memory can be used to store software programs and modules, such as program instructions / modules corresponding to the model training method and device in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, i.e., implements the above-mentioned model training method. The memory can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory can further include a memory remotely arranged with respect to the processor, which can be connected to the terminal A through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0216] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0217] Optionally, the processor can further execute program codes of the following steps: performing discretization processing on the adversarial image using a discretization model to obtain the adversarial discrete image, wherein the discretization model is used to encode intermediate features of the discretization model using a preset discrete codebook.

[0218] Optionally, the processor can further execute program codes of the following steps: performing image encoding on the adversarial image using an encoding module to obtain adversarial encoding features; aligning the adversarial encoding features and the preset discrete codebook using a vector quantization module to obtain adversarial visual words; and generating the adversarial discrete image based on the adversarial visual words using a decoding module.

[0219] Optionally, the processor can further execute program codes of the following steps: obtaining a training image; performing discretization processing on the training image using the discretization model to obtain a training discrete image; performing discrimination on the training discrete image using a discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image, and the discrimination result; and adjusting parameters of the discretization model based on the first loss function.

[0220] Optionally, the processor can further execute program codes of the following steps: performing discretization processing on the original image to obtain an original discrete image corresponding to the original image; and generating the noise data based on the original discrete image.

[0221] Optionally, the processor can further execute program codes of the following steps: processing the original discrete image using the image processing model to obtain a target gradient of the original discrete image; and generating the noise data based on the target gradient.

[0222] Optionally, the processor can further execute program codes of the following steps: obtaining an original processing result of the original image; processing the original discrete image using the image processing model to obtain a first processing result of the original image; constructing a second loss function of the image processing model based on the original processing result and the first processing result; and determining the target gradient based on the second loss function.

[0223] Optionally, the processor can further execute program codes of the following steps: obtaining an adversarial processing result of the adversarial image; processing the adversarial discrete image by using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting parameters of the image processing model based on the third loss function.

[0224] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: obtaining a target image; and processing the target image by using the image processing model to obtain a target processing result of the target image, wherein the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by discretizing the adversarial image, and the adversarial image is generated by superimposing noise data on the original image.

[0225] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: displaying the original image on the operation interface in response to an input instruction acting on the operation interface; and displaying a training result of the image processing model on the operation interface in response to a model training instruction acting on the operation interface, wherein the training result is used to represent that the image processing model is trained, the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by discretizing the adversarial image, and the adversarial image is generated by superimposing noise data on the original image.

[0226] The processor can call information and application programs stored in the memory through the transmission device to execute the following steps: displaying the original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; generating an adversarial image by superimposing noise data on the original image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training the image processing model by using the adversarial discrete image, wherein the image processing model is used to process an input image to obtain a processing result of the input image; and driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is used to represent that the image processing model is trained.

[0227] The processor can call information and an application stored in the memory through the transmission device to perform the following steps: obtaining a model training request by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter is the model training request; obtaining an original image and an image processing model corresponding to the model training request, wherein the image processing model is used for image processing on an input image to obtain a processing result of the input image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training the image processing model by using the adversarial discrete image to obtain a target processing model; and outputting the target processing model by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the target processing model.

[0228] In the embodiments of the present application, an original image is first obtained; noise data is superimposed on the original image to generate an adversarial image; discretization processing is performed on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and the image processing model is trained by using the adversarial discrete image, wherein the image processing model is used for image processing on an input image to obtain a processing result of the input image, thereby improving the robustness of the image processing model. It is easy to note that the adversarial image used for adversarial training can be obtained by superimposing noise data on the original image, the adversarial discrete image containing semantic information can be obtained by performing discretization processing on the adversarial image, the image processing model can reach the effect of visual perception by training the image processing model by using the adversarial discrete image, and the original classification effect will not be affected and the robustness of the model can be improved by training the image processing model by using the generated adversarial discrete image, thereby solving the technical problem of low recognition accuracy of the model in the related art.

[0229] Those skilled in the art can understand that, Figure 15 The structure shown is only schematic, and the computer terminal can also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, a Mobile Internet Device (MID), a PAD, and the like. Figure 15 It does not limit the structure of the electronic device. For example, the computer terminal 10 can further include more or fewer components (such as a network interface, a display device, etc.) than those shown in Figure 15 or have a different configuration than that shown in Figure 15 .

[0230] Those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0231] Embodiment 12

[0232] The embodiments of the present application also provide a computer readable storage medium. Optionally, in the embodiment, the computer readable storage medium can be used to save the program code executed by the model training method provided in the embodiment 1.

[0233] Optionally, in the embodiment, the computer readable storage medium can be located in any one of the computer terminals in the AR / VR device terminal group in the AR / VR device network, or in any one of the mobile terminals in the mobile terminal group.

[0234] Optionally, in the embodiment, the computer readable storage medium is configured to store program code for performing the following steps: obtaining an original image; superimposing noise data on the original image to generate an adversarial image; performing discretization processing on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; and training an image processing model by using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image.

[0235] Optionally, the storage medium is further configured to store program code for performing the following steps: performing discretization processing on the adversarial image by using a discretization model to obtain an adversarial discrete image, wherein the discretization model is used to encode intermediate features of the discretization model by using a preset discrete codebook.

[0236] Optionally, the storage medium is further configured to store program code for performing the following steps: performing image encoding on the adversarial image by using an encoding module to obtain adversarial encoding features; aligning the adversarial encoding features and the preset discrete codebook by using a vector quantization module to obtain adversarial visual words; and generating the adversarial discrete image based on the adversarial visual words by using a decoding module.

[0237] Optionally, the storage medium is further configured to store program code for performing the following steps: obtaining a training image; performing discretization processing on the training image by using the discretization model to obtain a training discrete image; performing discrimination on the training discrete image by using the discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image and the discrimination result; and adjusting parameters of the discretization model based on the first loss function.

[0238] Optionally, the storage medium is further configured to store program code for performing the following steps: performing discretization processing on the original image to obtain an original discrete image corresponding to the original image; and generating the noise data based on the original discrete image.

[0239] Optionally, the storage medium is further configured to store program code for performing the following steps: performing processing on the original discrete image by using the image processing model to obtain a target gradient of the original discrete image; and generating the noise data based on the target gradient.

[0240] Optionally, the storage medium is further configured to store program code for performing the following steps: obtaining an original processing result of the original image; performing processing on the original discrete image by using the image processing model to obtain a first processing result of the original image; constructing a second loss function of the image processing model based on the original processing result and the first processing result; and determining the target gradient based on the second loss function.

[0241] Optionally, the storage medium is further configured to store program code for performing the following steps: obtaining an adversarial processing result of the adversarial image; performing processing on the adversarial discrete image by using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; and adjusting parameters of the image processing model based on the third loss function.

[0242] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a target image; and performing image processing on the target image by using the image processing model to obtain a target processing result of the target image, wherein the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by performing discretization processing on the adversarial image, and the adversarial image is generated by superimposing the noise data on the original image.

[0243] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: in response to an input instruction acting on the operation interface, displaying the original image on the operation interface; in response to a model training instruction acting on the operation interface, displaying a training result of the image processing model on the operation interface, wherein the training result is used to indicate that the image processing model is trained, and the image processing model is obtained by training the adversarial discrete image, the adversarial discrete image is obtained by discretizing the adversarial image, and the adversarial image is generated by superimposing noise data on the original image.

[0244] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: displaying the original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; superimposing noise data on the original image to generate an adversarial image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image; and driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is used to indicate that the image processing model is trained.

[0245] Optionally, in the embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the model training request; obtaining an original image and an image processing model corresponding to the model training request, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image; superimposing noise data on the original image to generate an adversarial image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training the image processing model using the adversarial discrete image to obtain a target processing model; and outputting the target processing model by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the target processing model.

[0246] In the embodiments of the present application, first, an original image is acquired; noise data is superimposed on the original image to generate an adversarial image; the adversarial image is discretized to obtain an adversarial discrete image corresponding to the adversarial image; and the adversarial discrete image is used to train an image processing model, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image, thereby improving the robustness of the image processing model. It is easy to note that the adversarial image used for adversarial training can be obtained by superimposing noise data on the original image, the adversarial discrete image containing semantic information can be obtained by discretizing the adversarial image, the image processing model can achieve visual perception by training the image processing model using the adversarial discrete image, and the original classification effect will not be affected by training the image processing model using the generated adversarial discrete image, and the robustness of the model can be improved, thereby solving the technical problem of low recognition accuracy of the model in the related art.

[0247] The above sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0248] In the above embodiments of the present application, the description of each embodiment has its own emphasis, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0249] In the several embodiments provided by the present application, it should be understood that the disclosed technology can be implemented in other ways. Of course, the embodiment described above is only a schematic and illustrative, the division of the units is only a logical function division, and there can be another division manner in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, unit or module, and can be electrical or other forms.

[0250] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0251] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The above integrated unit can be realized in the form of hardware or in the form of software functional unit.

[0252] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or say the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0253] The above is only the preferred embodiment of the present application. It should be noted that for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, which should be considered as the protection scope of the present application.

Claims

1. A model training method, characterized in that, The method comprises: obtaining an original image; superimposing noise data on the original image to generate an adversarial image, wherein the noise data is generated based on an original discrete image obtained by discretizing the original image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model using the adversarial discrete image, wherein the image processing model is used to process an input image to obtain a processing result of the input image.

2. The method of claim 1, wherein, The method further comprises: obtaining a training image; 3. The method of claim 2, wherein, discretizing the training image using the discretization model to obtain a training discrete image; discriminating the training discrete image using a discriminator to obtain a discrimination result, wherein the discrimination result is used to represent whether the training discrete image is a real image; constructing a first loss function of the discretization model based on the training image, the training discrete image, and the discrimination result; adjusting parameters of the discretization model based on the first loss function.

4. The method of claim 2, wherein, The method further comprises: processing the original discrete image using the image processing model to obtain a target gradient of the original discrete image; generating the noise data based on the target gradient. The method further comprises: obtaining an original processing result of the original image; processing the original discrete image using the image processing model to obtain a first processing result of the original image; 5. The method of claim 1, wherein, constructing a second loss function of the image processing model based on the original processing result and the first processing result; determining the target gradient based on the second loss function. The method further comprises:

6. The method of claim 5, wherein, obtaining an adversarial processing result of the adversarial image; processing the adversarial discrete image using the image processing model to obtain a second processing result of the adversarial image; constructing a third loss function of the image processing model based on the adversarial processing result and the second processing result; ​ ​ 7. The method of claim 1, wherein, ​ ​ ​ ​ Adjust parameters of the image processing model based on the third loss function.

8. An image processing method characterized by, The method comprises: obtaining a target image; performing image processing on the target image by using an image processing model to obtain a target processing result of the target image, wherein the image processing model is obtained by training an adversarial discrete image, the adversarial discrete image is obtained by discretizing an adversarial image, the adversarial image is generated by superimposing noise data on an original image, the noise data is generated based on an original discrete image, and the original discrete image is obtained by discretizing the original image.

9. A model training method, comprising: The method comprises: displaying an original image on an operation interface in response to an input instruction acting on the operation interface; displaying a training result of an image processing model on the operation interface in response to a model training instruction acting on the operation interface, wherein the training result is used to represent that the image processing model is trained, the image processing model is obtained by training an adversarial discrete image, the adversarial discrete image is obtained by discretizing an adversarial image, the adversarial image is generated by superimposing noise data on an original image, the noise data is generated based on an original discrete image, and the original discrete image is obtained by discretizing the original image.

10. A model training method, comprising: The method comprises: displaying an original image on a presentation screen of a virtual reality (VR) device or an augmented reality (AR) device; superimposing noise data on the original image to generate an adversarial image, wherein the noise data is generated based on an original discrete image, and the original discrete image is obtained by discretizing the original image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; training an image processing model by using the adversarial discrete image, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image; driving the VR device or the AR device to display a training result of the image processing model, wherein the training result is used to represent that the image processing model is trained.

11. A model training method, comprising: The method comprises: obtaining a model training request by calling a first interface, wherein the first interface comprises a first parameter, and a parameter value of the first parameter is the model training request; obtaining an original image and an image processing model corresponding to the model training request, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image; superimposing noise data on the original image to generate an adversarial image, wherein the noise data is generated based on an original discrete image, and the original discrete image is obtained by discretizing the original image; discretizing the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image, and training the image processing model by using the adversarial discrete image to obtain a target processing model; outputting the target processing model by calling a second interface, wherein the second interface comprises a second parameter, and a parameter value of the second parameter is the target processing model.

12. A computer-readable storage medium, characterized in that, The computer readable storage medium comprises a stored program, wherein the program controls a device in which the computer readable storage medium is located to execute the method of any one of claims 1 to 11 when the program is running.

13. An electronic device, comprising: Comprise: A processor; A memory connected with the processor, used to provide the processor with instructions for processing the following processing steps: acquiring an original image; Superimposing noise data on the original image to generate an adversarial image; Discretization processing is performed on the adversarial image to obtain an adversarial discrete image corresponding to the adversarial image; the adversarial discrete image is used to train an image processing model, wherein the image processing model is used to perform image processing on an input image to obtain a processing result of the input image, the noise data is generated based on an original discrete image, and the original discrete image is obtained by discretizing the original image.

Images