Image processing method, apparatus and system and device

Abstract

The present application provides an image processing method, apparatus and system and a device. The method comprises: acquiring a privacy protection image corresponding to an original sensitive image, wherein the privacy protection image is an image obtained by performing a scrambling operation on the original sensitive image by using scrambling parameters; and inputting the privacy protection image to a target network model to obtain an image processing result corresponding to the privacy protection image, wherein weights in the target network model are scrambled by using the scrambling parameters.

Description

Image processing method, device, equipment and system Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, apparatus, device and system. Background Art

[0002] The client (such as a camera, mobile terminal, etc.) can obtain the original image and send it to the server. The server performs artificial intelligence processing on the original image based on the network model to obtain the image processing results of the original image (such as image detection results, image segmentation results, image recognition results, etc.).

[0003] Summary of the Invention

[0004] The present invention provides an image processing method for use on a server. The method includes:

[0005] Obtaining a privacy-preserving image corresponding to the original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter;

[0006] The privacy-preserving image is input into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0007] The present invention provides an image processing method, which is applied to a client. The method includes:

[0008] A pixel scrambling operation and / or an image block scrambling operation is performed on an original sensitive image using scrambling parameters to obtain a privacy-preserving image; wherein the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to perform a scrambling operation on multiple pixel values ​​within a corresponding image block, and the image block scrambling operation is used to perform a scrambling operation on multiple image blocks; the privacy-preserving image is sent to a server, so that the server inputs the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0009] The present invention provides an image processing device for use on a server. The device includes:

[0010] An acquisition module, configured to acquire a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using scrambling parameters;

[0011] A processing module is configured to input the privacy-preserving image into a target network model, obtain an image processing result corresponding to the privacy-preserving image, and output the image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0012] An embodiment of the present application provides a server device, including a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned image processing method.

[0013] An embodiment of the present application provides an image processing system, the system comprising:

[0014] The client is configured to perform a scrambling operation on the original sensitive image using a scrambling parameter to obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to the server;

[0015] The server is configured to obtain a privacy-preserving image corresponding to the original sensitive image, and input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0016] In the embodiments of the present application, scrambling parameters can be used to scramble the original sensitive image to obtain a privacy-preserving image. Furthermore, the client sends the privacy-preserving image to the server. Even if the privacy-preserving image is intercepted by an attacker, the attacker cannot obtain sensitive information from the privacy-preserving image, thereby preventing the leakage of sensitive information and, consequently, the user's private information. In other words, the embodiments of the present application can protect the privacy of the original sensitive image without the risk of privacy leakage, thus providing a high level of privacy protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments of the present application or the description of the prior art. Obviously, the drawings described below are only some of the embodiments described in this application. For those skilled in the art, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0018] FIG1 is a flow chart of an image processing method in one embodiment of the present application.

[0019] FIG2 is a schematic diagram of a scrambling operation in one embodiment of the present application.

[0020] FIG3 is a schematic structural diagram of a target network model in one embodiment of the present application.

[0021] FIG4 is a schematic structural diagram of an image processing system in one embodiment of the present application.

[0022] FIG5 is a schematic structural diagram of an image processing device in one embodiment of the present application.

[0023] FIG6 is a hardware structure diagram of a server device in one embodiment of the present application. DETAILED DESCRIPTION

[0024] The terms used in the embodiments of this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms "a," "the," and "the" used in this application and claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0025] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. In addition, depending on the context, the word "if" used may be interpreted as "at the time of" or "when" or "in response to determining".

[0026] If the original image obtained by the client is a sensitive image (such as a face image, a body image, etc.), then if the original image is intercepted by an attacker during transmission, the attacker can obtain sensitive information from the original image, causing the leakage of sensitive information, thereby leaking user privacy information and causing huge losses to the user.

[0027] In an embodiment of the present application, an image processing method is proposed, which can be applied to a server. Referring to FIG1 , which is a flow chart of the image processing method, the method may include steps 101 and 102 .

[0028] In step 101, a privacy-preserving image corresponding to an original sensitive image is obtained. The privacy-preserving image may be an image obtained by scrambling the original sensitive image using a scrambling parameter. For example, a pixel scrambling operation and / or an image block scrambling operation may be performed on the original sensitive image using the scrambling parameter to obtain the privacy-preserving image. The original sensitive image includes multiple image blocks, and the pixel scrambling operation is used to scramble multiple pixel values ​​within a corresponding image block, while the image block scrambling operation is used to scramble multiple image blocks.

[0029] The scrambling parameters include a pixel value protection key and an image block protection key. The original sensitive image can be divided into M*N image blocks, and multiple target image blocks to be protected are selected from the M*N image blocks, where M and N are both positive integers. For each target image block, the pixel value protection key corresponding to the target image block is obtained, the first scrambling order corresponding to the pixel value protection key is determined, and the multiple pixel values ​​in the target image block are scrambled based on the first scrambling order to obtain an intermediate image. The image block protection key corresponding to the original sensitive image is obtained, the second scrambling order corresponding to the image block protection key is determined, and the multiple target image blocks in the intermediate image are scrambled based on the second scrambling order to obtain a privacy-protected image.

[0030] For example, the client can use scrambling parameters to perform scrambling operations on the original sensitive image, obtain the privacy-protected image corresponding to the original sensitive image, and send the privacy-protected image to the server.

[0031] Exemplarily, the scrambling parameters may include a pixel value protection key and / or an image block protection key. The client may perform a pixel scrambling operation on the original sensitive image based on the pixel value protection key, and / or the client may perform an image block scrambling operation on the original sensitive image based on the image block protection key, thereby obtaining a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image may include multiple image blocks, and the pixel scrambling operation may be used to perform a scrambling operation on multiple pixel values ​​within the corresponding image blocks, and the image block scrambling operation may be used to perform a scrambling operation on multiple image blocks of the original sensitive image.

[0032] In a possible implementation, the client may divide the original sensitive image into M*N image blocks, and select multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers. For example, all the image blocks in the M*N image blocks are used as target image blocks, or some of the image blocks in the M*N image blocks are used as target image blocks. For each target image block, obtain the pixel value protection key corresponding to the target image block, determine the first scrambling order corresponding to the pixel value protection key, and perform a scrambling operation on the multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image (i.e., the image after pixel scrambling operation is performed on all target image blocks). Obtain the image block protection key corresponding to the original sensitive image, determine the second scrambling order corresponding to the image block protection key, and perform a scrambling operation on the multiple target image blocks in the intermediate image based on the second scrambling order to obtain the privacy-protected image.

[0033] Exemplarily, the pixel value protection keys corresponding to different target image blocks may be the same or different, and the image block protection keys corresponding to different original sensitive images may be the same or different.

[0034] In step 102, the privacy-preserving image is input into the target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using scrambling parameters.

[0035] In one possible implementation, the server can obtain scrambling parameters from the client and use them to scramble the weights in the trained initial network model to obtain a target network model. Alternatively, the server can obtain target weight information from the client and generate a target network model based on the target weight information; the target weight information is the weight information in the candidate network model, and the candidate network model is obtained by the client using the scrambling parameters to scramble the weights in the initial network model.

[0036] Exemplarily, the scrambling parameters are used to perform a scrambling operation on the weights in the trained initial network model to obtain a target network model, which may include but is not limited to: if the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the corresponding image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain a target network model.

[0037] Exemplarily, determining the first scrambling order corresponding to the pixel value protection key may include, but is not limited to: querying a configured first mapping relationship using the pixel value protection key to obtain the first scrambling order corresponding to the pixel value protection key. The first scrambling order may be used to represent a correspondence between pixel value positions before and after the scrambling operation; and the first mapping relationship may be used to record the correspondence between the pixel value protection key and the scrambling order.

[0038] Exemplarily, determining the second scrambling order corresponding to the image block protection key may include, but is not limited to: querying a configured second mapping relationship using the image block protection key to obtain the second scrambling order corresponding to the image block protection key. The second scrambling order may be used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; and the second mapping relationship may be used to record the correspondence between the image block protection key and the scrambling order.

[0039] In one possible implementation, scrambling the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain a target network model may include, but is not limited to: determining first-category weights and second-category weights in the initial network model, scrambling the first-category weights in the initial network model based on the first scrambling order, and scrambling the second-category weights in the initial network model based on the second scrambling order to obtain the target network model. The first-category weights may include, but are not limited to, weights in the initial network model that operate on pixel values ​​within an image block, and the second-category weights may include, but are not limited to, weights in the initial network model that are related to position information of the image block.

[0040] In one possible embodiment, the initial network model may be a network model including one fully connected layer; or, the initial network model may be a network model including multiple fully connected layers; or, the initial network model may be a network model of visual self-attention (Vision Transformer); or, the initial network model may be a network model of visual self-attention without position encoding; or, the initial network model may be a network model of a multi-layer perceptron and a mixer (MLP-Mixer). The weights in the initial network model are scrambled based on the first scrambling order and the second scrambling order to obtain a target network model, which may include but is not limited to: if the initial network model is a network model of visual self-attention, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then: the weights in the fully connected layer may be scrambled based on the first scrambling order, and the weights in the position encoding layer may be scrambled based on the second scrambling order to obtain the target network model.

[0041] In the embodiments of the present application, scrambling parameters can be used to scramble the original sensitive image to obtain a privacy-preserving image. Furthermore, the client sends the privacy-preserving image to the server. Even if the privacy-preserving image is intercepted by an attacker, the attacker cannot obtain sensitive information from the privacy-preserving image, thereby preventing the leakage of sensitive information and, consequently, the user's private information. In other words, the embodiments of the present application can protect the privacy of the original sensitive image without the risk of privacy leakage, thus providing a high level of privacy protection.

[0042] Based on the same inventive concept as the above-mentioned image processing method, an image processing method is proposed in an embodiment of the present application, which can be applied to a client. The method may include: using scrambling parameters to perform a pixel scrambling operation and / or an image block scrambling operation on an original sensitive image to obtain a privacy-preserving image; wherein the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to perform a scrambling operation on multiple pixel values ​​in the corresponding image block, and the image block scrambling operation is used to perform a scrambling operation on multiple image blocks; sending the privacy-preserving image to a server, so that the server inputs the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0043] Exemplarily, the scrambling parameters may include a pixel value protection key and an image block protection key. Using the scrambling parameters to perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image to obtain a privacy-protected image may include but is not limited to: dividing the original sensitive image into M*N image blocks, and selecting multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers; for each target image block, obtaining the pixel value protection key corresponding to the target image block, determining a first scrambling order corresponding to the pixel value protection key, and performing a scrambling operation on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image; obtaining an image block protection key corresponding to the original sensitive image, determining a second scrambling order corresponding to the image block protection key, and performing a scrambling operation on multiple target image blocks in the intermediate image based on the second scrambling order to obtain a privacy-protected image.

[0044] The above technical solutions of the embodiments of the present application are described below in conjunction with specific application scenarios.

[0045] An image processing system (such as an image recognition system) uses artificial intelligence technology to automatically identify (detect, segment, etc.) relevant information from an image. It performs artificial intelligence processing on images based on a network model to produce image processing results (such as image detection results, image segmentation results, image recognition results, etc.). The image processing system may include, but is not limited to, the following processes: image acquisition (or image acquisition), image transmission, image preprocessing, image recognition, and post-processing.

[0046] During image acquisition, the client can capture the original image. During image transmission, the client can send the original image to the server. During image preprocessing, the server can perform preprocessing on the original image, such as image scaling and image flipping. During image recognition, the server can perform artificial intelligence processing on the original image based on the network model to obtain the image processing result of the original image. During post-processing, the server performs post-processing on the original image based on the image processing result.

[0047] During image transmission, the client can send the original image to the server. If the original image is sensitive (such as a face image or a human body image), then if the original image is intercepted by an attacker during transmission, the attacker can obtain sensitive information from the original image, causing the leakage of sensitive information, which can easily leak user privacy information and cause huge losses to the user. Even if the client uses a cryptographic algorithm to encrypt the original image (such as public key encryption or private key encryption), the server needs to decrypt the encrypted original image during the image preprocessing or image recognition process. After the encrypted original image is decrypted, if the attacker intercepts the decrypted original image, it will also cause the leakage of sensitive information.

[0048] In response to the above findings, this embodiment proposes an image processing method that uses scrambling parameters to scramble the original sensitive image to obtain a privacy-preserving image. The client sends the privacy-preserving image to the server, thereby preventing the leakage of sensitive information and user privacy information.

[0049] In this embodiment, the image processing system (i.e., an image processing system based on image privacy protection) may include but is not limited to the following processes: image acquisition (or image acquisition) process, image privacy protection process, image transmission process, image preprocessing process, image recognition process, and post-processing process.

[0050] Exemplarily, during the image acquisition process, the client can acquire the original image. For ease of distinction, the original image can be recorded as an original sensitive image, indicating that the original image is a sensitive image, such as a face image, a body image, etc., and there is no restriction on the type of the original sensitive image. During the image privacy protection process, the client uses scrambling parameters to perform scrambling operations on the original sensitive image. For example, the client can use scrambling parameters to perform pixel scrambling operations and / or image block scrambling operations on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. During the image transmission process, the client sends the privacy-protected image to the server. During the image preprocessing process, the server can preprocess the privacy-protected image, such as performing image scaling, image flipping, and other preprocessing. During the image recognition process, the server performs artificial intelligence processing on the privacy-protected image based on the network model, thereby obtaining an image processing result of the privacy-protected image. During the post-processing process, the server implements post-processing of the privacy-protected image based on the image processing result.

[0051] During image transmission, the client can send a privacy-preserving image to the server. Because the privacy-preserving image is obtained after pixel scrambling and / or image block scrambling, even if the privacy-preserving image is intercepted by an attacker during transmission, the attacker cannot obtain sensitive information from the privacy-preserving image, thus preventing the leakage of sensitive information and user privacy information. This ensures privacy protection for the original sensitive image, demonstrating a high level of privacy protection.

[0052] The server does not need to restore the privacy-preserving image during image preprocessing or image recognition. For example, the server can directly input the privacy-preserving image into the network model. This prevents attackers from intercepting the restored original sensitive image, thus preventing the leakage of sensitive information.

[0053] In this embodiment, for the image privacy protection process, the client can use scrambling parameters to perform a scrambling operation on the original sensitive image to obtain a privacy-protected image. For example, the scrambling parameters may include a pixel value protection key and / or an image block protection key. The client can perform a pixel scrambling operation on the original sensitive image based on the pixel value protection key, and / or the client can perform an image block scrambling operation on the original sensitive image based on the image block protection key. In summary, in the image privacy protection process, the client can perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image. The image privacy protection process is described below. For example, in order to implement the image privacy protection process, the following three methods can be adopted.

[0054] Method 1: Perform pixel scrambling and image block scrambling operations on the original sensitive image to obtain a privacy-preserving image corresponding to the original sensitive image. For example, the original sensitive image is first subjected to pixel scrambling to obtain an intermediate image after the pixel scrambling operation, and then the intermediate image is subjected to image block scrambling to obtain the privacy-preserving image. Alternatively, the original sensitive image is first subjected to image block scrambling to obtain an intermediate image after the image block scrambling operation, and then the intermediate image is subjected to pixel scrambling to obtain the privacy-preserving image.

[0055] For ease of description, we take the example of performing pixel scrambling followed by image block scrambling. As shown in Figure 2, after obtaining the original sensitive image, the pixel scrambling operation is performed on the original sensitive image to obtain an intermediate image. Then, the image block scrambling operation is performed on the intermediate image to obtain a privacy-preserving image.

[0056] In a possible implementation, the following steps S11-S14 may be used to obtain a privacy-preserving image.

[0057] In step S11, the original sensitive image is divided into M*N image blocks, where M and N are both positive integers.

[0058] For example, after obtaining the original sensitive image, the original sensitive image can be divided into M*N image blocks, where M and N can be the same or different. M indicates that there are M image blocks in the horizontal direction of the original sensitive image, and N indicates that there are N image blocks in the vertical direction of the original sensitive image.

[0059] For example, the sizes of different image blocks can be the same or different. For example, all image blocks are the same size. For example, the size of all image blocks is m*n, where m and n are both positive integers and can be the same or different. Here, m represents the number of pixels in the horizontal direction of each image block, and n represents the number of pixels in the vertical direction of each image block.

[0060] In step S12, a plurality of target image blocks to be protected are selected from the M*N image blocks.

[0061] For example, all the image blocks in the M*N image blocks may be used as target image blocks. In this case, pixel scrambling operations and image block scrambling operations need to be performed on all the image blocks.

[0062] For another example, some image blocks among the M*N image blocks may be used as target image blocks. In this case, pixel scrambling operations and image block scrambling operations need to be performed on some image blocks.

[0063] For the convenience of description, it is taken that all image blocks are used as target image blocks as an example, that is, M*N image blocks of the original sensitive image are used as target image blocks, and the scrambling operation is performed on all image blocks.

[0064] In step S13, for each target image block, a pixel value protection key corresponding to the target image block is obtained, a first scrambling order corresponding to the pixel value protection key is determined, and a scrambling operation (i.e., a pixel scrambling operation) is performed on multiple pixel values ​​within the target image block based on the first scrambling order to obtain an intermediate image after the pixel scrambling operation. The pixel scrambling operation is used to scramble multiple pixel values ​​within the target image block. The pixel scrambling operation is an operation that disrupts the original order of the multiple pixel values ​​within the target image block.

[0065] For example, the pixel value protection keys corresponding to different target image blocks can be the same, or different. For the sake of convenience, the pixel value protection keys corresponding to all target image blocks are the same, and the pixel value protection key is recorded as Key 像素值 .

[0066] For example, assuming that the pixel value protection key range is [a, b], you can generate a random number in the range [a, b] and use this random number as the key. 像素值 For a large number of original sensitive images, the pixel value protection key corresponding to the target image block of each original sensitive image can be the same, which is this random number.

[0067] In summary, for each target image block of the original sensitive image, this random number can be used as the pixel value protection key corresponding to the target image block, and the pixel value protection key is subsequently recorded as Key 像素值 .

[0068] For example, a first mapping relationship can be pre-maintained, the first mapping relationship being used to record the correspondence between the pixel value protection key and the scrambling order. Table 1 shows an example of the first mapping relationship. The pixel value protection key is a value within the value range [a, b]. The scrambling order represents the correspondence between the pixel value positions before and after the scrambling operation. Different scrambling orders represent different correspondences.

[0069] Table 1

[0070] For example, after obtaining the key corresponding to the target image block 像素值 After that, you can use Key 像素值 Query the first mapping relationship shown in Table 1 to obtain Key 像素值 The corresponding first scrambling order. For example, if Key像素值 If it is 1, the first scrambling order is scrambling order 11. 像素值 is 2, then the first scrambling order is scrambling order 12, and so on, the first scrambling order corresponding to the target image block can be obtained.

[0071] For example, the first scrambling order represents the correspondence between the pixel value positions before and after the scrambling operation. Table 2 shows an example of the first scrambling order. Of course, this is just one example for illustration. When the first scrambling order is another scrambling order, the correspondence shown in Table 2 will change accordingly. This embodiment does not impose any restrictions on this scrambling order.

[0072] Table 2

[0073] Exemplarily, after obtaining the first scrambling order, a scrambling operation can be performed on multiple pixel values ​​in the target image block based on the first scrambling order. For example, assuming that the size of the target image block is 3*2, then based on the first scrambling order shown in Table 2, the pixel value of pixel position (1, 1) is moved to pixel position (2, 2), the pixel value of pixel position (1, 2) is moved to pixel position (1, 3), the pixel value of pixel position (1, 3) is moved to pixel position (2, 1), the pixel value of pixel position (2, 1) is moved to pixel position (1, 1), the pixel value of pixel position (2, 2) is moved to pixel position (2, 3), and the pixel value of pixel position (2, 3) is moved to pixel position (1, 2). After performing the above scrambling operation on the target image block, the target image block after the scrambling operation can be obtained.

[0074] Obviously, after performing the above scrambling operation on each target image block of the original sensitive image, an intermediate image after the pixel scrambling operation can be obtained, and the intermediate image includes the target image block after the scrambling operation.

[0075] In step S14, an image block protection key corresponding to the original sensitive image is obtained, a second scrambling order corresponding to the image block protection key is determined, and a scrambling operation (i.e., an image block scrambling operation) is performed on multiple target image blocks in the intermediate image based on the second scrambling order to obtain a privacy-protected image corresponding to the original sensitive image. The image block scrambling operation is used to scramble multiple target image blocks in the intermediate image. The image block scrambling operation is an operation that disrupts the original order of multiple target image blocks in the intermediate image.

[0076] For example, the image block protection keys corresponding to different original sensitive images can be the same, or they can be different. For the sake of convenience, the image block protection keys corresponding to all original sensitive images are the same, and the image block protection key is recorded as Key 图像块 .

[0077] For example, assuming that the image block protection key has a value range of [c, d], a random number in the value range [c, d] can be generated and used as the key. 图像块 For a large number of original sensitive images, the image block protection key corresponding to each original sensitive image can be the same, and can all be this random number.

[0078] In summary, for each original sensitive image, this random number can be used as the image block protection key corresponding to the original sensitive image, and the image block protection key is subsequently recorded as Key 图像块 .

[0079] Exemplarily, a second mapping relationship can be maintained in advance, and the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order. Table 3 shows an example of the second mapping relationship. The image block protection key is a value in the value range [c, d]. The scrambling order represents the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation. Different scrambling orders represent different correspondences.

[0080] Table 3

[0081] For example, after obtaining the key corresponding to the original sensitive image 图像块 After that, you can use Key 图像块 Query the second mapping relationship shown in Table 3 to obtain Key 图像块 The corresponding second scrambling order. For example, if Key 图像块 If the key is 1, the second scrambling order is scrambling order 21. 图像块 If it is 2, then the second scrambling order is scrambling order 22. Similarly, the second scrambling order corresponding to the original sensitive image can be obtained.

[0082] Exemplarily, the second scrambling order represents the correspondence between the positions of the image blocks before and after the scrambling operation. Table 4 shows an example of the second scrambling order. Of course, this description uses only one scrambling order as an example. When the second scrambling order is another scrambling order, the correspondence shown in Table 4 will vary accordingly. This embodiment does not impose any restrictions on this scrambling order.

[0083] Table 4

[0084] Exemplarily, after obtaining the second scrambling order, a scrambling operation can be performed on multiple target image blocks in the intermediate image based on the second scrambling order. For example, assuming that the original sensitive image (i.e., the intermediate image) includes 2*2 image blocks (assuming that these image blocks are all target image blocks), then, based on the second scrambling order shown in Table 4, the image block at image block position (1, 1) is moved to image block position (1, 2), the image block at image block position (1, 2) is moved to image block position (2, 2), the image block at image block position (2, 1) is moved to image block position (1, 1), and the image block at image block position (2, 2) is moved to image block position (2, 1). After performing the above scrambling operation on the intermediate image, the intermediate image after the scrambling operation is the privacy-protected image corresponding to the original sensitive image.

[0085] In summary, by performing pixel scrambling and image block scrambling operations on the original sensitive image, the privacy-protected image corresponding to the original sensitive image can be obtained. The order of pixel scrambling operations (the first scrambling order) is determined by Key 像素值 Control, the order of image block scrambling operation (second scrambling order) is controlled by Key 图像块 Control. Key 图像块 Shared or not shared by all images, Key 像素值 Shared by all image tiles or not.

[0086] For example, by performing pixel scrambling on the original sensitive image, visual texture information can be eliminated, thereby avoiding leakage of texture information of the original sensitive image. By performing image block scrambling on the original sensitive image, visual contour information can be eliminated, thereby avoiding leakage of contour information of the original sensitive image.

[0087] For privacy-preserving images that employ pixel and block scrambling operations, only brute force can restore the privacy-preserving image to the original sensitive image. Assuming the original sensitive image is a 224*224 face image, and is partitioned into 16*16 image blocks, 14*14 image blocks can be obtained. The complexity of the brute force attack is (16*16*3)! *(14*14)!, or 768! *196!. Clearly, the complexity of the brute force attack is very high. This means that the embodiments of the present application have a very high privacy protection capability.

[0088] Method 2: Perform pixel scrambling operation on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image is divided into M*N image blocks, where M and N are both positive integers. Multiple target image blocks to be protected are selected from the M*N image blocks. All image blocks in the M*N image blocks can be used as target image blocks, or some image blocks in the M*N image blocks can be used as target image blocks. For each target image block, the pixel value protection key corresponding to the target image block is obtained, the first scrambling order corresponding to the pixel value protection key is determined, and a scrambling operation is performed on multiple pixel values ​​in the target image block based on the first scrambling order (i.e., pixel scrambling operation, pixel scrambling operation can be used to perform scrambling operation on multiple pixel values ​​in the image block) to obtain a privacy-protected image after the pixel scrambling operation.

[0089] For example, the implementation process of Method 2 can refer to Method 1 and will not be repeated here.

[0090] Method 3: Perform an image block scrambling operation on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image is divided into M*N image blocks, where M and N are both positive integers. Multiple target image blocks to be protected are selected from the M*N image blocks. All of the image blocks in the M*N image blocks can be used as target image blocks, or some of the image blocks in the M*N image blocks can be used as target image blocks. Obtain the image block protection key corresponding to the original sensitive image, determine the second scrambling order corresponding to the image block protection key, and perform a scrambling operation on the multiple target image blocks in the original sensitive image based on the second scrambling order (i.e., an image block scrambling operation, which can be used to perform a scrambling operation on multiple target image blocks of the original sensitive image) to obtain a privacy-protected image after the image block scrambling operation.

[0091] For example, the implementation process of Method 3 can refer to Method 1 and will not be repeated here.

[0092] In this embodiment, during the image recognition process, the server can perform artificial intelligence processing on the privacy-preserving image based on the target network model to obtain an image processing result corresponding to the privacy-preserving image. For example, after receiving the privacy-preserving image, the server inputs the privacy-preserving image into the target network model to obtain an image processing result corresponding to the privacy-preserving image. The image recognition process is described below.

[0093] For example, a network model (such as a neural network model) can be obtained in advance as the target network model. The target network model can be a network model including one fully connected layer, that is, the target network model only includes one fully connected layer. Assuming that the size of the input image of the target network model is 224*224*3=150528, and the target network model is used to implement a 10-category image recognition task, the weights of the target network model are a matrix of [150528, 10], that is, the weights of one fully connected layer are a matrix of [150528, 10].

[0094] The target network model can also be a network model containing multiple fully connected layers. That is, the target network model contains multiple fully connected layers. Assuming that the size of the input image of the target network model is 224*224*3=150528, and the target network model is used to implement a 10-class image recognition task, the weight of the first fully connected layer of the target network model is a matrix of [150528, 10]. There is no restriction on the other fully connected layers of the target network model.

[0095] The target network model can also be a visual self-attention network model (Vision Transformer), which can include an image segmentation layer (used to divide the image into M*N image blocks), a fully connected layer (that is, all image blocks pass through a shared fully connected layer), a position encoding layer (used to encode the position of each image block), and a self-attention network layer (used for feature fusion). There is no restriction on the network layers of the target network model.

[0096] The target network model can also be a visual self-attention network model without position encoding, that is, the target network model can be a visual self-attention model without position encoding (Vision Transformer). The target network model without position encoding can include an image segmentation layer (for dividing the image into M*N image blocks), a fully connected layer (that is, all image blocks pass through a shared fully connected layer), and a self-attention network layer (for feature fusion). There is no restriction on the network layers of this target network model.

[0097] The target network model can also be a network model of a multilayer perceptron and a mixer, that is, the target network model can be a multilayer perceptron-mixer (MLP-Mixer) model. The target network model of a multilayer perceptron and a mixer can include an image segmentation layer (for dividing the image into M*N image blocks), a fully connected layer (that is, all image blocks pass through a shared fully connected layer), and a mixer layer (for feature fusion. In the mixer layer, two groups of fully connected layers can be used to perform feature fusion in the spatial domain and feature fusion in the channel domain, respectively). There is no restriction on the mixer layer of this target network model.

[0098] Of course, the above are just a few examples of target network models. There is no restriction on the type of target network model, as long as the target network model can be obtained. For example, the target network model can be a visual self-attention model, MLP-Mixer, Vision Permutator, ConvMixer, etc. For various types of target network models, the model structure of the target network model can have the following characteristics: the image can be divided into M*N image blocks, the image blocks can be used for feature extraction using a shared fully connected layer, and the features of all image blocks can be fed into a series of isotropic network layers for feature fusion.

[0099] For the sake of convenience, the target network model is a visual self-attention network model as an example. The implementation of other types of target network models is similar. Referring to Figure 3, a schematic diagram of the structure of the target network model is shown. The target network model can include an image segmentation layer, a fully connected layer, a position encoding layer, and a self-attention network layer. After obtaining the target network model, the following steps S21-S24 can be used to implement the image recognition process.

[0100] In step S21, after receiving the privacy-preserving image, the privacy-preserving image is input into the image segmentation layer of the target network model, and the image segmentation layer divides the privacy-preserving image into M*N image blocks, where M and N are both positive integers.

[0101] Exemplarily, the sizes of different image blocks may be the same or different. The following example assumes that the sizes of all image blocks are the same, for example, the sizes of all image blocks are m*n, where m and n are both positive integers.

[0102] In step S22, after obtaining M*N image blocks, each image block is input into the fully connected layer (that is, all image blocks are input into the shared fully connected layer), and the shared fully connected layer performs feature extraction on each image block to obtain a fixed-length fully connected feature corresponding to each image block.

[0103] For example, assuming that the size of each image block is m*n and the size of the weight of the fully connected layer is m*n, then when image block 1 is input to the fully connected layer, the fully connected layer can map image block 1 to a fixed-length fully connected feature 1 through a shared linear mapping. When image block 2 is input to the fully connected layer, the fully connected layer can map image block 2 to a fixed-length fully connected feature 2 through a shared linear mapping, and so on. The linear mapping is a linear operation involving matrix multiplication, that is, a linear operation that performs matrix multiplication on image blocks (such as image block 1, image block 2, etc.) and the weights of the fully connected layer.

[0104] In step S23, after obtaining the fully connected features corresponding to each image block, the fully connected features corresponding to each image block are input to the position encoding layer, which performs position encoding on the fully connected features corresponding to each image block to obtain the position encoding features corresponding to each image block. For example, the fully connected feature 1 corresponding to image block 1 can be input to the position encoding layer, and the position encoding layer encodes the learnable position information on the fully connected feature 1 to obtain the position encoding feature 1 containing the position information. The fully connected feature 2 corresponding to image block 2 can be input to the position encoding layer, and the position encoding layer encodes the learnable position information on the fully connected feature 2 to obtain the position encoding feature 2 containing the position information, and so on.

[0105] In step S24, after obtaining the position coding features corresponding to each image block, the position coding features corresponding to each image block are input into the self-attention network layer, and the self-attention network layer performs feature fusion on the position coding features corresponding to all image blocks, and finally obtains the image processing result corresponding to the privacy-preserving image.

[0106] Exemplarily, the self-attention network layer may include a self-attention encoder (Transformer Encoder) and a multilayer perceptron (MLP), etc. The position encoding features corresponding to each image block may be input into the self-attention encoder, and the self-attention encoder performs self-attention encoding on the position encoding features corresponding to each image block to obtain the self-attention encoding features corresponding to each image block, and the self-attention encoding features corresponding to each image block are input into the multilayer perceptron, and the multilayer perceptron performs feature fusion on the self-attention encoding features corresponding to all image blocks to obtain fused features, and the image processing results corresponding to the privacy-protected image, such as image recognition results, are determined based on the fused features.

[0107] In summary, the image processing results corresponding to the privacy-protected image can be obtained, completing the image recognition process.

[0108] In one possible implementation, the privacy-preserving image is input into the target network model to obtain an image processing result. The privacy-preserving image is obtained by scrambling the original sensitive image using scrambling parameters. Therefore, to process the privacy-preserving image, the weights in the target network model are also scrambled using the scrambling parameters. For example, the target network model can be obtained using the following two methods.

[0109] Method 1: The server can obtain the scrambling parameters from the client and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model.

[0110] For example, the client can obtain the scrambling parameters and send them to the server, which then obtains the scrambling parameters from the client. The scrambling parameters may include a pixel value protection key (which may be denoted as Key 像素值 ) and / or image block protection key (which can be recorded as Key 图像块 For example, if the client trusts the server (ie the server will not disclose the Key 像素值 and Key 图像块 ), then the client can use the Key 像素值 and Key 图像块 Sent to the server. Among them, Key 像素值 Key to the image privacy protection process 像素值 Same, Key 图像块 Key to the image privacy protection process 图像块 The same, no longer repeated here.

[0111] Among them, the client trusts the server means that the server provides a secure system operation and key management process, that is, the server guarantees the key 像素值 and Key 图像块 To ensure communication security, after completing a transformation operation, the key must be deleted on the server. 像素值 and Key 图像块 , and delete the weights before transformation to prevent third-party attacks. In addition, the server provider does not use its own pre-trained weights or downstream task weights to compare with the corresponding transformed weights to try to crack the key. 像素值 and Key 图像块 .

[0112] For example, the server can pre-train a network model as the initial network model, such as obtaining the initial network model based on a large amount of training data, without any restrictions on the training process. The initial network model can be a network model containing one fully connected layer; or a network model containing multiple fully connected layers; or a network model using visual self-attention; or a network model using visual self-attention without position encoding; or a network model using a multi-layer perceptron and a hybrid, without any restrictions on the above.

[0113] Exemplarily, the server can use scrambling parameters to scramble the weights in the initial network model to obtain the target network model. For example, if the initial network model is a network model containing one fully connected layer, that is, the initial network model only contains one fully connected layer, then the scrambling parameters can be used to scramble the weights of one fully connected layer to obtain the target network model. For another example, if the initial network model is a network model containing multiple fully connected layers, that is, the initial network model contains multiple fully connected layers, then the scrambling parameters can be used to scramble the weights of the first fully connected layer to obtain the target network model. For another example, if the initial network model is a visual self-attention network model, that is, the initial network model includes an image segmentation layer, a fully connected layer, a position encoding layer, and a self-attention network layer, then the scrambling parameters can be used to scramble the weights of the fully connected layer and the position encoding layer to obtain the target network model, such as using Key 像素值 The weights of the fully connected layer are scrambled and Key 图像块 The weights of the position encoding layer are scrambled. For example, if the initial network model is a visual self-attention network model without position encoding, that is, the initial network model includes an image segmentation layer, a fully connected layer, and a self-attention network layer, then the scrambling parameters can be used to scramble the weights of the fully connected layer to obtain the target network model, such as using Key 像素值 Perform scrambling operations on the weights of the fully connected layer. For another example, if the initial network model is a network model of a multi-layer perceptron and a mixer, that is, the initial network model includes an image segmentation layer, a fully connected layer, and a mixer layer (in the mixer layer, two groups of fully connected layers perform feature fusion in the spatial domain and channel domain respectively), then the scrambling parameters can be used to scramble the weights of the fully connected layer and the mixer layer to obtain the target network model, such as using Key 像素值 The weights of the shared fully connected layer are scrambled and Key 图像块 The weights of the fully connected layer responsible for spatial domain feature fusion are scrambled.

[0114] In summary, the model structure that can be adjusted by scrambling parameters includes the following features: dividing the image into blocks, using a shared fully connected layer to extract features from the image blocks, and sending them to the isotropic network layer for feature fusion. Based on this, Key 像素值 The weights of the shared fully connected layer are scrambled using Key 图像块 Scramble the weights of the network layers used to extract spatial information.

[0115] Since privacy-preserving images cannot be recognized by normal network models (such as the initial network model), this embodiment proposes a model weight adjustment method based on scrambling parameters. The scrambling parameters are consistent with the scrambling parameters of the image scrambling operation. That is, the weights of the initial network model are scrambled accordingly using the scrambling parameters. This operation is completed once during deployment, resulting in a target network model that can correctly adapt to the scrambled data (i.e., the privacy-preserving image), and the scrambled data is processed by the target network model.

[0116] Since model weight adjustment is independent of the training process, it is possible to directly perform model weight adjustment on a normal network model (such as the initial network model), allowing the adjusted network model to adapt to the scrambled data with virtually no performance loss. Of course, it is also possible to retrain the network model on privacy-preserving images and adjust the model weights of the pre-trained model to achieve better performance.

[0117] In a possible implementation, the server uses scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain a target network model, which may include but is not limited to the following steps S31-S34.

[0118] In step S31, the server determines a first scrambling sequence corresponding to a pixel value protection key.

[0119] For example, if the scrambling parameters include a pixel value protection key, Key 像素值 ,The original sensitive image includes multiple image blocks,Key 像素值 It is used to perform pixel scrambling operation on multiple pixel values ​​in the image block, so the Key 像素值 The corresponding first scrambling order. For example, through Key 像素值 Query the configured first mapping relationship and get the Key 像素值 The first scrambling sequence is used to represent the correspondence between the pixel value positions before the scrambling operation and the pixel value positions after the scrambling operation. The first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling sequence.

[0120] Exemplarily, the processing of step S31 is similar to that of step S13 and will not be described again here.

[0121] In step S32, the server determines a second scrambling sequence corresponding to the image block protection key.

[0122] For example, if the scrambling parameters include the image block protection key, that is, Key 图像块 ,The original sensitive image includes multiple image blocks,Key 图像块 It is used to perform image block scrambling operation on multiple image blocks, so the Key can be determined图像块 The corresponding second scrambling order. For example, through Key 图像块 Query the configured second mapping relationship and get the Key 图像块 A corresponding second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation. The second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

[0123] Exemplarily, the processing of step S32 is similar to that of step S14 and will not be described again here.

[0124] In step S33, the first type of weights in the initial network model are determined, and the first type of weights in the initial network model are scrambled based on the first scrambling order to obtain a scrambled initial network model. The first type of weights may be weights in the initial network model that are calculated with pixel values ​​in the image block.

[0125] For example, the initial network model includes a large number of weights. For each weight, it can be determined whether the weight needs to be calculated with the pixel values ​​in the image block, that is, whether the weight will be calculated with the pixel values ​​in the image block during the operation of the network model. If so, the weight is used as the first type of weight in the initial network model. If not, the weight is not used as the first type of weight in the initial network model.

[0126] After determining the first type of weights in the initial network model, the first type of weights in the initial network model can be scrambled based on the first scrambling order. The scrambling operation is used to scramble the multiple weight values ​​in the first type of weights (the first type of weights can include multiple weight values, such as m*n weight values, and the size of the first type of weights is the same as the size of the above-mentioned image block, that is, the number of weight values ​​is the same as the number of pixels in the image block), that is, the original order of the multiple weight values ​​in the first type of weights is scrambled. The scrambling operation process of step S33 is similar to the scrambling operation process of step S13 and will not be repeated here.

[0127] In step S34, the second type of weights in the initial network model are determined, and the second type of weights in the initial network model (e.g., the initial network model after scrambling the first type of weights) are scrambled based on the second scrambling order to obtain a scrambled target network model. The second type of weights may be weights in the initial network model related to position information of the image blocks (e.g., the order of the image blocks).

[0128] For example, the initial network model includes a large number of weights. For each weight, it can be determined whether the weight is related to the position information of the image block, that is, whether the weight is calculated based on the position information of the image block during the operation of the network model. If so, the weight is used as the second type of weight in the initial network model. If not, the weight is not used as the second type of weight in the initial network model.

[0129] After determining the second type of weights in the initial network model, the second type of weights in the initial network model can be scrambled based on the second scrambling order. The scrambling operation is used to scramble the multiple weight values ​​in the second type of weights (the second type of weights can include multiple weight values, such as M*N weight values, and the size of the second type of weights is the same as the number of image blocks, that is, the number of weight values ​​is the same as the number of all image blocks), that is, the original order of the multiple weight values ​​in the second type of weights is scrambled. The scrambling operation process of step S34 is similar to the scrambling operation process of step S14 and will not be repeated here.

[0130] In one possible implementation, if the initial network model is a visual self-attention network model, that is, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then the weights in the fully connected layer can be used as the first type of weights, and the weights in the position encoding layer can be used as the second type of weights. The weights in the fully connected layer can be scrambled based on the first scrambling order, and the weights in the position encoding layer can be scrambled based on the second scrambling order to obtain the target network model.

[0131] For example, you can use Key 像素值 The corresponding first scrambling order scrambles the weights in the shared linear mapping so that the scrambled pixels in each image block in the privacy-preserving image can still be calculated with the corresponding linear mapping weights. 图像块 The corresponding second scrambling order scrambles the weights of the position codes so that the image block codes corresponding to each scrambled image block in the privacy-preserving image are still calculated with the weights of the corresponding position codes, thereby ensuring that the related operations do not damage the model application performance.

[0132] Method 2: The server obtains target weight information from the client and generates a target network model based on the target weight information; wherein the target weight information can be weight information in the candidate network model, and the candidate network model is obtained by the client using a scrambling parameter to scramble the weights in the initial network model.

[0133] For example, the client can obtain the scrambling parameters, but the client does not send the scrambling parameters to the server. The scrambling parameters may include a pixel value protection key (which can be recorded as Key 像素值 ) and / or image block protection key (which can be recorded as Key 图像块 ). Among them, Key 像素值 Key to the image privacy protection process 像素值 Same, Key 图像块 Key to the image privacy protection process 图像块 The same, no longer repeated here.

[0134] Exemplarily, the client can obtain a network model as the initial network model. For example, the client obtains a configured network model, the weights of which are weights to be trained, which are weights that have not yet been trained. The client obtains the initial network model based on a large amount of training data, and there is no restriction on this training process. For another example, the client obtains a configured network model, the weights of which are pre-trained weights (also called downstream task weights), which are weights that have been trained, such as weights obtained by training on other data sets. The client obtains the initial network model based on a small amount of training data, and there is no restriction on this training process. For another example, the client obtains a configured network model, the weights of which are trained weights (the above-mentioned pre-trained weights or downstream task weights can also be used as trained weights and no longer trained), the trained weights are weights that have completed training, and the client directly uses the configured network model as the initial network model and no longer trains the configured network model.

[0135] Exemplarily, the client can use scrambling parameters to scramble the weights in the initial network model to obtain a candidate network model. For example, the client can determine a first scrambling order corresponding to a pixel value protection key and determine a second scrambling order corresponding to an image block protection key. Determine the first type of weights in the initial network model, and scramble the first type of weights in the initial network model based on the first scrambling order to obtain a scrambled initial network model. The first type of weights are weights in the initial network model that are calculated with pixel values ​​in the image block. Determine the second type of weights in the initial network model, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain a scrambled candidate network model. The second type of weights are weights in the initial network model that are related to the position information of the image block. The process of the client performing scrambling operations on the weights in the initial network model can be referred to Method 1, which will not be repeated here.

[0136] After obtaining the candidate network model, the client can send the target weight information in the candidate network model to the server, so that the server can generate the target network model based on the target weight information. For example, after receiving the target weight information, the server can directly obtain the target network model based on the target weight information, that is, the weight of the target network model is determined based on the target weight information, and the weight of the target network model is the same as the weight of the candidate network model. For another example, after receiving the target weight information, the server can obtain the network model to be trained based on the target weight information, that is, the weight of the network model to be trained is determined based on the target weight information, and then, the network model to be trained is trained with sample data to obtain the trained target network model, that is, training is performed based on the weights provided by the client, thereby obtaining good model performance.

[0137] In one possible implementation, as shown in FIG4 , which is a schematic diagram of the structure of an image processing system, the image processing system may include an image acquisition module (for implementing the image acquisition process), an image privacy protection module (for implementing the image privacy protection process), an image recognition module (for implementing the image recognition process), and a post-processing module (for implementing the post-processing process). The image acquisition module and the image privacy protection module are located on the client side, while the image recognition module and the post-processing module are located on the server side.

[0138] The image acquisition module can collect the original sensitive image and send the original sensitive image to the image privacy protection module. The image privacy protection module can perform pixel scrambling operation and / or image block scrambling operation on the original sensitive image to obtain a privacy-protected image, and send the privacy-protected image to the server.

[0139] After obtaining the privacy-preserving image, the server can provide it to the image recognition module. The image recognition module performs artificial intelligence processing on the privacy-preserving image based on the target network model, obtains image processing results (such as image detection results, image segmentation results, image recognition results, etc.), and sends the image processing results to the post-processing module. After obtaining the image processing results, the post-processing module can perform post-processing of the privacy-preserving image based on the image processing results.

[0140] In an embodiment of the present application, pixel scrambling operations and / or image block scrambling operations can be performed on the original sensitive image to obtain a privacy-protected image, and the client sends the privacy-protected image to the server. Even if the privacy-protected image is intercepted by an attacker, the attacker cannot obtain sensitive information from the privacy-protected image, thereby avoiding the leakage of sensitive information and thus avoiding the leakage of user privacy information. In other words, the embodiment of the present application can perform privacy protection on the original sensitive image without the risk of privacy leakage, and has a very high privacy protection capability. By performing a pixel scrambling operation on the original sensitive image, the visual texture information can be eliminated, thereby avoiding the leakage of the texture information of the original sensitive image. By performing an image block scrambling operation on the original sensitive image, the visual contour information can be eliminated, thereby avoiding the leakage of the contour information of the original sensitive image. Using an isotropic model structure represented by a visual self-attention model, the target network model is trained on a normal data set, and then the scrambling parameters are used to scramble the weights in the trained model. The target network model after the scrambling operation can be used for reasoning, and only requires one-time key configuration and deployment, which is more convenient to deploy.

[0141] Based on the same inventive concept as the above-mentioned image processing method, an image processing device is proposed in an embodiment of the present application, which is applied to a server.

[0142] The acquisition module 51 is used to obtain a privacy-preserving image corresponding to the original sensitive image; wherein the privacy-preserving image is an image obtained by performing a scrambling operation on the original sensitive image using a scrambling parameter.

[0143] The processing module 52 is used to input the privacy-preserving image into the target network model, obtain the image processing result corresponding to the privacy-preserving image, and output the image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0144] Exemplarily, the processing module 52 is further used to obtain the scrambling parameters from the client, and use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain a target network model; or, obtain target weight information from the client, and generate a target network model based on the target weight information; wherein the target weight information is the weight information in the candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to perform a scrambling operation on the weights in the initial network model.

[0145] Exemplarily, the processing module 52 uses the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model. Specifically, if the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the corresponding image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model.

[0146] Exemplarily, when the processing module 52 determines the first scrambling order corresponding to the pixel value protection key, it is specifically configured to: query a configured first mapping relationship through the pixel value protection key to obtain the first scrambling order corresponding to the pixel value protection key, wherein the first scrambling order is used to represent the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation; and the first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling order. When the processing module 52 determines the second scrambling order corresponding to the image block protection key, it is specifically configured to: query a configured second mapping relationship through the image block protection key to obtain the second scrambling order corresponding to the image block protection key, wherein the second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; and the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

[0147] Exemplarily, the processing module 52 performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, and is specifically used to: determine the first type of weights and the second type of weights in the initial network model, scramble the first type of weights in the initial network model based on the first scrambling order, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights are the weights in the initial network model that are operated on the pixel values ​​in the image block, and the second type of weights are the weights in the initial network model that are related to the position information of the image block.

[0148] Exemplarily, the initial network model is a network model including one fully connected layer; or the initial network model is a network model including multiple fully connected layers; or the initial network model is a network model of visual self-attention; or the initial network model is a network model of visual self-attention without position encoding; or the initial network model is a network model of a multi-layer perceptron and a mixer. The processing module 52 performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model. Specifically, if the initial network model is a network model of visual self-attention, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then the weights in the fully connected layer are scrambled based on the first scrambling order, and the weights in the position encoding layer are scrambled based on the second scrambling order to obtain the target network model.

[0149] Based on the same inventive concept as the aforementioned image processing method, embodiments of the present application provide a server-side device. Referring to FIG6 , the server-side device includes a processor 61 and a machine-readable storage medium 62 . The machine-readable storage medium 62 stores machine-executable instructions, such as image processing instructions, that can be executed by the processor 61 . The processor 61 is configured to execute the machine-executable instructions to implement the image processing method disclosed in the aforementioned embodiments of the present application.

[0150] Based on the same inventive concept as the above-mentioned image processing method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above-mentioned embodiment of the present application can be implemented.

[0151] The machine-readable storage medium may be any electronic, magnetic, optical, or other physical storage device that may contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium may be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or similar storage media, or a combination thereof.

[0152] Based on the same inventive concept as the above-mentioned image processing method, an embodiment of the present application further provides an image processing system, which includes: a client, configured to perform a scrambling operation on an original sensitive image using scrambling parameters to obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to a server; a server, configured to obtain a privacy-preserving image corresponding to the original sensitive image, and input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0153] Exemplarily, the client is further used to send the scrambling parameters to the server; the server is further used to obtain the scrambling parameters from the client, and use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model.

[0154] Alternatively, the client is further used to use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain a candidate network model, and send the target weight information in the candidate network model to the server; the server is further used to obtain the target weight information from the client and generate the target network model based on the target weight information.

[0155] Exemplarily, the scrambling parameters include a pixel value protection key and an image block protection key; the client uses the scrambling parameters to perform a scrambling operation on the original sensitive image, and when obtaining the privacy-protected image corresponding to the original sensitive image, it is specifically used to: divide the original sensitive image into M*N image blocks, and select multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers; for each target image block, obtain the pixel value protection key corresponding to the target image block, determine a first scrambling order corresponding to the pixel value protection key, and perform a scrambling operation on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image; obtain the image block protection key corresponding to the original sensitive image, determine a second scrambling order corresponding to the image block protection key, and perform a scrambling operation on multiple target image blocks in the intermediate image based on the second scrambling order to obtain the privacy-protected image corresponding to the original sensitive image.

[0156] Exemplarily, the server uses the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model: if the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the corresponding image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then:

[0157] Determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model.

[0158] Exemplarily, the server performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, specifically for: determining the first type of weights and the second type of weights in the initial network model, scrambling the first type of weights in the initial network model based on the first scrambling order, and scrambling the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights are the weights in the initial network model that are operated on the pixel values ​​in the image block, and the second type of weights are the weights in the initial network model that are related to the position information of the image block.

[0159] The systems, devices, modules, or units described in the above embodiments may be implemented by a computer entity or by a product having certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email transceiver, game console, tablet computer, wearable device, or any combination of these devices.

[0160] For the convenience of description, the above devices are described as being divided into various units according to their functions. Of course, when implementing this application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0161] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0162] The present application is described with reference to the flow chart and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow process and / or box in the flow chart and / or block diagram and the combination of the flow process and / or box in the flow chart and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow chart flow or multiple flows and / or one box or multiple boxes of the block diagram.

[0163] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable data processing device to produce computer-implemented processing, so that the instructions executed on the computer or other programmable data processing device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0165] The foregoing is merely an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. An image processing method, applied to a server, the method include: Acquire a privacy-preserving image corresponding to the original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter; The privacy-preserving image is input into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

2. The method according to claim 1, in, Before obtaining the privacy-preserving image corresponding to the original sensitive image, the method further includes: Obtaining the scrambling parameters from the client, and using the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model; or, Obtain target weight information from the client, and generate the target network model based on the target weight information; wherein the target weight information is weight information in a candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the initial network model.

3. The method according to claim 2, in, The step of using the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model includes: If the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: Determine a first scrambling order corresponding to the pixel value protection key; Determining a second scrambling order corresponding to the image block protection key; The weights in the initial network model are scrambled based on the first scrambling order and the second scrambling order to obtain the target network model.

4. The method according to claim 3, in, The determining a first scrambling order corresponding to the pixel value protection key includes: querying a configured first mapping relationship through the pixel value protection key to obtain a first scrambling order corresponding to the pixel value protection key, wherein the first scrambling order is used to represent a correspondence between a pixel value position before a scrambling operation and a pixel value position after a scrambling operation; wherein the first mapping relationship is used to record a correspondence between a pixel value protection key and a scrambling order; The determining a second scrambling order corresponding to the image block protection key includes: The configured second mapping relationship is queried through the image block protection key to obtain a second scrambling order corresponding to the image block protection key, wherein the second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; wherein the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

5. The method according to claim 3, in, The step of performing a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model includes: Determine the first type of weights and the second type of weights in the initial network model, scramble the first type of weights in the initial network model based on the first scrambling order, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights includes weights in the initial network model that are operated on pixel values ​​in an image block, and the second type of weights includes weights in the initial network model that are related to position information of image blocks.

6. The method according to claim 3 or 5, in, The initial network model is a network model including one fully connected layer; or, the initial network model is a network model including multiple fully connected layers; or, the initial network model is a network model of visual self-attention; or, the initial network model is a network model of visual self-attention without position encoding; or, the initial network model is a network model of multi-layer perceptron and mixer; The step of performing a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model includes: If the initial network model is a visual self-attention network model, the initial network model at least includes a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then: The weights in the fully connected layer are scrambled based on the first scrambling order, and the weights in the position encoding layer are scrambled based on the second scrambling order to obtain the target network model.

7. An image processing method, applied to a client, the method include: Using scrambling parameters to perform pixel scrambling operations and / or image block scrambling operations on the original sensitive image to obtain a privacy-preserving image; wherein the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to perform a scrambling operation on multiple pixel values ​​in the corresponding image block, and the image block scrambling operation is used to perform a scrambling operation on multiple image blocks; The privacy-preserving image is sent to a server, so that the server inputs the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

8. The method according to claim 7, in, The scrambling parameters include a pixel value protection key and an image block protection key; The step of using the scrambling parameters to perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image to obtain a privacy-preserving image includes: Divide the original sensitive image into M*N image blocks, and select multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers; For each of the target image blocks, obtaining a pixel value protection key corresponding to the target image block, determining a first scrambling order corresponding to the pixel value protection key, and performing a scrambling operation on a plurality of pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image; An image block protection key corresponding to the original sensitive image is obtained, a second scrambling order corresponding to the image block protection key is determined, and a scrambling operation is performed on multiple target image blocks in the intermediate image based on the second scrambling order to obtain the privacy-preserving image.

9. An image processing device, applied to a server, the device include: An acquisition module, used to acquire a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter; A processing module is used to input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

10. The device according to claim 9, in, The processing module is further used to obtain the scrambling parameters from the client, and use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model.

11. The device according to claim 9, in, The processing module is also used to obtain target weight information from the client and generate the target network model based on the target weight information; wherein the target weight information is the weight information in the candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the initial network model.

12. A server device, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the image processing method described in any one of claims 1-6.

13. An image processing system, comprising a client and a server, The client is used to perform a scrambling operation on the original sensitive image using a scrambling parameter to obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to the server; The server is used to obtain a privacy-preserving image corresponding to the original sensitive image, and input the privacy-preserving image into the target network model to obtain an image processing result corresponding to the privacy-preserving image; in, The target network The weights in the network model are scrambled using the scrambling parameters.

14. The system according to claim 13, in, The client is further used to send the scrambling parameters to the server; The server is also used to obtain the scrambling parameters from the client, and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model.

15. The system according to claim 13, in, The client is further configured to use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain a candidate network model, and send target weight information in the candidate network model to the server; The server is further configured to obtain the target weight information from the client and generate the target network model based on the target weight information.

Images