Image processing method, device and equipment

Abstract

The invention provides an image processing method, device and equipment. The method comprises the following steps: acquiring a privacy protection image corresponding to an original sensitive image; wherein the privacy protection image is an image obtained by performing scrambling operation on the original sensitive image by adopting a scrambling parameter; and inputting the privacy protection image to a target network model to obtain an image processing result corresponding to the privacy protection image. Through the technical scheme of the application, leakage of sensitive information can be avoided, leakage of user privacy information can be avoided, privacy protection can be carried out on the original sensitive image, the risk of privacy leakage does not exist, and the privacy protection capability is very high.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, device and equipment. Background Art

[0002] The client (such as a camera, mobile terminal, etc.) can obtain the original image and send it to the server. The server performs artificial intelligence processing on the original image based on the network model to obtain the image processing result of the original image (such as image detection result, image segmentation result, image recognition result, etc.).

[0003] If the original image is a sensitive image (such as a face image, a body image, etc.), during the transmission process of the original image, if the original image is intercepted by an attacker, the attacker can obtain sensitive information based on the original image, causing the leakage of sensitive information, which is likely to leak user privacy information and cause huge losses to the user. Summary of the invention

[0004] The present application provides an image processing method, which is applied to a server, and the method comprises:

[0005] Acquire a privacy-preserving image corresponding to the original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter;

[0006] The privacy-preserving image is input into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0007] The present application provides an image processing method, which is applied to a client, and the method comprises:

[0008] A pixel scrambling operation and / or an image block scrambling operation are performed on an original sensitive image using scrambling parameters to obtain a privacy-preserving image; wherein the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to perform a scrambling operation on multiple pixel values ​​in an image block, and the image block scrambling operation is used to perform a scrambling operation on multiple image blocks; the privacy-preserving image is sent to a server, so that the server inputs the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0009] The present application provides an image processing device, which is applied to a server, and the device includes:

[0010] An acquisition module, used to acquire a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter;

[0011] A processing module is used to input the privacy-preserving image into a target network model, obtain an image processing result corresponding to the privacy-preserving image, and output the image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0012] The present application provides a server device, comprising: a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores machine-executable instructions that can be executed by the processor; the processor is used to execute the machine-executable instructions to implement the above-mentioned image processing method.

[0013] The present application provides an image processing system, the system comprising:

[0014] The client is used to perform a scrambling operation on the original sensitive image using a scrambling parameter to obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to the server;

[0015] The server is used to obtain a privacy-preserving image corresponding to an original sensitive image, and input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0016] It can be seen from the above technical solution that in the embodiment of the present application, the scrambling parameters can be used to perform scrambling operations on the original sensitive image to obtain a privacy-protected image, and the client sends the privacy-protected image to the server. Even if the privacy-protected image is intercepted by an attacker, the attacker cannot obtain sensitive information based on the privacy-protected image, thereby avoiding the leakage of sensitive information and user privacy information. The original sensitive image can be privacy-protected without the risk of privacy leakage, and has a high privacy protection capability. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments of the present application or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings of the embodiments of the present application.

[0018] Figure 1 is a flowchart of an image processing method in one embodiment of the present application;

[0019] Figure 2 is a schematic diagram of a scrambling operation in one embodiment of the present application;

[0020] Figure 3 is a schematic diagram of the structure of a target network model in one embodiment of the present application;

[0021] Figure 4 is a schematic diagram of the structure of an image processing system in one embodiment of the present application;

[0022] Figure 5 is a schematic structural diagram of an image processing device in one embodiment of the present application;

[0023] Figure 6 It is a hardware structure diagram of a server device in one implementation mode of the present application. DETAILED DESCRIPTION

[0024] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, rather than limiting the present application. The singular forms of "a", "said" and "the" used in the present application and claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to any or all possible combinations of one or more associated listed items.

[0025] It should be understood that, although the terms first, second, third, etc. may be used to describe various information in the embodiments of the present application, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, in addition, the word "if" used may be interpreted as "at..." or "when..." or "in response to determination".

[0026] In the embodiment of the present application, an image processing method is proposed, which can be applied to the server. Figure 1 FIG. 1 is a flow chart of the image processing method, which may include:

[0027] Step 101, obtaining a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image may be an image obtained by scrambling the original sensitive image using scrambling parameters. For example, the original sensitive image is subjected to pixel scrambling and / or image block scrambling operations using scrambling parameters to obtain a privacy-preserving image; the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to scramble multiple pixel values ​​within the image block, and the image block scrambling operation is used to scramble multiple image blocks.

[0028] The scrambling parameters include a pixel value protection key and an image block protection key. The original sensitive image can be divided into M*N image blocks, and multiple target image blocks to be protected are selected from the M*N image blocks, where M and N are both positive integers; for each target image block, the pixel value protection key corresponding to the target image block is obtained, the first scrambling order corresponding to the pixel value protection key is determined, and the multiple pixel values ​​in the target image block are scrambled based on the first scrambling order to obtain an intermediate image. The image block protection key corresponding to the original sensitive image is obtained, the second scrambling order corresponding to the image block protection key is determined, and the multiple target image blocks in the intermediate image are scrambled based on the second scrambling order to obtain a privacy-protected image.

[0029] For example, the client can use the scrambling parameters to perform a scrambling operation on the original sensitive image, obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to the server.

[0030] Exemplarily, the scrambling parameters may include a pixel value protection key and / or an image block protection key, and the client may perform a pixel scrambling operation on the original sensitive image based on the pixel value protection key, and / or the client may perform an image block scrambling operation on the original sensitive image based on the image block protection key, thereby obtaining a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image may include multiple image blocks, and the pixel scrambling operation may be used to perform a scrambling operation on multiple pixel values ​​within an image block, and the image block scrambling operation may be used to perform a scrambling operation on multiple image blocks of the original sensitive image.

[0031] In a possible implementation, the client may divide the original sensitive image into M*N image blocks, and select multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers; for example, all of the M*N image blocks are used as target image blocks, or some of the M*N image blocks are used as target image blocks. For each target image block, obtain the pixel value protection key corresponding to the target image block, determine the first scrambling order corresponding to the pixel value protection key, and perform scrambling operations on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image (i.e., an image after scrambling all target image blocks). Obtain the image block protection key corresponding to the original sensitive image, determine the second scrambling order corresponding to the image block protection key, and perform scrambling operations on multiple target image blocks in the intermediate image based on the second scrambling order to obtain the privacy-protected image.

[0032] Exemplarily, the pixel value protection keys corresponding to different target image blocks may be the same or different; the image block protection keys corresponding to different original sensitive images may be the same or different.

[0033] Step 102: Input the privacy-preserving image into the target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using scrambling parameters.

[0034] In a possible implementation, the server may obtain scrambling parameters from the client, and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model. Alternatively, the server may obtain target weight information from the client, and generate the target network model based on the target weight information; wherein the target weight information is the weight information in the candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the initial network model.

[0035] Exemplarily, the scrambling parameters are used to perform a scrambling operation on the weights in the trained initial network model to obtain a target network model, which may include but is not limited to: if the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model.

[0036] Exemplarily, determining the first scrambling order corresponding to the pixel value protection key may include, but is not limited to: querying a configured first mapping relationship through the pixel value protection key to obtain the first scrambling order corresponding to the pixel value protection key. The first scrambling order may be used to represent the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation; the first mapping relationship may be used to record the correspondence between the pixel value protection key and the scrambling order.

[0037] Exemplarily, determining the second scrambling order corresponding to the image block protection key may include, but is not limited to: querying the configured second mapping relationship through the image block protection key to obtain the second scrambling order corresponding to the image block protection key. The second scrambling order may be used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; the second mapping relationship may be used to record the correspondence between the image block protection key and the scrambling order.

[0038] In a possible implementation, the weights in the initial network model are scrambled based on the first scrambling order and the second scrambling order to obtain the target network model, which may include but is not limited to: determining the first type of weights and the second type of weights in the initial network model, the first type of weights in the initial network model may be scrambled based on the first scrambling order, and the second type of weights in the initial network model may be scrambled based on the second scrambling order to obtain the target network model. The first type of weights may include but is not limited to weights in the initial network model that are calculated with pixel values ​​in the image block, and the second type of weights may include but is not limited to weights in the initial network model that are related to the position information of the image block.

[0039] In a possible implementation, the initial network model may be a network model including one fully connected layer; or, the initial network model may be a network model including multiple fully connected layers; or, the initial network model may be a network model of visual self-attention; or, the initial network model may be a network model of visual self-attention without position coding; or, the initial network model may be a network model of multi-layer perceptron and mixer. The weights in the initial network model are scrambled based on the first scrambling order and the second scrambling order to obtain the target network model, which may include but is not limited to: if the initial network model is a network model of visual self-attention, the initial network model includes at least a fully connected layer and a position coding layer, the fully connected layer is used to extract features from each image block, and the position coding layer is used to perform position coding on each image block, then: the weights in the fully connected layer may be scrambled based on the first scrambling order, and the weights in the position coding layer may be scrambled based on the second scrambling order to obtain the target network model.

[0040] It can be seen from the above technical solution that in the embodiment of the present application, the scrambling parameters can be used to perform scrambling operations on the original sensitive image to obtain a privacy-protected image, and the client sends the privacy-protected image to the server. Even if the privacy-protected image is intercepted by an attacker, the attacker cannot obtain sensitive information based on the privacy-protected image, thereby avoiding the leakage of sensitive information and user privacy information. The original sensitive image can be privacy-protected without the risk of privacy leakage, and has a high privacy protection capability.

[0041] Based on the same application concept as the above method, an image processing method is proposed in an embodiment of the present application, which can be applied to a client, and the method may include: using scrambling parameters to perform pixel scrambling operations and / or image block scrambling operations on the original sensitive image to obtain a privacy-preserving image; wherein the original sensitive image includes multiple image blocks, the pixel scrambling operation is used to perform scrambling operations on multiple pixel values ​​in the image block, and the image block scrambling operation is used to perform scrambling operations on multiple image blocks. Send the privacy-preserving image to the server, so that the server inputs the privacy-preserving image into the target network model to obtain the image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using scrambling parameters.

[0042] Exemplarily, the scrambling parameters may include a pixel value protection key and an image block protection key; using the scrambling parameters to perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image to obtain a privacy-protected image may include but is not limited to: dividing the original sensitive image into M*N image blocks, and selecting multiple target image blocks to be protected from the M*N image blocks, where M and N are both positive integers; for each target image block, obtaining a pixel value protection key corresponding to the target image block, determining a first scrambling order corresponding to the pixel value protection key, and performing a scrambling operation on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image; obtaining an image block protection key corresponding to the original sensitive image, determining a second scrambling order corresponding to the image block protection key, and performing a scrambling operation on multiple target image blocks in the intermediate image based on the second scrambling order to obtain a privacy-protected image.

[0043] The above technical solutions of the embodiments of the present application are described below in combination with specific application scenarios.

[0044] Image processing systems (such as image recognition systems, etc.) are systems that use artificial intelligence technology to automatically identify (detect, segment, etc.) relevant information from images. They can perform artificial intelligence processing on images based on network models to obtain image processing results (such as image detection results, image segmentation results, image recognition results, etc.). Among them, image processing systems may include but are not limited to the following processes: image acquisition (or image acquisition) process, image transmission process, image preprocessing process, image recognition process, and post-processing process.

[0045] During the image acquisition process, the client can acquire the original image. During the image transmission process, the client can send the original image to the server. During the image preprocessing process, the server can preprocess the original image, such as image scaling, image flipping, etc. During the image recognition process, the server can perform artificial intelligence processing on the original image based on the network model to obtain the image processing result of the original image. During the post-processing process, the server implements post-processing of the original image based on the image processing result.

[0046] In the image transmission process, the client can send the original image to the server. If the original image is a sensitive image (such as a face image, a body image, etc.), during the transmission of the original image, if the original image is intercepted by an attacker, the attacker can obtain sensitive information based on the original image, resulting in the leakage of sensitive information, which is easy to leak user privacy information and cause huge losses to the user. Even if the client uses a cryptographic algorithm to encrypt the original image (such as public key encryption or private key encryption, etc.), the server needs to decrypt the original image during the image preprocessing process or image recognition process. After the original image is decrypted, if the attacker intercepts the decrypted original image, it will also cause the leakage of sensitive information.

[0047] In view of the above findings, an image processing method is proposed in this embodiment, which can use scrambling parameters to scramble the original sensitive image to obtain a privacy-protected image. The client sends the privacy-protected image to the server, thereby avoiding the leakage of sensitive information and user privacy information.

[0048] In this embodiment, the image processing system (i.e., the image processing system based on image privacy protection) may include but is not limited to the following processes: image acquisition (or image acquisition) process, image privacy protection process, image transmission process, image preprocessing process, image recognition process, and post-processing process.

[0049] Exemplarily, during the image acquisition process, the client can acquire the original image. For the convenience of distinction, the original image can be recorded as the original sensitive image, indicating that the original image is a sensitive image, such as a face image, a body image, etc., and there is no restriction on the type of the original sensitive image. During the image privacy protection process, the client uses the scrambling parameters to perform a scrambling operation on the original sensitive image. For example, the client can use the scrambling parameters to perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. During the image transmission process, the client sends the privacy-protected image to the server. During the image preprocessing process, the server can preprocess the privacy-protected image, such as performing image scaling, image flipping and other preprocessing. During the image recognition process, the server performs artificial intelligence processing on the privacy-protected image based on the network model, thereby obtaining the image processing result of the privacy-protected image. During the post-processing process, the server implements post-processing of the original sensitive image based on the image processing result.

[0050] For the image transmission process, the client can send the privacy-preserving image to the server. Since the privacy-preserving image is an image that has undergone pixel scrambling operations and / or image block scrambling operations, during the transmission of the privacy-preserving image, even if the privacy-preserving image is intercepted by an attacker, the attacker cannot obtain sensitive information based on the privacy-preserving image, thereby avoiding the leakage of sensitive information and user privacy information. Therefore, the original sensitive image can be protected from privacy, and the method has a high privacy protection capability.

[0051] The server does not need to restore the privacy-preserving image during the image preprocessing or image recognition process. For example, the server can directly input the privacy-preserving image into the network model. In this way, attackers cannot intercept the restored original sensitive image, thereby preventing the leakage of sensitive information.

[0052] In this embodiment, for the image privacy protection process, the client may use scrambling parameters to perform scrambling operations on the original sensitive image to obtain a privacy-protected image. For example, the scrambling parameters may include a pixel value protection key and / or an image block protection key, and the client may perform a pixel scrambling operation on the original sensitive image based on the pixel value protection key, and / or the client may perform an image block scrambling operation on the original sensitive image based on the image block protection key. In summary, in the image privacy protection process, the client may perform a pixel scrambling operation and / or an image block scrambling operation on the original sensitive image, and the image privacy protection process is described below. For example, in order to implement the image privacy protection process, the following method may be used:

[0053] Method 1: Perform pixel scrambling and image block scrambling operations on the original sensitive image to obtain a privacy-preserving image corresponding to the original sensitive image. For example, first perform a pixel scrambling operation on the original sensitive image to obtain an intermediate image after the pixel scrambling operation, and then perform an image block scrambling operation on the intermediate image to obtain a privacy-preserving image. Alternatively, first perform an image block scrambling operation on the original sensitive image to obtain an intermediate image after the image block scrambling operation, and then perform a pixel scrambling operation on the intermediate image to obtain a privacy-preserving image.

[0054] For the convenience of description, the pixel scrambling operation is performed first and then the image block scrambling operation is performed as an example. Figure 2 As shown, after obtaining the original sensitive image, a pixel scrambling operation is performed on the original sensitive image to obtain an intermediate image, and then an image block scrambling operation is performed on the intermediate image to obtain a privacy-preserving image.

[0055] In a possible implementation, the privacy-preserving image may be obtained by taking the following steps:

[0056] Step S11, dividing the original sensitive image into M*N image blocks, where M and N are both positive integers.

[0057] Exemplarily, after obtaining the original sensitive image, the original sensitive image can be divided into M*N image blocks, where M and N can be the same or different. M indicates that there are M image blocks in the horizontal direction of the original sensitive image, and N indicates that there are N image blocks in the vertical direction of the original sensitive image.

[0058] Exemplarily, the sizes of different image blocks may be the same or different. The size of all image blocks is the same as an example for explanation. For example, the size of all image blocks is m*n, where m and n are both positive integers, and m and n may be the same or different. Among them, m means that there are m pixels in the horizontal direction of the image block, and n means that there are n pixels in the vertical direction of the image block.

[0059] Step S12: Select multiple target image blocks to be protected from the M*N image blocks.

[0060] For example, all the image blocks in the M*N image blocks may be used as target image blocks. In this case, it is necessary to perform pixel scrambling operations and image block scrambling operations on all the image blocks.

[0061] For another example, some image blocks among the M*N image blocks may be used as target image blocks. In this case, it is necessary to perform pixel scrambling operations and image block scrambling operations on the some image blocks.

[0062] For the convenience of description, it is taken that all image blocks are used as target image blocks as an example, that is, M*N image blocks of the original sensitive image are used as target image blocks, and scrambling operations are performed on all image blocks.

[0063] Step S13: for each target image block, obtain the pixel value protection key corresponding to the target image block, determine the first scrambling order corresponding to the pixel value protection key, and perform a scrambling operation (i.e., a pixel scrambling operation) on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image after the pixel scrambling operation. The pixel scrambling operation is used to perform a scrambling operation on multiple pixel values ​​in the target image block, and the pixel scrambling operation is an operation to scramble the original order of multiple pixel values ​​in the target image block.

[0064] For example, the pixel value protection keys corresponding to different target image blocks may be the same, or the pixel value protection keys corresponding to different target image blocks may be different. For the convenience of description, the pixel value protection keys corresponding to all target image blocks are the same, and the pixel value protection key is recorded as Key 像素值 .

[0065] For example, assuming that the pixel value protection key range is [a, b], you can generate a random number in the range [a, b] and use this random number as the Key. 像素值 For a large number of original sensitive images, the pixel value protection key corresponding to the target image block of each original sensitive image may be the same, which is the random number.

[0066] In summary, for each target image block of the original sensitive image, this random number can be used as the pixel value protection key corresponding to the target image block, and the pixel value protection key is subsequently recorded as Key 像素值 .

[0067] Exemplarily, a first mapping relationship can be maintained in advance, and the first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling order. See Table 1, which is an example of the first mapping relationship. The pixel value protection key is a value in the value range [a, b]. The scrambling order represents the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation. Different scrambling orders represent different correspondences.

[0068] Table 1

[0069] Pixel value protection key Shuffle order 1 Scramble order 11 2 Scramble order 12 3 Scramble order 13 4 Scramble order 14 5 Scramble order 15 6 Scramble order 16

[0070] For example, after obtaining the Key corresponding to the target image block 像素值 After that, you can use Key 像素值 Query the first mapping relationship shown in Table 1 to obtain Key 像素值 The corresponding first scrambling order. For example, if Key 像素值 is 1, then the first scrambling sequence is scrambling sequence 11. If Key 像素值 is 2, then the first scrambling order is scrambling order 12, and so on, the first scrambling order corresponding to the target image block can be obtained.

[0071] Exemplarily, the first scrambling order represents the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation. See Table 2, which is an example of the first scrambling order. Of course, only one scrambling order is used as an example for explanation here. When the first scrambling order is other scrambling orders, the correspondence shown in Table 2 will change accordingly. This embodiment does not limit this scrambling order.

[0072] Table 2

[0073] Pixel value position before scrambling operation Pixel value position after scrambling operation (1，1) (2，2) (1，2) (1，3) (1，3) (2，1) (2，1) (1，1) (2，2) (2，3) (2，3) (1，2)

[0074] Exemplarily, after obtaining the first scrambling order, a scrambling operation can be performed on multiple pixel values ​​in the target image block based on the first scrambling order. For example, assuming that the size of the target image block is 3*2, then based on the first scrambling order shown in Table 2, the pixel value at pixel position (1, 1) is moved to pixel position (2, 2), the pixel value at pixel position (1, 2) is moved to pixel position (1, 3), the pixel value at pixel position (1, 3) is moved to pixel position (2, 1), the pixel value at pixel position (2, 1) is moved to pixel position (1, 1), the pixel value at pixel position (2, 2) is moved to pixel position (2, 3), and the pixel value at pixel position (2, 3) is moved to pixel position (1, 2). After performing the above scrambling operation on the target image block, the target image block after the scrambling operation can be obtained.

[0075] Obviously, after performing the above scrambling operation on each target image block of the original sensitive image, an intermediate image after the pixel scrambling operation can be obtained, and the intermediate image includes the target image block after the scrambling operation.

[0076] Step S14: Obtain an image block protection key corresponding to the original sensitive image, determine a second scrambling order corresponding to the image block protection key, and perform a scrambling operation (i.e., an image block scrambling operation) on multiple target image blocks in the intermediate image based on the second scrambling order to obtain a privacy-protected image corresponding to the original sensitive image. The image block scrambling operation is used to perform a scrambling operation on multiple target image blocks of the intermediate image, and the image block scrambling operation is an operation to disrupt the original order of multiple target image blocks of the intermediate image.

[0077] For example, the image block protection keys corresponding to different original sensitive images may be the same, or they may be different. For the sake of convenience, the image block protection keys corresponding to all original sensitive images are the same, and the image block protection key is recorded as Key 图像块 .

[0078] For example, assuming that the value range of the image block protection key is [c, d], a random number in the value range [c, d] can be generated and used as the Key 图像块 For a large number of original sensitive images, the image block protection key corresponding to each original sensitive image may be the same, and may all be the random number.

[0079] In summary, for each original sensitive image, this random number can be used as the image block protection key corresponding to the original sensitive image, and the image block protection key is subsequently recorded as Key 图像块 .

[0080] Exemplarily, a second mapping relationship can be maintained in advance, and the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order. See Table 3, which is an example of the second mapping relationship. The image block protection key is a value in the value range [c, d]. The scrambling order represents the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation. Different scrambling orders represent different correspondences.

[0081] Table 3

[0082] Pixel value protection key Shuffle order 1 Scramble order 21 2 Scramble order 22 3 Scramble order 23 4 Scramble order 24

[0083] For example, after obtaining the Key corresponding to the original sensitive image 图像块 After that, you can use Key 图像块 Query the second mapping relationship shown in Table 3 to obtain Key 图像块 The corresponding second scrambling order. For example, if Key 图像块 is 1, then the second scrambling sequence is scrambling sequence 21. If Key 图像块 is 2, then the second scrambling order is scrambling order 22, and so on, the second scrambling order corresponding to the original sensitive image can be obtained.

[0084] Exemplarily, the second scrambling order represents the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation. See Table 4, which is an example of the second scrambling order. Of course, only one scrambling order is used as an example for explanation here. When the second scrambling order is other scrambling orders, the correspondence shown in Table 4 will change accordingly. This embodiment does not limit this scrambling order.

[0085] Table 4

[0086]

[0087]

[0088] Exemplarily, after obtaining the second scrambling order, the scrambling operation can be performed on multiple target image blocks in the intermediate image based on the second scrambling order. For example, assuming that the original sensitive image (i.e., the intermediate image) includes 2*2 image blocks (assuming that these image blocks are all target image blocks), then, based on the second scrambling order shown in Table 4, the image block at image block position (1, 1) is moved to image block position (1, 2), the image block at image block position (1, 2) is moved to image block position (2, 2), the image block at image block position (2, 1) is moved to image block position (1, 1), and the image block at image block position (2, 2) is moved to image block position (2, 1). After performing the above scrambling operation on the intermediate image, the intermediate image after the scrambling operation is the privacy protection image corresponding to the original sensitive image.

[0089] In summary, by performing pixel scrambling and image block scrambling operations on the original sensitive image, the privacy-preserving image corresponding to the original sensitive image can be obtained. The order of pixel scrambling operations (the first scrambling order) is determined by Key 像素值 Control, the order of image block scrambling operation (second scrambling order) is controlled by Key 图像块 Control. Key 图像块 Shared or not shared by all images, Key 像素值 Shared by all image tiles or not.

[0090] Exemplarily, by performing pixel scrambling operation on the original sensitive image, visual texture information can be eliminated, thereby avoiding leakage of texture information of the original sensitive image. By performing image block scrambling operation on the original sensitive image, visual contour information can be eliminated, thereby avoiding leakage of contour information of the original sensitive image.

[0091] For privacy-preserving images that use pixel scrambling and image block scrambling operations, only brute force cracking can restore the privacy-preserving image to the original sensitive image. Assuming that the original sensitive image is a 224*224 face image, and is divided into 16*16 image blocks, it can be divided into 14*14 image blocks. Then, the complexity of brute force cracking is (16*16*3)! *(14*14)!, that is, 768! *196!. Obviously, the complexity of brute force cracking is very large, that is, it has a high privacy protection capability.

[0092] Method 2: Perform pixel scrambling operation on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image is divided into M*N image blocks. Multiple target image blocks to be protected are selected from the M*N image blocks. All of the M*N image blocks can be used as target image blocks, or some of the M*N image blocks can be used as target image blocks. For each target image block, obtain the pixel value protection key corresponding to the target image block, determine the first scrambling order corresponding to the pixel value protection key, and perform a scrambling operation on multiple pixel values ​​in the target image block based on the first scrambling order (i.e., pixel scrambling operation, pixel scrambling operation can be used to perform scrambling operation on multiple pixel values ​​in the image block) to obtain a privacy-protected image after the pixel scrambling operation.

[0093] For example, the implementation process of method 2 can refer to method 1, and will not be repeated here.

[0094] Method 3: Perform an image block scrambling operation on the original sensitive image to obtain a privacy-protected image corresponding to the original sensitive image. For example, the original sensitive image is divided into M*N image blocks. Multiple target image blocks to be protected are selected from the M*N image blocks. All of the M*N image blocks can be used as target image blocks, or some of the M*N image blocks can be used as target image blocks. Obtain the image block protection key corresponding to the original sensitive image, determine the second scrambling order corresponding to the image block protection key, and perform a scrambling operation on multiple target image blocks in the original sensitive image based on the second scrambling order (i.e., image block scrambling operation, image block scrambling operation can be used to perform scrambling operations on multiple target image blocks of the original sensitive image) to obtain a privacy-protected image after the image block scrambling operation.

[0095] For example, the implementation process of method 3 can refer to method 1, which will not be repeated here.

[0096] In this embodiment, for the image recognition process, the server can perform artificial intelligence processing on the privacy-preserving image based on the target network model to obtain the image processing result corresponding to the privacy-preserving image. For example, after receiving the privacy-preserving image, the server inputs the privacy-preserving image into the target network model to obtain the image processing result corresponding to the privacy-preserving image. The image recognition process is described below.

[0097] Exemplarily, a network model (such as a neural network model) can be obtained in advance as a target network model, and the target network model can be a network model including a fully connected layer, that is, the target network model only includes a fully connected layer. Assuming that the size of the input image of the target network model is 224*224*3=150528, and the target network model is used to implement a 10-class image recognition task, the weight of the target network model is a matrix of [150528, 10], that is, the weight of a fully connected layer is a matrix of [150528, 10].

[0098] The target network model can also be a network model containing multiple layers of fully connected layers, that is, the target network model contains multiple layers of fully connected layers. Assuming that the size of the input image of the target network model is 224*224*3=150528, and the target network model is used to implement a 10-class image recognition task, the weight of the first connection layer of the target network model is a matrix of [150528, 10], and there is no restriction on other connection layers of the target network model.

[0099] The target network model can also be a visual self-attention network model (Vision Transformer), which can include an image segmentation layer (used to divide the image into M*N image blocks), a fully connected layer (that is, a fully connected layer shared by all image blocks), a position encoding layer (used to encode the position of each image block), and a self-attention network layer (used for feature fusion). There is no restriction on the network layers of the target network model.

[0100] The target network model may also be a network model of visual self-attention without position coding, that is, the target network model may be a visual self-attention model without position coding (Vision Transformer). The target network model of visual self-attention without position coding may include an image segmentation layer (for dividing the image into M*N image blocks), a fully connected layer (that is, a fully connected layer through which all image blocks pass), and a self-attention network layer (for feature fusion), and there is no restriction on the network layers of the target network model.

[0101] The target network model can also be a network model of a multi-layer perceptron and a mixer, that is, the target network model can be a multi-layer perceptron-mixer (MLP-Mixer) model. The target network model of a multi-layer perceptron and a mixer can include an image segmentation layer (for dividing the image into M*N image blocks), a fully connected layer (that is, all image blocks pass through a shared fully connected layer), and a mixer layer (mixer layer, for feature fusion, in which the mixer layer can be composed of two groups of fully connected layers for feature fusion in the spatial domain and feature fusion in the channel domain respectively), and there is no restriction on the network layers of the target network model.

[0102] Of course, the above are just a few examples of target network models, and there is no restriction on the type of the target network model, as long as the target network model can be obtained, such as the target network model can be a visual self-attention model, MLP-Mixer, Vision Permutator, ConvMixer, etc. For various types of target network models, the model structure of the target network model can have the following characteristics: the image can be divided into M*N image blocks, the image blocks can be feature extracted using a shared fully connected layer, and the features of all image blocks can be sent to a series of isotropic network layers for feature fusion.

[0103] For the convenience of description, the target network model is a visual self-attention network model as an example. The implementation methods of other types of target network models are similar. Figure 3As shown in FIG. 1 , it is a schematic diagram of the structure of the target network model. The target network model may include an image segmentation layer, a fully connected layer, a position encoding layer, and a self-attention network layer. After obtaining the target network model, the following steps can be used to implement the image recognition process:

[0104] Step S21: After receiving the privacy-preserving image, the privacy-preserving image is input into the image segmentation layer of the target network model, and the image segmentation layer divides the privacy-preserving image into M*N image blocks.

[0105] Exemplarily, the sizes of different image blocks may be the same, or may be different. The following description will be given taking the case where the sizes of all image blocks are the same, such as the size of all image blocks is m*n.

[0106] Step S22: After obtaining M*N image blocks, each image block is input into a fully connected layer (i.e., all image blocks are input into a shared fully connected layer), and the shared fully connected layer performs feature extraction on each image block to obtain a fully connected feature of a fixed length corresponding to each image block.

[0107] For example, assuming that the size of each image block is m*n, and the size of the weight of the fully connected layer is m*n, then when image block 1 is input to the fully connected layer, the fully connected layer can map image block 1 to a fixed-length fully connected feature 1 through a shared linear mapping, and when image block 2 is input to the fully connected layer, the fully connected layer can map image block 2 to a fixed-length fully connected feature 2 through a shared linear mapping, and so on. Among them, the linear mapping is a linear operation involving matrix multiplication, that is, a linear operation of matrix multiplication of image blocks (such as image block 1, image block 2, etc.) and the weights of the fully connected layer.

[0108] Step S23, after obtaining the fully connected features corresponding to each image block, the fully connected features corresponding to each image block are input to the position coding layer, and the position coding layer performs position coding on the fully connected features corresponding to each image block to obtain the position coding features corresponding to each image block. For example, the fully connected feature 1 corresponding to the image block 1 can be input to the position coding layer, and the position coding layer encodes the learnable position information on the fully connected feature 1 to obtain the position coding feature 1 containing the position information, and the fully connected feature 2 corresponding to the image block 2 can be input to the position coding layer, and the position coding layer encodes the learnable position information on the fully connected feature 2 to obtain the position coding feature 2 containing the position information, and so on.

[0109] Step S24: After obtaining the position coding features corresponding to each image block, the position coding features corresponding to each image block are input into the self-attention network layer, and the self-attention network layer performs feature fusion on the position coding features corresponding to all image blocks, and finally obtains the image processing result corresponding to the privacy-preserving image.

[0110] Exemplarily, the self-attention network layer may include a self-attention encoder (Transformer Encoder) and a multilayer perceptron (Multilayer Perceptron, MLP), etc. The position encoding features corresponding to each image block may be input into the self-attention encoder, and the self-attention encoder performs self-attention encoding on the position encoding features corresponding to each image block to obtain the self-attention encoding features corresponding to each image block, and the self-attention encoding features corresponding to each image block are input into the multilayer perceptron, and the multilayer perceptron performs feature fusion on the self-attention encoding features corresponding to all image blocks to obtain fused features, and the image processing results corresponding to the privacy-preserving image, such as image recognition results, are determined based on the fused features.

[0111] In summary, the image processing result corresponding to the privacy-preserving image can be obtained, completing the image recognition process.

[0112] In a possible implementation, since the privacy-preserving image is input into the target network model to obtain an image processing result, and the privacy-preserving image is obtained by scrambling the original sensitive image using scrambling parameters, in order to process the privacy-preserving image, the weights in the target network model are also scrambled using the scrambling parameters. For example, the target network model can be obtained in the following manner.

[0113] Method 1: The server can obtain the scrambling parameters from the client, and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model.

[0114] For example, the client may obtain the scrambling parameters and send the scrambling parameters to the server, and the server obtains the scrambling parameters from the client. The scrambling parameters may include a pixel value protection key (which may be denoted as Key 像素值 ) and / or image block protection key (which can be recorded as Key 图像块 For example, if the client trusts the server (that is, the server will not disclose the Key 像素值 and Key 图像块 ), then the client can use the Key 像素值 and Key 图像块 Sent to the server. Among them, Key 像素值 Key to the image privacy protection process 像素值 Same, Key 图像块 Key to the image privacy protection process 图像块 The same is not repeated here.

[0115] Among them, the client trusts the server means that the server provides a secure system operation and key management process, that is, the server guarantees the key 像素值 and Key 图像块 To ensure the communication security, after completing a transformation operation, the Key needs to be deleted on the server. 像素值 and Key 图像块 , and delete the weights before transformation to prevent third-party attacks, and the service provider does not use its own pre-trained weights or downstream task weights to compare with the corresponding transformed weights to try to crack the key Key 像素值 and Key 图像块 .

[0116] Exemplarily, the server can pre-train a network model as the initial network model, such as obtaining the initial network model based on a large amount of training data, and there is no restriction on this training process. The initial network model can be a network model including one fully connected layer; or, the initial network model can be a network model including multiple fully connected layers; or, the initial network model can be a network model of visual self-attention; or, the initial network model can be a network model of visual self-attention without position encoding; or, the initial network model can be a network model of multi-layer perceptron and mixer, and there is no restriction on this.

[0117] Exemplarily, the server may use scrambling parameters to scramble the weights in the initial network model to obtain the target network model. For example, if the initial network model is a network model containing one fully connected layer, that is, the initial network model only contains one fully connected layer, then the scrambling parameters may be used to scramble the weights of one fully connected layer to obtain the target network model. For another example, if the initial network model is a network model containing multiple fully connected layers, that is, the initial network model contains multiple fully connected layers, then the scrambling parameters may be used to scramble the weights of the first connection layer to obtain the target network model. For another example, if the initial network model is a visual self-attention network model, that is, the initial network model includes an image segmentation layer, a fully connected layer, a position encoding layer, and a self-attention network layer, then the scrambling parameters may be used to scramble the weights of the fully connected layer and the weights of the position encoding layer to obtain the target network model, such as using Key 像素值 The weights of the fully connected layer are scrambled and Key 图像块 The weights of the position encoding layer are scrambled. For another example, if the initial network model is a visual self-attention network model without position encoding, that is, the initial network model includes an image segmentation layer, a fully connected layer, and a self-attention network layer, then the scrambling parameters can be used to scramble the weights of the fully connected layer to obtain the target network model, such as using Key 像素值The weights of the fully connected layer are scrambled. For another example, if the initial network model is a network model of a multi-layer perceptron and a mixer, that is, the initial network model includes an image segmentation layer, a fully connected layer, and a mixer layer (in the mixer layer, two groups of fully connected layers perform feature fusion in the spatial domain and the channel domain respectively), then the scrambling parameters can be used to scramble the weights of the fully connected layer and the mixer layer to obtain the target network model, such as using Key 像素值 The weights of the shared fully connected layer are scrambled and Key 图像块 The weights of the fully connected layer responsible for spatial domain feature fusion are scrambled.

[0118] In summary, the model structure that can be adjusted by scrambling parameters includes the following features: dividing the image into blocks, using the shared fully connected layer to extract features from the image blocks, and sending them to the isotropic network layer for feature fusion. Based on this, Key 像素值 The weights of the shared fully connected layer are scrambled using Key 图像块 Scramble the weights of the network layers used to extract spatial information.

[0119] Since the privacy-preserving image cannot be recognized by a normal network model (such as the initial network model), a model weight adjustment method based on a scrambling parameter is proposed in this embodiment. The scrambling parameter is consistent with the scrambling parameter of the image scrambling operation, that is, the weight of the initial network model is scrambled accordingly using the scrambling parameter. The operation is completed once during deployment to obtain a target network model that can correctly adapt to the scrambled data (i.e., the privacy-preserving image), and the scrambled data is processed by the target network model.

[0120] Since the model weight adjustment is independent of the training process, the model weight adjustment operation can be directly performed on the normal network model (such as the initial network model), so that the adjusted network model can adapt to the scrambled data almost without loss of performance. Of course, the network model can also be retrained on the privacy-preserving images and the model weights of the pre-trained model can be adjusted to obtain better performance.

[0121] In a possible implementation, the server uses scrambling parameters to scramble the weights in the initial network model to obtain a target network model, which may include but is not limited to the following steps:

[0122] Step S31: The server determines a first scrambling sequence corresponding to a pixel value protection key.

[0123] For example, if the scrambling parameters include a pixel value protection key, Key 像素值 ,The original sensitive image includes multiple image blocks,Key 像素值It is used to perform pixel scrambling operation on multiple pixel values ​​in the image block, so the Key 像素值 The corresponding first scrambling order. For example, through Key 像素值 Query the configured first mapping relationship and get the Key 像素值 The first scrambling order is used to indicate the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation. The first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling order.

[0124] Exemplarily, the processing of step S31 is similar to that of step S13, and will not be repeated here.

[0125] Step S32: The server determines a second scrambling sequence corresponding to the image block protection key.

[0126] For example, if the scrambling parameters include an image block protection key, that is, Key 图像块 ,The original sensitive image includes multiple image blocks,Key 图像块 It is used to perform image block scrambling operation on multiple image blocks, so the Key can be determined 图像块 The corresponding second scrambling order. For example, through Key 图像块 Query the configured second mapping relationship and get the Key 图像块 A corresponding second scrambling order is used to indicate the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation. The second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

[0127] Exemplarily, the processing of step S32 is similar to that of step S14, and will not be repeated here.

[0128] Step S33, determining the first type of weights in the initial network model, and scrambling the first type of weights in the initial network model based on the first scrambling order to obtain the scrambled initial network model. The first type of weights may be weights in the initial network model that are calculated with pixel values ​​in the image block.

[0129] Exemplarily, the initial network model includes a large number of weights. For each weight, it can be determined whether the weight needs to be calculated with the pixel value in the image block, that is, whether the weight will be calculated with the pixel value in the image block during the operation of the network model. If so, the weight is used as the first type of weight in the initial network model, and if not, the weight is not used as the first type of weight in the initial network model.

[0130] After determining the first type of weights in the initial network model, the first type of weights in the initial network model can be scrambled based on the first scrambling order. The scrambling operation is used to scramble multiple weight values ​​in the first type of weights (the first type of weights can include a large number of weight values, such as m*n weight values, and the size of the first type of weights is the same as the size of the above-mentioned image block, that is, the number of weight values ​​is the same as the number of pixels in the image block), that is, the original order of the multiple weight values ​​in the first type of weights is scrambled. The scrambling operation process of step S33 is similar to the scrambling operation process of step S13, and will not be repeated here.

[0131] Step S34, determining the second type of weights in the initial network model, and scrambling the second type of weights in the initial network model (such as the initial network model after scrambling the first type of weights) based on the second scrambling order to obtain a scrambled target network model. The second type of weights may be weights in the initial network model related to the position information of the image blocks (such as the order of the image blocks).

[0132] Exemplarily, the initial network model includes a large number of weights. For each weight, it can be determined whether the weight is related to the position information of the image block, that is, whether the weight is calculated based on the position information of the image block during the operation of the network model. If so, the weight is used as the second type of weight in the initial network model. If not, the weight is not used as the second type of weight in the initial network model.

[0133] After determining the second type of weights in the initial network model, the second type of weights in the initial network model can be scrambled based on the second scrambling order. The scrambling operation is used to scramble multiple weight values ​​in the second type of weights (the second type of weights may include multiple weight values, such as M*N weight values, and the size of the second type of weights is the same as the number of divisions of the image blocks, that is, the number of weight values ​​is the same as the number of all image blocks), that is, the original order of the multiple weight values ​​in the second type of weights is scrambled. The scrambling operation process of step S34 is similar to the scrambling operation process of step S14, and will not be repeated here.

[0134] In a possible implementation, if the initial network model is a network model of visual self-attention, that is, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then the weights in the fully connected layer can be used as the first type of weights, and the weights in the position encoding layer can be used as the second type of weights. The weights in the fully connected layer in the initial network model can be scrambled based on the first scrambling order, and the weights in the position encoding layer can be scrambled based on the second scrambling order to obtain the target network model.

[0135] For example, you can use Key像素值 The corresponding first scrambling order scrambles the weights in the shared linear mapping so that the scrambled pixels in each image block in the privacy-preserving image can still be calculated with the corresponding linear mapping weights. 图像块 The corresponding second scrambling order scrambles the position coding order so that the image block coding corresponding to each scrambled image block in the privacy-preserving image is still operated with the corresponding position coding, thereby ensuring that the related operations have no loss on the model application performance.

[0136] Method 2: The server obtains target weight information from the client and generates a target network model based on the target weight information; wherein the target weight information may be weight information in a candidate network model, and the candidate network model is obtained after the client scrambles the weights in the initial network model using scrambling parameters.

[0137] For example, the client can obtain the scrambling parameters, but the client does not send the scrambling parameters to the server. The scrambling parameters may include a pixel value protection key (which may be denoted as Key 像素值 ) and / or image block protection key (which can be recorded as Key 图像块 ). Among them, Key 像素值 Key to the image privacy protection process 像素值 Same, Key 图像块 Key to the image privacy protection process 图像块 The same is not repeated here.

[0138] Exemplarily, the client can obtain a network model as the initial network model. For example, the client obtains a configured network model, the weight of the network model is the weight to be trained, the weight to be trained is the weight that has not been trained, and the client obtains the initial network model based on a large amount of training data, and there is no restriction on this training process. For another example, the client obtains a configured network model, the weight of the network model is the pre-trained weight (also called the downstream task weight), the pre-trained weight is the weight that has been trained, such as the weight obtained by training on other data sets, and the client obtains the initial network model based on a small amount of training data, and there is no restriction on this training process. For another example, the client obtains a configured network model, the weight of the network model is the trained weight (the above-mentioned pre-trained weight or downstream task weight can also be used as the trained weight and no longer trained), the trained weight is the weight that has completed training, and the client directly uses the configured network model as the initial network model and no longer trains the configured network model.

[0139] Exemplarily, the client may use scrambling parameters to scramble the weights in the initial network model to obtain a candidate network model. For example, the client may determine a first scrambling order corresponding to a pixel value protection key, and determine a second scrambling order corresponding to an image block protection key. Determine the first type of weights in the initial network model, and scramble the first type of weights in the initial network model based on the first scrambling order to obtain a scrambled initial network model. The first type of weights are weights in the initial network model that are calculated with pixel values ​​in the image block. Determine the second type of weights in the initial network model, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain a scrambled candidate network model. The second type of weights are weights in the initial network model that are related to the position information of the image block. The process of the client scrambling the weights in the initial network model can be referred to Method 1, which will not be repeated here.

[0140] After obtaining the candidate network model, the client can send the target weight information in the candidate network model to the server, so that the server can generate the target network model based on the target weight information. For example, after receiving the target weight information, the server can directly obtain the target network model based on the target weight information, that is, the weight of the target network model is determined based on the target weight information, and the weight of the target network model is the same as the weight of the candidate network model. For another example, after receiving the target weight information, the server can obtain the network model to be trained based on the target weight information, that is, the weight of the network model to be trained is determined based on the target weight information, and then, the network model to be trained is trained with sample data to obtain the trained target network model, that is, training is performed based on the weights provided by the client, thereby obtaining good model performance.

[0141] In one possible implementation, see Figure 4 , which is a structural diagram of an image processing system. The image processing system may include an image acquisition module (for implementing the image acquisition process), an image privacy protection module (for implementing the image privacy protection process), an image recognition module (for implementing the image recognition process) and a post-processing module (for implementing the post-processing process). The image acquisition module and the image privacy protection module are located on the client side, while the image recognition module and the post-processing module are located on the server side.

[0142] The image acquisition module can acquire the original sensitive image and send the original sensitive image to the image privacy protection module. The image privacy protection module can perform pixel scrambling operation and / or image block scrambling operation on the original sensitive image to obtain a privacy-protected image, and send the privacy-protected image to the server.

[0143] After obtaining the privacy-preserving image, the server can provide the privacy-preserving image to the image recognition module. The image recognition module performs artificial intelligence processing on the privacy-preserving image based on the target network model, obtains the image processing result of the privacy-preserving image (such as image detection result, image segmentation result, image recognition result, etc.), and sends the image processing result to the post-processing module. After obtaining the image processing result, the post-processing module can implement post-processing of the original sensitive image based on the image processing result.

[0144] It can be seen from the above technical solutions that in the embodiment of the present application, the original sensitive image can be subjected to pixel scrambling operation and / or image block scrambling operation to obtain a privacy-protected image, and the client sends the privacy-protected image to the server. Even if the privacy-protected image is intercepted by an attacker, the attacker cannot obtain sensitive information based on the privacy-protected image, thereby avoiding the leakage of sensitive information and user privacy information. The original sensitive image can be protected from privacy leakage, and there is no risk of privacy leakage, which has a high privacy protection capability. By performing pixel scrambling operation on the original sensitive image, the visual texture information can be eliminated, thereby avoiding the leakage of the texture information of the original sensitive image. By performing image block scrambling operation on the original sensitive image, the visual contour information can be eliminated, thereby avoiding the leakage of the contour information of the original sensitive image. Using an isotropically independent model structure represented by a visual self-attention model, the target network model is trained on a normal data set, and then the scrambling parameters are used to scramble the weights in the trained model, and the target network model after the scrambling operation can be used for reasoning, and only one-time key configuration and deployment are required, which makes deployment more convenient.

[0145] Based on the same application concept as the above method, an image processing device is proposed in the embodiment of the present application and applied to the server, see Figure 5 FIG. 1 is a schematic diagram of the structure of the device, wherein the device comprises:

[0146] An acquisition module 51 is used to acquire a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter;

[0147] The processing module 52 is used to input the privacy-preserving image into the target network model, obtain the image processing result corresponding to the privacy-preserving image, and output the image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0148] Exemplarily, the processing module 52 is also used to obtain the scrambling parameters from the client, and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model; or, obtain target weight information from the client, and generate the target network model based on the target weight information; the target weight information is the weight information in the candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the initial network model.

[0149] Exemplarily, the processing module 52 uses the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model. Specifically, if the scrambling parameters include a pixel value protection key and an image block protection key, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in an image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model.

[0150] Exemplarily, when the processing module 52 determines the first scrambling order corresponding to the pixel value protection key, it is specifically used to: query the configured first mapping relationship through the pixel value protection key to obtain the first scrambling order corresponding to the pixel value protection key, and the first scrambling order is used to represent the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation; the first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling order; when the processing module 52 determines the second scrambling order corresponding to the image block protection key, it is specifically used to: query the configured second mapping relationship through the image block protection key to obtain the second scrambling order corresponding to the image block protection key, and the second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

[0151] Exemplarily, the processing module 52 performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, and is specifically used to: determine the first type of weights and the second type of weights in the initial network model, scramble the first type of weights in the initial network model based on the first scrambling order, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights are weights in the initial network model that are operated on pixel values ​​in an image block, and the second type of weights are weights in the initial network model that are related to position information of image blocks.

[0152] Exemplarily, the initial network model is a network model including one fully connected layer; or the initial network model is a network model including multiple fully connected layers; or the initial network model is a network model of visual self-attention; or the initial network model is a network model of visual self-attention without position encoding; or the initial network model is a network model of a multi-layer perceptron and a mixer; the processing module 52 performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, specifically used for: if the initial network model is a network model of visual self-attention, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then the weights in the fully connected layer are scrambled based on the first scrambling order, and the weights in the position encoding layer are scrambled based on the second scrambling order to obtain the target network model.

[0153] Based on the same application concept as the above method, a server device is proposed in the embodiment of the present application, see Figure 6 As shown, the server device includes a processor 61 and a machine-readable storage medium 62, and the machine-readable storage medium 62 stores machine-executable instructions that can be executed by the processor 61; the processor 61 is used to execute the machine-executable instructions to implement the image processing method disclosed in the above example of this application.

[0154] Based on the same application concept as the above method, an embodiment of the present application also provides a machine-readable storage medium, on which a number of computer instructions are stored. When the computer instructions are executed by a processor, the image processing method disclosed in the above example of the present application can be implemented.

[0155] The above-mentioned machine-readable storage medium may be any electronic, magnetic, optical or other physical storage device, which may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as CD, DVD, etc.), or similar storage medium, or a combination thereof.

[0156] Based on the same application concept as the above method, an embodiment of the present application also provides an image processing system, which includes: a client, used to use scrambling parameters to perform a scrambling operation on an original sensitive image, obtain a privacy-preserving image corresponding to the original sensitive image, and send the privacy-preserving image to a server; a server, used to obtain a privacy-preserving image corresponding to the original sensitive image, and input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image; wherein the weights in the target network model are scrambled using the scrambling parameters.

[0157] Exemplarily, the client is further used to send the scrambling parameters to the server; the server is further used to obtain the scrambling parameters from the client, and use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model;

[0158] Alternatively, the client is further used to use the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain a candidate network model, and send the target weight information in the candidate network model to the server; the server is further used to obtain the target weight information from the client and generate the target network model based on the target weight information.

[0159] Exemplarily, the scrambling parameters include a pixel value protection key and an image block protection key; the client uses the scrambling parameters to perform a scrambling operation on the original sensitive image, and when obtaining a privacy-protected image corresponding to the original sensitive image, it is specifically used to: divide the original sensitive image into M*N image blocks, and select multiple target image blocks to be protected from the M*N image blocks; for each target image block, obtain the pixel value protection key corresponding to the target image block, determine a first scrambling order corresponding to the pixel value protection key, and perform a scrambling operation on multiple pixel values ​​in the target image block based on the first scrambling order to obtain an intermediate image; obtain the image block protection key corresponding to the original sensitive image, determine a second scrambling order corresponding to the image block protection key, and perform a scrambling operation on multiple target image blocks in the intermediate image based on the second scrambling order to obtain a privacy-protected image corresponding to the original sensitive image.

[0160] Exemplarily, the server uses the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model to obtain the target network model: if the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then:

[0161] Determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; and perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model.

[0162] Exemplarily, the server performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, specifically for: determining the first type of weights and the second type of weights in the initial network model, scrambling the first type of weights in the initial network model based on the first scrambling order, and scrambling the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights are weights in the initial network model that are operated on pixel values ​​in an image block, and the second type of weights are weights in the initial network model that are related to position information of image blocks.

[0163] The systems, devices, modules or units described in the above embodiments may be implemented by a computer entity or by a product having a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver device, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0164] For the convenience of description, the above device is described in various units according to their functions. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0165] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0166] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0167] Moreover, these computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0168] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more boxes. The above description is only the embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. An image processing method, characterized in that: Applied to the server, the method includes: Acquire a privacy-preserving image corresponding to the original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter; The privacy-preserving image is input into a target network model to obtain an image processing result corresponding to the privacy-preserving image, where the image processing result is obtained after processing the privacy-preserving image.

2. The method according to claim 1, characterized in that The weights in the target network model are scrambled using the scrambling parameters, and before the privacy-preserving image is input into the target network model to obtain an image processing result corresponding to the privacy-preserving image, the method further includes: The scrambling parameters are obtained from the client, and the weights in the trained initial network model are scrambled using the scrambling parameters to obtain the target network model.

3. The method according to claim 1, characterized in that The weights in the target network model are scrambled using the scrambling parameters, and before the privacy-preserving image is input into the target network model to obtain an image processing result corresponding to the privacy-preserving image, the method further includes: Acquire target weight information from the client; wherein the target weight information is weight information in a candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the trained initial network model; The target network model is generated based on the target weight information.

4. The method according to claim 2, characterized in that: The step of using the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model includes: If the scrambling parameters include a pixel value protection key and an image block protection key, the original sensitive image includes multiple image blocks, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in the image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: Determine a first scrambling order corresponding to the pixel value protection key; Determining a second scrambling order corresponding to the image block protection key; The weights in the initial network model are scrambled based on the first scrambling order and the second scrambling order to obtain the target network model.

5. The method according to claim 4, characterized in that The determining a first scrambling order corresponding to the pixel value protection key includes: Querying a configured first mapping relationship through the pixel value protection key to obtain a first scrambling order corresponding to the pixel value protection key, wherein the first scrambling order is used to represent a correspondence between a pixel value position before a scrambling operation and a pixel value position after a scrambling operation; wherein the first mapping relationship is used to record a correspondence between a pixel value protection key and a scrambling order; The determining a second scrambling order corresponding to the image block protection key includes: The configured second mapping relationship is queried through the image block protection key to obtain a second scrambling order corresponding to the image block protection key, wherein the second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; wherein the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order.

6. The method according to claim 4, characterized in that The step of performing a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model includes: Determine the first type of weights and the second type of weights in the initial network model, scramble the first type of weights in the initial network model based on the first scrambling order, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein the first type of weights are weights in the initial network model that are operated on pixel values ​​in an image block, and the second type of weights are weights in the initial network model that are related to position information of image blocks.

7. The method according to claim 4 or 6, characterized in that: The initial network model is a network model including one fully connected layer; or, the initial network model is a network model including multiple fully connected layers; or, the initial network model is a network model of visual self-attention; or, the initial network model is a network model of visual self-attention without position encoding; or, the initial network model is a network model of multi-layer perceptron and mixer; The step of performing a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model includes: If the initial network model is a visual self-attention network model, the initial network model at least includes a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then: The weights in the fully connected layer are scrambled based on the first scrambling order, and the weights in the position encoding layer are scrambled based on the second scrambling order to obtain the target network model.

8. An image processing device, characterized in that: Applied to the server, the device comprises: An acquisition module, used to acquire a privacy-preserving image corresponding to an original sensitive image; wherein the privacy-preserving image is an image obtained by scrambling the original sensitive image using a scrambling parameter; A processing module is used to input the privacy-preserving image into a target network model to obtain an image processing result corresponding to the privacy-preserving image, wherein the image processing result is obtained after processing the privacy-preserving image.

9. The device according to claim 8, It is characterized in that in, The weights in the target network model are scrambled using the scrambling parameters, and the processing module is further used to obtain the scrambling parameters from the client, and use the scrambling parameters to scramble the weights in the trained initial network model to obtain the target network model; The processing module is further used to obtain target weight information from the client, wherein the target weight information is weight information in the candidate network model, and the candidate network model is obtained after the client uses the scrambling parameters to scramble the weights in the trained initial network model; generate the target network model based on the target weight information; Wherein, the processing module uses the scrambling parameters to perform a scrambling operation on the weights in the trained initial network model, and when obtaining the target network model, it is specifically used for: if the scrambling parameters include a pixel value protection key and an image block protection key, the pixel value protection key is used to perform a pixel scrambling operation on multiple pixel values ​​in an image block, and the image block protection key is used to perform an image block scrambling operation on multiple image blocks, then: determine a first scrambling order corresponding to the pixel value protection key; determine a second scrambling order corresponding to the image block protection key; perform a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model; Wherein, when the processing module determines the first scrambling order corresponding to the pixel value protection key, it is specifically used to: query the configured first mapping relationship through the pixel value protection key to obtain the first scrambling order corresponding to the pixel value protection key, and the first scrambling order is used to represent the correspondence between the pixel value position before the scrambling operation and the pixel value position after the scrambling operation; the first mapping relationship is used to record the correspondence between the pixel value protection key and the scrambling order; when the processing module determines the second scrambling order corresponding to the image block protection key, it is specifically used to: query the configured second mapping relationship through the image block protection key to obtain the second scrambling order corresponding to the image block protection key, and the second scrambling order is used to represent the correspondence between the image block position before the scrambling operation and the image block position after the scrambling operation; the second mapping relationship is used to record the correspondence between the image block protection key and the scrambling order; Wherein, the processing module performs a scrambling operation on the weights in the initial network model based on the first scrambling order and the second scrambling order, and when obtaining the target network model, is specifically used to: determine the first type of weights and the second type of weights in the initial network model, scramble the first type of weights in the initial network model based on the first scrambling order, and scramble the second type of weights in the initial network model based on the second scrambling order to obtain the target network model; wherein, the first type of weights are weights in the initial network model that are calculated with pixel values ​​in an image block, and the second type of weights are weights in the initial network model that are related to position information of an image block; Among them, the initial network model is a network model including one fully connected layer; or the initial network model is a network model including multiple fully connected layers; or the initial network model is a network model of visual self-attention; or the initial network model is a network model of visual self-attention without position encoding; or the initial network model is a network model of a multi-layer perceptron and a mixer; the processing module scrambles the weights in the initial network model based on the first scrambling order and the second scrambling order to obtain the target network model, specifically used for: if the initial network model is a network model of visual self-attention, the initial network model includes at least a fully connected layer and a position encoding layer, the fully connected layer is used to extract features from each image block, and the position encoding layer is used to perform position encoding on each image block, then the weights in the fully connected layer are scrambled based on the first scrambling order, and the weights in the position encoding layer are scrambled based on the second scrambling order to obtain the target network model.

10. A server device, characterized in that: include: a processor and a machine-readable storage medium storing machine-executable instructions executable by the processor; The processor is used to execute machine executable instructions to implement the method according to any one of claims 1 to 7.

Images