Image processing method and apparatus

Abstract

The application discloses an image processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring an image to be processed; determining a target object in the image to be processed according to the image to be processed and a rule base; and performing a target operation on the target object in the image to be processed. The target object is an object in the image to be processed that is to be subjected to the target operation; and the rule base is extracted based on images in a data set. For a first rule in the rule base that is extracted based on a first image in the data set, the first rule is used to indicate an object included in the first image and a condition of a relationship between the objects, and the object in the first image is labeled with a target label. Since the rules in the rule base are interpretable rules, when the target object in the image to be processed that is to be subjected to the target operation is detected by the method and a corresponding operation is performed on the target object to complete image processing, the method has strong interpretability.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to an image processing method and device. BACKGROUND

[0002] With the development of the information age and the popularity of smart phones, people are used to sharing and exchanging on social platforms through the network. According to incomplete statistics, some social applications will generate hundreds of millions of photos and hundreds of millions of hours of video browsing every day. In addition, personal pictures are also perceived and collected by the outside world in many other different ways, such as mobile application access, cloud storage, and the like.

[0003] However, although sharing and spreading pictures on social applications through the network brings people a lot of convenience, a large amount of personal information contained in the shared and spread pictures has a risk of leakage, that is, there is a risk of user privacy leakage through sharing and spreading pictures. For example, the background in the picture will expose the user's geographical location and whereabouts, and the people appearing in the picture will expose the user's social relationship, and the like. Therefore, it is necessary to perform privacy protection processing on the pictures published and spread by users on the network.

[0004] In the related art, a deep learning model or a support vector machine algorithm model is used to classify pictures into pictures containing privacy objects and pictures not containing privacy objects, and to perform desensitization processing on the pictures containing privacy objects, so as to realize privacy protection of the pictures. However, the deep learning model and the support vector machine algorithm model both depend on a training set and lack interpretability, and thus do not have a use value. Therefore, there is an urgent need for a picture processing method with strong interpretability to realize privacy protection processing of pictures. SUMMARY

[0005] The present application provides an image processing method and device. The rules in the rule library provided in the method are interpretable rules, so that when the target object to which a target operation is to be performed in a to-be-processed image is detected and the corresponding operation is performed on the target object to complete image processing, the method has strong interpretability.

[0006] The technical solutions provided by the present application are as follows:

[0007] In a first aspect, the present application provides an image processing method, comprising: obtaining a to-be-processed image; determining a target object in the to-be-processed image according to the to-be-processed image and a rule base; and performing a target operation on the target object in the to-be-processed image. The target object is an object in the to-be-processed image that is to be subjected to the target operation. The rule base is extracted based on images in a data set. For a first rule in the rule base, the first rule is any rule in the rule base, and the first rule is extracted based on a first image in the data set. The first rule is used to indicate an object in the first image that is labeled with a target label under the condition of an object and a relationship between objects included in the first image. The data set includes a plurality of images, and each image in the plurality of images includes an object labeled with a target label. The object labeled with the target label is an object subjected to the target operation.

[0008] According to the present application, the target object subjected to the target operation in the to-be-processed image can be detected based on the rule base, and the to-be-processed image can be processed after the target operation is performed on the target object. For example, when the target operation is a desensitization operation, the target object is a private object in the to-be-processed image, and thus the privacy protection of the to-be-processed image can be achieved by performing the desensitization operation on the private object in the to-be-processed image. Since the rules in the rule base provided by the present application are explainable rules, the target object subjected to the target operation in the to-be-processed image can be detected, and the to-be-processed image can be processed by performing the target operation on the target object, which has strong explainability.

[0009] In a possible design, when the target label is a label indicating privacy, the target operation is a desensitization operation.

[0010] According to this possible design, the private object in the image can be detected and processed, and the privacy protection of the image is achieved.

[0011] In another possible design, when the target label is a label indicating special attention of a user, the target operation is a highlight display operation.

[0012] According to this possible design, the object attracting the attention of the user in the image can be detected and processed, and the user experience is improved.

[0013] In a further possible design, the rules in the rule base are represented by directed paths in a first knowledge graph, the first knowledge graph is constructed according to the plurality of images, the directed paths in the first knowledge graph include nodes and directed edges, the nodes of the directed paths represent objects contained in the plurality of images, the directed edges of the directed paths are used to describe relationships between the objects contained in the plurality of images, and a last node in a direction of a directed edge in the directed paths is a label node, the label node is used to record an object represented by at least one node in the directed paths, and the object represented by the at least one node is an object in the plurality of images that is labeled with a target label.

[0014] Through this possible design, the common rules contained in the image set are expressed and fused in the form of the knowledge graph, the rule logic is displayed in a visual manner, and any node and edge in the knowledge graph has a real semantic that can be understood by humans, so that the image processing method provided in this application has strong explainability when detecting a target object to be executed on a target operation, and has high detection accuracy.

[0015] In a further possible design, the determining the target object in the to-be-processed image according to the to-be-processed image and the rule base includes: constructing a second knowledge graph according to the to-be-processed image; querying a target path including nodes and directed edges in the second knowledge graph in the first knowledge graph; and determining an object recorded by a label node of the target path as the target object. The nodes in the second knowledge graph represent objects contained in the to-be-processed image, the directed edges in the second knowledge graph are used to describe relationships between the objects contained in the to-be-processed image, and the target path is one of a plurality of directed paths.

[0016] Through this possible design, since the image processing method provided in this application uses the graph path matching manner to infer a target object to be executed on a target operation in a to-be-processed image, and the target object to be executed on the target operation in the to-be-processed image is detected based on the knowledge graph, logical reasoning is adopted, so that the image processing method provided in this application has strong explainability and high detection accuracy when detecting the target object to be executed on the target operation in the to-be-processed image based on the first knowledge graph. In addition, since the reasoning of the graph structure only includes logical implication (i.e., path matching) calculation, that is, the reasoning based on the graph structure is to infer the target object through logical operation, unlike the large-scale neural network calculation in the related art, the complexity of the reasoning is theoretically strictly limited, the computing resources are saved, and thus the reasoning speed is faster than that of the deep learning model.

[0017] In a further possible design, after the determining the target object in the to-be-processed image according to the to-be-processed image and the rule base, the method further includes: in response to a first indication of user feedback, updating the rule base according to the to-be-processed image. The first indication is used to indicate that a target operation is performed on an object other than the target object in the to-be-processed image.

[0018] Through the possible design, the purpose of automatically detecting and updating the rule library based on user feedback is achieved.

[0019] In yet another possible design, when the first knowledge graph is a knowledge graph constructed according to multiple images, and the rules in the rule library are represented by directed paths in the first knowledge graph, the above-mentioned updating the rule library according to the to-be-processed image comprises: deleting a target path in the first knowledge graph; and adding a directed path in the first knowledge graph according to the to-be-processed image and the first indication. Wherein, the nodes in the target path for representing objects and the directed edges in the target path for describing the relationship between the objects exist in the second knowledge graph, and the object recorded by the label node in the target path is the target object, the second knowledge graph is a knowledge graph constructed according to the to-be-processed image; and the label nodes in the added directed path are used to record other objects.

[0020] Through the possible design, since the updating of the knowledge graph according to the user personalized data in the knowledge graph only involves the addition and deletion of the paths in the graph, not only can the demand of rule personalization and dynamic change be efficiently met, but also the efficiency of updating the knowledge graph according to the user personalized data in the present application is higher than that of fine-tuning the deep learning model for detecting the target object in the to-be-processed image in the related art.

[0021] In yet another possible design, the above-mentioned method is executed by a terminal device, and the terminal device is deployed with the rule library. Then, the above-mentioned obtaining the to-be-processed image comprises: obtaining the to-be-processed image stored in the terminal device locally or sent to the terminal device by other devices.

[0022] In yet another possible design, the above-mentioned method is executed by an application server, and the application server is deployed with the rule library. Then, the above-mentioned obtaining the to-be-processed image comprises: receiving the to-be-processed image uploaded by the terminal device.

[0023] Through the two possible designs, the present application can be flexibly applied to different scenarios.

[0024] In a further possible design, before determining the target object in the image to be processed according to the image to be processed and the rule base, the method further includes: constructing a plurality of knowledge graphs according to the plurality of images; adding a label node at the tail end of a directed path containing a preset node in the plurality of knowledge graphs to obtain a first knowledge graph; the number of the plurality of knowledge graphs is the same as the number of images in the data set; the plurality of knowledge graphs include a plurality of directed paths containing the preset node, wherein the nodes on the directed paths represent objects contained in the plurality of images, the directed edges in the directed paths are used to describe the relationship between the objects contained in the plurality of images, and the preset node contained in the directed path is a node located on the directed path and used to represent an object in the plurality of images labeled with a target label; the tail end of the directed path is an end where the terminal point of the direction of the directed edge of the directed path is located, and the label node added at the tail end of the directed path is used to record the object represented by the preset node on the directed path.

[0025] Through the possible design, the first knowledge graph can be constructed based on the data set.

[0026] In a further possible design, after adding the label node at the tail end of the directed path containing the preset node in the plurality of knowledge graphs, the method further includes: merging the directed paths with the added label nodes by the nodes representing the same object and the same type of object on the directed paths with the added label nodes to obtain the first knowledge graph; and recording the merged nodes in the first knowledge graph and the directed paths to which the merged directed edges belong. Wherein, the same type of object refers to the same type of object; the merged node is a node obtained by merging a plurality of nodes belonging to different directed paths; and the merged directed edge is a directed edge obtained by merging a plurality of directed edges belonging to different directed paths.

[0027] Through the possible design, the first knowledge graph can be simplified.

[0028] In yet another possible design, the rule library includes at least one of the following first rule to sixth rule: the first rule: under the condition that the image includes an outdoor scene containing a person, the person wearing a mask, and the person containing a face, the object labeled with the privacy label in the image is a face; the second rule: under the condition that the image includes a bedroom / dormitory scene containing a person and the person containing a face, the object labeled with the privacy label in the image is a bedroom / dormitory and a face; the third rule: under the condition that the image includes an office scene containing a screen and the attribute of the screen is bright, the object labeled with the privacy label in the image is a screen; the fourth rule: under the condition that the image includes an outdoor / garage scene containing a car and the car containing a license plate, the object labeled with the privacy label in the image is a license plate; the fifth rule: under the condition that the image is a chat screenshot and the chat screenshot contains a left-side avatar / left-side nickname, the object labeled with the privacy label in the image is a left-side avatar / left-side nickname; and the sixth rule: under the condition that the image includes a card containing an identity document (ID), name, age, and address, the object labeled with the privacy label in the image is the ID, name, age, and address.

[0029] Through this possible design, typical privacy rules are given.

[0030] In a second aspect, the present application provides an image processing apparatus, which includes one or more units / modules for performing the method provided in the first aspect and any possible design of the first aspect.

[0031] In a possible design, the apparatus includes: an acquisition unit configured to acquire a to-be-processed image; a determination unit configured to determine a target object in the to-be-processed image according to the to-be-processed image and a rule library; and a processing unit configured to perform a target operation on the target object in the to-be-processed image. The target object is an object in the to-be-processed image that is to be subjected to the target operation. The rule library is extracted based on images in a data set. For a first rule in the rule library, the first rule is any rule in the rule library, the first rule is extracted based on a first image in the data set, and the first rule is used to indicate an object in the first image that is labeled with a target label under the condition that the object and a relationship between objects included in the first image. The data set includes a plurality of images, and each image in the plurality of images contains an object labeled with the target label, which is an object subjected to the target operation.

[0032] In another possible design, when the target label is a label representing privacy, the target operation is a desensitization operation.

[0033] In yet another possible design, when the target label is a label representing special attention of a user, the target operation is a highlighting operation.

[0034] In a further possible design, the rules in the rule base are represented by directed paths in a first knowledge graph, and the first knowledge graph is constructed according to the plurality of images. The directed paths in the first knowledge graph include nodes and directed edges, the nodes of the directed paths represent objects contained in the plurality of images, the directed edges of the directed paths are used to describe relationships between the objects contained in the plurality of images, a last node in a direction of a directed edge in the directed paths is a label node, and the label node is used to record an object represented by at least one node in the directed path, the object represented by the at least one node is an object labeled with a target label in the plurality of images.

[0035] In a further possible design, the processing unit is further configured to construct a second knowledge graph according to the to-be-processed image, and query the target path including the nodes and the directed edges in the second knowledge graph in the first knowledge graph. The determining unit is specifically configured to determine the object recorded by the label node of the target path as the target object. The nodes in the second knowledge graph represent objects contained in the to-be-processed image, the directed edges in the second knowledge graph are used to describe relationships between the objects contained in the to-be-processed image, and the target path is one of a plurality of directed paths.

[0036] In a further possible design, the processing unit is further configured to, after determining the target object in the to-be-processed image according to the to-be-processed image and the rule base, update the rule base according to the to-be-processed image in response to a first indication of a user feedback. The first indication is used to instruct to perform the target operation on other objects different from the target object in the to-be-processed image.

[0037] In a further possible design, when the first knowledge graph is a knowledge graph constructed according to the plurality of images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the processing unit is specifically configured to delete the target path in the first knowledge graph, and add a directed path in the first knowledge graph according to the to-be-processed image and the first indication. The nodes used to represent objects and the directed edges used to describe relationships between the objects on the target path exist in a second knowledge graph, the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed according to the to-be-processed image. The label node on the added directed path is used to record other objects.

[0038] In a further possible design, the apparatus is applied to a terminal device, and the terminal device is deployed with the rule base. The obtaining unit is specifically configured to obtain the to-be-processed image stored in the terminal device locally or sent to the terminal device by another device.

[0039] In a further possible design, the apparatus is applied to an application server, and the application server is deployed with the rule base. The obtaining unit is specifically configured to receive the to-be-processed image uploaded by the terminal device.

[0040] In a further possible design, the processing unit is further configured to, before determining the target object in the image to be processed according to the image to be processed and the rule base, construct a plurality of knowledge graphs according to the plurality of images, and add a label node at a tail end of a directed path containing the preset node in the plurality of knowledge graphs to obtain the first knowledge graph. A number of the plurality of knowledge graphs is the same as a number of the images in the data set. The plurality of knowledge graphs includes a plurality of directed paths containing the preset node, wherein a node on the directed path represents an object contained in the plurality of images, a directed edge in the directed path is used to describe a relationship between the objects contained in the plurality of images, the preset node contained in the directed path is a node located on the directed path and used to represent an object in the plurality of images that is labeled with a target label, and the tail end of the directed path is an end at which a terminal point of a direction indicated by the directed edge of the directed path is located. The label node added at the tail end of the directed path is used to record the object represented by the preset node on the directed path.

[0041] In a further possible design, the processing unit is further configured to, after adding the label node at the tail end of the directed path containing the preset node in the plurality of knowledge graphs, merge the directed paths to which the label nodes are added according to nodes representing the same object and the same type of object on the directed paths to which the label nodes are added, to obtain the first knowledge graph, and record a merged node in the first knowledge graph and a directed path to which a merged directed edge belongs. The same type of object refers to the same type of object. The merged node is a node obtained by merging a plurality of nodes belonging to different directed paths. The merged directed edge is a directed edge obtained by merging a plurality of directed edges belonging to different directed paths.

[0042] In a further possible design, the rule base includes at least one of the following first rule to sixth rule: the first rule: in a condition that the image includes an outdoor scene containing a person, the person is wearing a mask, and the person contains a face, an object labeled with a privacy label in the image is a face; the second rule: in a condition that the image includes a bedroom / dormitory scene containing a person and the person contains a face, an object labeled with a privacy label in the image is the bedroom / dormitory and the face; the third rule: in a condition that the image includes an office scene containing a screen and an attribute of the screen is bright, an object labeled with a privacy label in the image is the screen; the fourth rule: in a condition that the image includes an outdoor / garage scene containing a vehicle and the vehicle contains a license plate, an object labeled with a privacy label in the image is the license plate; the fifth rule: in a condition that the image is a chat screenshot and the chat screenshot contains a left-side avatar / left-side nickname, an object labeled with a privacy label in the image is the left-side avatar / left-side nickname; and the sixth rule: in a condition that the image includes a card containing an ID number of an ID card, a name, an age, and an address, an object labeled with a privacy label in the image is the ID number, the name, the age, and the address.

[0043] It should be understood that the description of possible technical solutions and beneficial effects performed by each functional module / unit in the above division can refer to the solutions provided by the first aspect and any possible design of the first aspect, and will not be described here again.

[0044] In a third aspect, the present application provides an image processing apparatus. The apparatus comprises a memory, a communication interface and one or more processors. Wherein the one or more processors receive or send data through the communication interface, and the one or more processors are configured to read program instructions stored in the memory to perform the method provided by the first aspect and any possible design of the first aspect.

[0045] Wherein the image processing apparatus of the second aspect and the third aspect can be any computing device / apparatus with computing processing capability, such as a mobile phone, a tablet, a vehicle, a notebook computer, a general-purpose computer, an artificial intelligence (AI) terminal, a wearable device, or a server such as an application server or a storage server. Or it can be a component in the foregoing computing device / apparatus, such as a single board, or a chip for implementing part or all of the operations in the method provided by any of the aspects and any possible design.

[0046] In a fourth aspect, the present application provides a computer readable storage medium, which is a non-volatile computer readable storage medium, and includes computer program instructions. When the computer program instructions are executed by a computing device or a processor, the computing device or the processor performs the method provided by the first aspect and any possible design of the first aspect.

[0047] In a fifth aspect, the present application provides a computer program product containing instructions, which, when executed by a computing device or a processor, cause the computing device or the processor to perform the method provided by the first aspect and any possible design of the first aspect.

[0048] In a sixth aspect, the present application provides a chip containing a processor for running program instructions or code. The chip or the device containing the chip can be used to perform the method provided by the first aspect and any possible design of the first aspect. For example, the chip further comprises an input interface, an output interface and a memory. Wherein the input interface, the output interface, the processor and the memory of the chip are connected through the internal connection path of the chip, and the memory in the chip is used to store the program instructions or code run by the processor, and the input interface and the output interface of the chip are used for the connection and communication of the chip with other chips or devices.

[0049] It can be understood that any of the above provided image processing apparatuses, computer readable storage media, computer program products or chips, etc. can be applied to the corresponding methods provided above, and thus the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods, which will not be repeated here.

[0050] In the present application, the names of the above image processing apparatuses, etc. do not constitute a limitation on the devices or functional modules themselves, and in actual implementation, these devices or functional modules can appear with other names. As long as the functions of each device or functional module are similar to those in the present application, they are within the protection scope of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 is a schematic diagram of a system framework for implementing image privacy protection;

[0052] Figure 2 is a schematic diagram of an application scenario of the method provided by the embodiment of the present application;

[0053] Figure 3 is another schematic diagram of an application scenario of the method provided by the embodiment of the present application;

[0054] Figure 4 is a schematic diagram of a method for constructing a rule base provided by the embodiment of the present application;

[0055] Figure 5 is a schematic diagram of a method for constructing a first knowledge graph provided by the embodiment of the present application;

[0056] Figure 6 is a schematic diagram of a knowledge graph provided by the embodiment of the present application;

[0057] Figure 7 is a schematic diagram of another knowledge graph provided by the embodiment of the present application;

[0058] Figure 8 is a schematic diagram of still another knowledge graph provided by the embodiment of the present application;

[0059] Figure 9 is a schematic diagram of still another knowledge graph provided by the embodiment of the present application;

[0060] Figure 10 is a schematic diagram of a first knowledge graph provided by the embodiment of the present application;

[0061] Figure 11 is a schematic diagram of another method for constructing a first knowledge graph provided by the embodiment of the present application;

[0062] Figure 12 is a schematic diagram of another first knowledge graph provided by the embodiment of the present application;

[0063] Figure 13 is another schematic diagram of a first knowledge graph provided by an embodiment of the present application;

[0064] Figure 14 is a schematic diagram of a method for image processing provided by an embodiment of the present application;

[0065] Figure 15 is a schematic diagram of a process for determining a target object in a to-be-processed image according to the to-be-processed image and a first knowledge graph provided by an embodiment of the present application;

[0066] Figure 16 is a schematic diagram of a process for querying a target path including nodes and directed edges in a second knowledge graph in a first knowledge graph provided by an embodiment of the present application;

[0067] Figure 17 is a schematic diagram of a process for deleting a target path in a first knowledge graph provided by an embodiment of the present application;

[0068] Figure 18 is a schematic diagram of a process for adding a directed path in a first knowledge graph according to a to-be-processed image and a first indication provided by an embodiment of the present application;

[0069] Figure 19 is another schematic diagram of a method for image processing provided by an embodiment of the present application;

[0070] Figure 20 is a schematic diagram of a structure of an image processing apparatus provided by an embodiment of the present application;

[0071] Figure 21 is another schematic diagram of a structure of an image processing apparatus provided by an embodiment of the present application;

[0072] Figure 22 is a schematic diagram of a structure of a computing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the drawings.

[0074] For the convenience of understanding, the technologies and backgrounds involved in the embodiments of the present application will be explained first.

[0075] 1), expression of graph structure

[0076] The graph structure expression refers to expressing the image content using a directed graph. For example, nodes of the directed graph are constituted by information in the image, and edges of the directed graph are constituted by position relationships and / or semantic relationships between elements / objects (such as backgrounds, objects, etc.) in the image. The directed graph used to express the image content is also referred to as a knowledge graph used to describe the image content.

[0077] In the embodiments of the present application, the objects in the image include but are not limited to articles, people, etc.

[0078] 2), desensitization

[0079] Desensitizing information / data refers to transforming data of certain sensitive information / data by a certain rule, so as to realize reliable protection of sensitive private data.

[0080] Common data desensitization methods include but are not limited to: a, replacement: replacing sensitive data with other values, such as replacing sensitive data with a specific symbol or a random number; b, encryption: encrypting sensitive data, only authorized personnel can decrypt; c, shielding: hiding part of the sensitive data, such as displaying only part of the characters; d, hashing: converting sensitive data into a hash value, and prohibiting reverse restoration; e, data transformation: changing the format or structure of the data; f, generalization: using more general information to replace specific sensitive data; g, date offset: offsetting date-type sensitive data; h, field truncation: only keeping part of the sensitive data; i, adding noise: adding random noise to the sensitive data.

[0081] Data desensitization technology has been widely applied, including safely using desensitized real data sets in development, testing and outsourcing environments, so as to maintain the availability of data while protecting sensitive private data.

[0082] In the field of image processing, methods for desensitizing images in a certain region of a picture include but are not limited to: a, blurring the image region that needs to be desensitized; b, mosaicking the image region that needs to be desensitized; c, setting the pixel value of the image region that needs to be desensitized to 0 or 255; and the like.

[0083] In the related art, in order to realize image privacy protection, reference Figure 1 , Figure 1 shows a system framework schematic diagram for realizing image privacy protection. As Figure 1As shown, the system includes an image privacy detection module and a desensitization module. The image privacy detection module is used to detect privacy objects in an image to be published by the terminal side, and the desensitization module is used to desensitize the image of the detected privacy object. Further, the terminal side publishes the image after desensitization to the network side, for example, to a social network or cloud. Wherein, the objects in the image include but are not limited to objects, people, places, etc. in the image. The privacy object is an object in the image with user privacy information, such as a license plate in the image.

[0084] Wherein, the process of detecting privacy objects in the image to be published by the terminal side by the image privacy detection module generally includes: first extracting the features of the objects from the image to be published by the visual feature extraction network to obtain the probability that the object corresponding to the features has a privacy label, then weighting the probability that the object corresponding to the features has a privacy label by the attention mechanism, and determining whether the object corresponding to the features has a privacy label or not according to the weighted probability, and then determining the object with a privacy label as a privacy object, thus achieving the purpose of detecting privacy objects in the image. Wherein, the visual feature extraction network includes but is not limited to visual geometry group network (VGG) and residual network (RESNET) and the like.

[0085] As can be seen, Figure 1 The system shown uses the binary classification function of the deep learning model to achieve the purpose of detecting privacy objects in the image. However, in this way, the data set used to train the deep learning model lacks explainability, and in the images included in the data set, the privacy rules of the objects are hidden in the model, so it is not intuitive. Moreover, the deep learning model usually needs a large data set to be trained, so the calculation cost is high. In addition, a fixed single deep learning model cannot meet the personalized privacy rule requirements of users, and when the privacy rules contained in the deep learning model do not match the privacy needs of the user, the deep learning model needs to be trained based on the user data, so the cost of personalization is high.

[0086] Based on this, the embodiment of the present application provides an image processing method, which can detect a target object to be executed (or understood as needing to be executed) target operation in a to-be-processed image through a rule base extracted from a data set image, and then execute the target operation on the target object, that is, the processing of the to-be-processed image can be completed. Wherein, the rule in the rule base is used to indicate the object and the relationship between the objects in the image under the condition that the object and the relationship between the objects in the image are labeled with a target label. And the object labeled with the target label in the image is the object to execute the target operation. Therefore, when detecting the target object to be executed in the to-be-processed image through the rule in the rule base, when the object and the relationship between the objects in the to-be-processed image satisfy the object and the relationship between the objects indicated by a certain rule in the rule base, the object labeled with the target label indicated by the rule is determined as the target object to be executed in the to-be-processed image. It can be seen that the method has strong explainability, and the method can improve the accuracy of detecting the target object in the to-be-processed image.

[0087] Wherein, the object in the image can be understood as an element in the image, including but not limited to objects, characters, places and the like contained in the image, without limitation.

[0088] Optionally, when the above-mentioned target label is a label representing privacy (denoted as a privacy label), the above-mentioned target operation is a desensitization operation. In this case, after executing the method provided by the embodiment of the present application, the privacy object in the to-be-processed image can be detected, and the privacy object is executed desensitization processing, thereby realizing the privacy protection of the to-be-processed image, and the method has strong explainability and can improve the accuracy of detecting the target object in the to-be-processed image.

[0089] Optionally, when the above-mentioned target label is a label representing special attention of a user (denoted as a special attention label), the above-mentioned target operation is a highlight display operation. In this case, after executing the method provided by the embodiment of the present application, the object to which the user pays special attention in the to-be-processed image can be determined, and the object is executed highlight display processing, thereby the personalized needs of the user for the to-be-processed image can be met, the user experience is improved, and the method has strong explainability and can improve the accuracy of detecting the target object in the to-be-processed image.

[0090] Reference Figure 2 , Figure 2 An application scenario of the method provided by the embodiment of the present application is shown. As shown in Figure 2 , the application scenario includes a computing device 20. The computing device 20 includes an image processing apparatus. The image processing apparatus is used to obtain a to-be-processed image from the memory of the computing device 20, or receive the to-be-processed image from other devices, and execute the method provided by the embodiment of the present application for the to-be-processed image to complete the processing of the to-be-processed image, for example, complete the processing of the privacy protection of the to-be-processed image.

[0091] In some examples, the computing device 20 is implemented as a terminal device, which includes but is not limited to a mobile phone, a tablet, a vehicle-mounted device, a notebook computer, a general-purpose computer, an AI terminal, a wearable device, and the like.

[0092] In other examples, the computing device 20 is implemented as a server, such as an application server, a storage server, and the like. The application server is, for example, a social application server, and the like, without being limited thereto.

[0093] Reference Figure 3 , Figure 3 Another application scenario of the method provided by the embodiments of the present application is shown. As shown in Figure 3 , the application scenario includes an application server 30 and a terminal device 31. The terminal device 31 communicates with the application server 30 through a network, such as the Internet. After the application server 30 applies the method provided by the embodiments of the present application to the image uploaded / published by the terminal device, the processing of the image uploaded / published by the terminal device can be completed, such as the processing of the privacy protection of the image uploaded / published by the terminal device.

[0094] It should be understood that the above is an exemplary description of the application scenario of the method provided by the embodiments of the present application, and does not constitute a limitation on the application scenario of the method. Those skilled in the art can know that the application scenario can be adjusted according to the application requirements as the business requirements change, and the embodiments of the present application do not enumerate them one by one.

[0095] The embodiments of the present application also provide an image processing apparatus, which is used to execute the image processing method provided by the embodiments of the present application. Exemplarily, the apparatus can be implemented as a computing device or a functional module in a computing device, such as a terminal device, for example, a mobile phone, a tablet, a vehicle-mounted device, a notebook computer, a general-purpose computer, an artificial intelligence (AI) terminal, a wearable device, or a server, for example, an application server, a storage server, and the like. No limitation is made thereto.

[0096] The implementation process of the image processing method provided by the embodiments of the present application is described below.

[0097] Before executing the image processing method provided by the embodiments of the present application, a rule base needs to be constructed based on a data set in advance, so that the image processing apparatus detects the target object to be executed in the target operation in the image to be processed according to the rule base. The rule base is extracted based on the images in the data set. First, the process of constructing the rule base described in the embodiments of the present application is described in detail below.

[0098] Reference Figure 4 , Figure 4A flowchart of a method for constructing a rule library is shown. The method is executed by any computing device (or apparatus) with computing processing capability, which is not limited in the embodiments of the present application. In one example, the method can be executed by the image processing apparatus described above. As shown in Figure 4 The method comprises the following steps.

[0099] Step 101, the computing device acquires a data set.

[0100] The data set comprises multiple images. In the embodiments of the present application, each image in the data set contains an object labeled with a target label, and the object labeled with the target label is an object performing a target operation.

[0101] Optionally, the number of objects labeled with the target label in the image is not limited in the embodiments of the present application. That is, in the embodiments of the present application, for any image in the data set, the image can contain only one object labeled with the target label, or can contain multiple objects labeled with the target label.

[0102] In one possible implementation, the computing device downloads multiple images containing objects labeled with the target label from a network to obtain the data set. Optionally, a third-party device downloads multiple images containing objects labeled with the target label from the network to obtain the data set, and then sends the data set to the computing device or copies the downloaded data set to the computing device through any storage device, which is not limited. In response, the computing device acquires the data set.

[0103] In one example, for the downloaded image containing the object labeled with the target label from the network, the object labeled with the target label in the image is the object performing the target operation in the image. Taking the target label as a privacy label and the target operation as a blurring operation of the image as an example, for any downloaded image from the network, when the image contains a car and the car displayed in the image contains a blurred license plate, the blurred license plate is a specific implementation form of the blurring operation performed on the license plate image. Since the blurring operation is an operation performed on the object with the privacy label, the blurred license plate in the image is the object labeled with the privacy label in the image. It can be seen that the license plate in the image is the user's private information.

[0104] In another possible implementation, the computing device pre-acquires multiple images, and labels the object in the image on which the target operation is expected to be performed with the target label according to the user demand, thereby obtaining the data set. The multiple images can be downloaded from the network or taken by the computing device, which is not limited.

[0105] For example, taking the case that the target label is a privacy label and the target operation is a desensitization operation, for any image acquired by the computing device, when the image contains an office scene and the office scene displayed in the image contains a display screen that is on, i.e., the display screen contains content, since the content displayed on the display screen in the office scene is enterprise internal data, when the content displayed on the display screen is not protected from privacy and is published on a social network or uploaded to the cloud, it will cause the enterprise secret to be leaked, i.e., the content displayed on the display screen in the office scene is the privacy information of the enterprise, therefore, the computing device labels (e.g., manually labels) the privacy label for the display screen that is on in the image.

[0106] At step 102, the computing device extracts rules from the images in the dataset to obtain a rule library.

[0107] The computing device can first detect and identify objects in the image by using a target detection algorithm, and identify the relationships between the objects in the image. Optionally, the computing device also identifies the states and / or attributes of the objects in the image. The relationships between the objects can be logical relationships and / or spatial position relationships, which are not limited.

[0108] Optionally, for each image in the dataset, the computing device can first pre-classify the image according to the objects to be identified in the image, and then detect and identify the objects in the pre-classified image using a corresponding target detection algorithm.

[0109] For example, when the image contains objects, the computing device divides the image into a first category that needs to identify the objects in the image. Then the computing device detects and identifies the objects in the first category image using a target detection algorithm that can detect the objects in the image, and uses a related algorithm to detect the relationships between the objects in the image and the states and / or attributes of the objects. For example, the computing device uses a deep learning model in the related technology (e.g., a faster region with convolutional neural network feature (faster-RCNN) model) to detect and identify the objects in the first category image, and uses a deep learning model in the related technology to identify the relationships between the objects in the image. It should be understood that when the relationships between the objects in the image are identified by the deep learning model, a confidence threshold can be set, and when the confidence of the relationship between the objects detected by the model exceeds the confidence threshold, the model outputs the detected relationship between the objects.

[0110] For example, when an image contains text, the computing device divides the image into a second category that requires recognizing the text in the image. Then the computing device detects and recognizes the text in the second category of images using a target detection algorithm that can detect text in images, and uses a related algorithm to recognize the relationship between the text in the image and the state and / or attributes of the text. In one example, the relationship between the text in the image can be a positional relationship.

[0111] For another example, when an image contains both an object and text, the computing device divides the image into a third category that requires first recognizing the object in the image and then recognizing the text in the image. In this case, the computing device detects and recognizes the object in the third category of images using a target detection algorithm that can detect objects in images, and then detects and recognizes the text in the third category of images using a target detection algorithm that can detect text in images, and uses a related algorithm to recognize the relationship between the objects in the image and the state and / or attributes of the objects. The relationship between the objects in the image may, for example, be a positional relationship between the object and the text in the image, a relationship between the object and the object, a relationship between the text and the text, and the like. Alternatively, when an image contains both an object and text, the computing device divides the image into a fourth category that requires first recognizing the text in the image and then recognizing the object in the image. In this case, the computing device detects and recognizes the text in the third category of images using a target detection algorithm that can detect text in images, and then detects and recognizes the object in the third category of images using a target detection algorithm that can detect objects in images, and uses a related algorithm to recognize the relationship between the objects in the image and the state and / or attributes of the objects.

[0112] Further, the computing device records the objects detected from each image of the data set, the relationship between the objects, the state and / or attributes of the objects, and the objects labeled with the target label, as rules extracted from the images.

[0113] In one example, when the objects detected by the computing device from a first image of the data set include an outdoor scene, a person, a mask, and a face, and the relationship between the detected objects includes: the outdoor scene contains the person, the person is wearing a mask, and the person contains the face, and the object labeled with the privacy label in the first image is the face, the computing device records a first rule extracted from the first image: under the condition that the first image includes an outdoor scene containing a person, the person wearing a mask, and the person containing a face, the object labeled with the privacy label in the first image (i.e., the privacy object) is the face.

[0114] In some embodiments, the first rule can also be expressed as: <outdoor scene, contains, person> ^ <person, wears, mask> ^ <person, contains, face> -> privacy object: face. In the expression, "^" is a conjunction symbol, and conjunction is a concept in logic, which means that the entire compound proposition is true when two or more propositions are true at the same time. "->" represents logical implication, which is a logical relationship, and specifically represents a conditional relationship. For example, if A logically implies B, it is denoted as A -> B, and specifically means that if condition A is true, then B is also true.

[0115] In another example, when the objects detected by the computing device from the second image of the data set include a bedroom / dorm scene, a person, and a face, and the relationships between the detected objects include that the bedroom / dorm scene contains the person, the person contains the face, and the objects labeled with the privacy label in the second image include the bedroom / dorm scene and the face, the computing device records a second rule extracted from the second image: under the condition that the second image includes the bedroom / dorm scene containing the person, and the person containing the face, the objects labeled with the privacy label in the second image are the bedroom / dorm scene and the face. In some embodiments, the second rule can also be expressed as: <bedroom / dorm scene, contains, person> ^ <person, contains, face> -> privacy object: bedroom / dorm scene, face.

[0116] In yet another example, when the objects detected by the computing device from the third image of the data set include an office scene and a screen, and the relationships between the detected objects include that the office scene contains the screen, and the state of the detected objects includes that the state of the screen is on (for example, the state of the screen is determined to be on by the screen containing displayed text / images, or the state of the screen is determined to be on by the pixel value of the area in the image where the screen is located), and the objects labeled with the privacy label in the third image include the screen, the computing device records a third rule extracted from the third image: under the condition that the third image includes the office scene containing the screen, and the state of the screen is on, the objects labeled with the privacy label in the third image are the screen. In some embodiments, the third rule can also be expressed as: <office, contains, screen> ^ <screen, state, on> -> privacy object: screen.

[0117] In yet another example, when the objects detected by the computing device from the fourth image of the dataset include an outdoor / garage scene, a car, a license plate, and the relationships among the detected objects include that the outdoor / garage scene contains the car, the car contains the license plate, and the object labeled with the privacy label in the fourth image is the license plate, the computing device records a fourth rule extracted from the fourth image: under the condition that the fourth image includes an outdoor / garage scene containing a car, and the car contains a license plate, the object labeled with the privacy label in the fourth image is the license plate. In some embodiments, the fourth rule can also be expressed as: <outdoor / garage scene, contains, car> ^ <car, contains, license plate> -> privacy object: license plate.

[0118] In yet another example, when the objects detected by the computing device from the fifth image of the dataset include a left profile picture / left nickname, and it is detected that the fifth image is a chat screenshot, and the relationships among the fifth image and the objects included in the fifth image include that the chat screenshot contains the left profile picture / left nickname, and the object labeled with the privacy label in the fifth image is the left profile picture / left nickname, the computing device records a fifth rule extracted from the fifth image: under the condition that the fifth image is a chat screenshot, and the chat screenshot contains a left profile picture / left nickname, the object labeled with the privacy label in the fifth image is the left profile picture / left nickname. In some embodiments, the fifth rule can also be expressed as: <chat screenshot, contains, left profile picture / left nickname, etc.> -> privacy object: left profile picture / left nickname, etc.

[0119] In yet another example, when the objects detected by the computing device from the sixth image of the dataset include a card, an identity document (ID) / name / age / address, and the relationships among the detected objects include that the card contains the ID / name / age / address, and the object labeled with the privacy label in the sixth image is the ID / name / age / address, the computing device records a sixth rule extracted from the sixth image: under the condition that the sixth image includes a card containing an ID / name / age / address, the object labeled with the privacy label in the sixth image is the ID / name / age / address. In some embodiments, the sixth rule can also be expressed as: <card, contains, ID / name / age / address> -> privacy object: ID / name / age / address, etc.

[0120] Thus, the rule set consisting of the rules (e.g., the first rule to the sixth rule described above) extracted by the computing device based on each image in the dataset, i.e., the rule base described in the embodiments of the present application. As can be seen, for any rule in the rule base, such as the first rule, the first rule is extracted based on the first image in the dataset, and the first rule is used to indicate the object labeled with the target label in the first image under the condition of the objects included in the first image and the relationships among the objects.

[0121] In some embodiments, the rule base described above can be represented as a first knowledge graph (or a rule knowledge graph), and the rules in the rule base can be represented by directed paths in the first knowledge graph. The first knowledge graph is a directed graph that expresses image content through a graph structure. The first knowledge graph is constructed based on multiple images in the dataset. The directed paths in the first knowledge graph include nodes and directed edges. Nodes in the directed paths represent objects contained in the images of the dataset, and the directed edges describe the relationships between the objects contained in the images of the dataset. Furthermore, the last node in the direction pointed to by the directed edge in the directed path is a label node. The label node records the object represented by at least one node on the directed path containing that label node, and the objects represented by at least one node are all objects in the images of the dataset labeled with target labels.

[0122] In this case, refer to Figure 5 , Figure 5 This illustration shows a flowchart of a method for constructing a first knowledge graph according to an embodiment of this application. This method can be executed by any computing device (or apparatus) with computational processing capabilities, and this application embodiment does not limit this. In one example, the method can be executed by the image processing apparatus described above. Figure 5 As shown, after performing step 101, the method performs the following steps 103 to 105.

[0123] Step 103: The computing device detects objects in each image in the dataset and identifies the relationships between objects.

[0124] For a detailed explanation of how the computing device detects objects in the images in the dataset and identifies the relationships between objects, please refer to the relevant description in step 102, which will not be repeated here.

[0125] In one example, the set of objects detected by a computing device from any image (such as image I, where I is a positive integer) in a dataset can be represented as: O I ={o1,o2,…,o n}, and any two objects in the image (such as o) i ,o j The relationship between ) can be denoted as r i,j Where n is a positive integer greater than 1, and I, i, j are integers between [1, n]. Where r i,j It can be a logical relationship or a spatial relationship. As can be seen, o i ,o i and r j,j This constitutes the triple T i,j = <o i ,r i,j ,o j >

[0126] Step 104, the computing device constructs a plurality of knowledge graphs according to the plurality of images included in the data set.

[0127] The computing device can construct a knowledge graph corresponding to each image in the data set according to each image in the data set, taking the objects detected in each image as nodes and the relationships between the objects detected in the image as directed edges. Therefore, the number of knowledge graphs constructed by the computing device according to each image in the data set is the same as the number of images in the data set.

[0128] In combination with the example described in step 103, the computing device takes the object set O I in the image I as nodes of a knowledge graph (denoted as G I ), takes the relationships r i,j between the objects in the image I as directed edges of the knowledge graph G I , and connects the plurality of triples T i,j corresponding to the image I with the same nodes as common nodes, to obtain the knowledge graph G I corresponding to the image I. Therefore, the knowledge graph G I corresponding to the image I can be regarded as a set composed of a plurality of triples T i,j : G I = {T i,j ,i≠j,o i ,o j ∈O I}. Since the knowledge graph G I contains all the objects in the image I and the relationships between the objects, the knowledge graph G I is defined as a relationship graph of the image I.

[0129] In embodiments of the present application, for the scene objects and the objects / persons detected in the image, the direction of the directed edge is usually that the scene object points to the object / person.

[0130] In one example, in combination with the example described in step 103, when the objects detected from the first image include an outdoor scene, a person, a mask, and a face, and the relationships between the detected objects include that the outdoor scene contains the person, the person wears the mask, and the person contains the face, the computing device takes the objects in the first image as nodes and the relationships between the objects as directed edges, to draw a knowledge graph 60 as shown in Figure 6 .

[0131] In embodiments of the present application, when the computing device draws the knowledge graph corresponding to the image, the knowledge graph further includes nodes for representing the state and / or attribute of the objects detected in the image, and includes directed edges from the nodes representing the objects to the nodes representing the state and / or attribute of the objects.

[0132] In one example, in combination with the example described in step 103, when the objects detected by the computing device from the third image include an office scene and a screen, and the relationships between the detected objects include: the office scene contains the screen, and the state of the detected screen is on, the computing device draws a knowledge graph 70 as shown in FIG. 7, taking the objects and the states of the objects in the third image as nodes, and taking the relationships between the objects, and the relationships between the objects and the states as directed edges. Figure 7

[0133] In another example, when the objects detected by the computing device from the seventh image of the data set include a table, a person, a cup, a mobile phone, and a screen, and the relationships between the detected objects include: the table is near the person, the person is holding the cup and the mobile phone, the mobile phone contains the screen, and the states and / or attributes of the detected objects include: the state of the person is standing, the color of the cup is white, and the state of the screen is on, the computing device draws a knowledge graph 80 as shown in FIG. 8, taking the objects and the states / attributes of the objects in the seventh image as nodes, and taking the relationships between the objects, and the relationships between the objects and the states (or attributes) as directed edges. Figure 8

[0134] In yet another example, when the objects detected by the computing device from the eighth image of the data set include a cabinet, a person, a tablet computer, and a screen, and the relationships between the detected objects include: the cabinet is near the person, the person is holding the tablet computer, the tablet computer contains the screen, and the states and / or attributes of the detected objects include: the state of the person is lying, and the state of the screen is on, the computing device draws a knowledge graph 90 as shown in FIG. 9, taking the objects and the states / attributes of the objects in the eighth image as nodes, and taking the relationships between the objects, and the relationships between the objects and the states (or attributes) as directed edges. Figure 9

[0135] Optionally, for the knowledge graph constructed by the computing device for each image, the computing device can use a minimum generalization algorithm to merge the nodes of the knowledge graph to simplify the knowledge graph. For example, when the knowledge graph includes a node representing a mobile phone and a node representing a tablet computer, since the mobile phone and the tablet computer are both electronic devices with screens, the computing device can merge the two nodes into one node representing an electronic device.

[0136] Step 105, the computing device adds a label node to the tail end of the directed path containing the preset node in the plurality of knowledge graphs to obtain a first knowledge graph.

[0137] ​​​The multiple knowledge graphs are constructed based on multiple images included in the data set. It should be understood that at least one directed path is included in one knowledge graph. The directed path in the knowledge graph refers to a path from a node including only outgoing edges to a node including only incoming edges along the direction of the directed edges. The outgoing edges included by a node refer to directed edges pointing from the node to other nodes, and the incoming edges included by a node refer to directed edges pointing from other nodes to the node. In addition, the nodes on the directed path in the knowledge graph represent objects included in the image from which the knowledge graph is constructed, and the directed edges in the directed path in the knowledge graph are used to describe the relationship between the objects included in the image from which the knowledge graph is constructed.

[0138] For example, in combination with Figure 8 The outgoing edges of the node "table" in the knowledge graph 80 refer to the directed edges pointing from the node "table" to the node "person", and the incoming edges of the node "light" refer to the directed edges pointing from the node "screen" to the node "light". From Figure 8 It can be seen that the node "table" includes only outgoing edges, and the nodes "light", "white", and "stand" include only incoming edges, so Figure 8 The knowledge graph 80 shown in FIG. 8 includes three directed paths, which are directed path 1: node "table"→node "person"→node "mobile phone"→node "screen"→node "light", directed path 2: node "table"→node "person"→node "water cup"→node "white", and directed path 3: node "table"→node "person"→node "stand".

[0139] In an embodiment of the present application, the multiple knowledge graphs constructed by the computing device based on the multiple images included in the data set include multiple directed paths containing a preset node. The nodes on the directed path represent the objects included in the multiple images, the directed edges in the directed path are used to describe the relationship between the objects included in the multiple images, and the preset node contained in the directed path is a node located on the directed path and used to represent an object labeled with a target label in the multiple images.

[0140] For example, for any one of the multiple knowledge graphs constructed by the computing device based on the images in the data set, for example, the first graph, when the first graph is constructed based on the first image in the data set, the first graph includes at least one directed path containing a preset node. The nodes on the directed path represent the objects included in the first image, the directed edges in the directed path are used to describe the relationship between the objects included in the first image, and the preset node contained in the directed path is a node located on the directed path and used to represent an object labeled with a target label in the first image. For example, in combination with Figure 6 The preset node contained in the directed path in the knowledge graph 60 is a node representing a face in the first image. For another example, in combination withFigure 7 The preset node contained in the directed path in the knowledge graph 70 is a node representing a screen in the third image.

[0141] It should be understood that, since the number of objects labeled with the target label in the first image used to construct the first graph is one or more, the number of preset nodes in the first graph is also one or more, and further the first graph includes at least one directed path containing a preset node. Each of the directed paths containing a preset node contains at least one preset node on the directed path.

[0142] Specifically, for the first graph constructed based on the first image in the data set, the computing device adds a label node at the tail end of the directed path containing the preset node in the plurality of knowledge graphs to obtain the first knowledge graph, including: the computing device adds a label node at the tail end of the directed path containing the preset node in the first graph to obtain the directed path containing the label node included in the first knowledge graph. The tail end of the directed path refers to the end where the terminal point of the direction of the directed edge on the directed path is located. And the label node added at the tail end of the directed path containing the preset node is used to record the object represented by the preset node on the directed path. For the first graph constructed based on the first image, the label node added at the tail end of the directed path containing the preset node in the first graph is used to record the object represented by the preset node on the directed path, which is labeled with the target label in the first image. Optionally, the computing device can set a label flag for the label node, so that the computing device or the image processing device can identify the label node.

[0143] For example, in combination with Figure 8 , referring to Figure 10 , when the object labeled with the target label in the seventh image used to construct the knowledge graph 80 shown in Figure 8 includes a screen, that is, the node "screen" in the knowledge graph 80 is a preset node, the directed path containing the preset node in the knowledge graph 80 is the directed path 1 described above, and thus the computing device adds a label node at the tail end of the directed path 1 to record the object represented by the preset node in the directed path 1, thereby obtaining the directed path 1 containing the label node included in the first knowledge graph shown in Figure 10 .

[0144] For another example, in combination with Figure 9 , referring to Figure 10 , when the object labeled with the target label in the seventh image used to construct the knowledge graph 80 shown in Figure 9The object in the eighth image of the knowledge graph 90 shown in the figure that is labeled with the target label includes a screen, that is, the node "screen" in the knowledge graph 90 is a preset node, and thus the directed path in the knowledge graph 90 that includes the preset node is the directed path 4 described above: node "cabinet" → node "person" → node "tablet computer" → node "screen" → node "bright", and thus the computing device adds a label node at the tail end of the directed path 4 to record the object represented by the preset node in the directed path 4, thereby obtaining Figure 10 The first knowledge graph shown includes the directed path 4 that includes the label node.

[0145] It should be understood that when a directed path that includes a preset node includes a plurality of preset nodes, the label node added at the tail end of the directed path is used to record the objects represented by the plurality of preset nodes on the directed path, that is, the label node records the set of objects represented by the plurality of preset nodes.

[0146] It should also be understood that when a knowledge graph corresponding to an image includes a plurality of directed paths that include preset nodes, correspondingly, the first knowledge graph that can be constructed by the computing device based on the knowledge graph corresponding to the image includes a plurality of directed paths that include label nodes.

[0147] Optionally, the computing device can use a minimum generalization algorithm to generalize and combine the plurality of directed paths that include label nodes obtained based on the knowledge graph constructed based on the images in the data set, and details are described below with reference to the description of step 106, which will not be described again.

[0148] In this way, based on the above process, the computing device can obtain the directed paths that include label nodes based on the knowledge graph constructed based on each image, thereby obtaining the first knowledge graph composed of the set of directed paths that include label nodes.

[0149] It can also be known from the above that when the directed path (the conjunctive result of the triples corresponding to the image) in the knowledge graph corresponding to the image is defined as the body of a rule, and the label node used to record the set of objects represented by the preset nodes in the directed path (that is, the set of objects labeled with the target label in the image) is defined as the head of the rule, the rule can be expressed as a logical implication expression: body→head. Wherein, body represents a series of conditions, and head represents the set of objects labeled with the target label in the image, so that when body is true as a condition, head is true. In the embodiments of the present application, one logical implication expression represents one rule.

[0150] Since the expression of the first knowledge graph structure is linked in the form of a triple through nodes and edges, the granularity is fine and easy to change, and the nodes and edges both have semantic relationships, so the first knowledge graph structure can be used in various application modes for downstream tasks. Therefore, by representing the rule base through the first knowledge graph, and used for detecting a target object to be operated on in a to-be-processed image in the image processing method described below, the scheme provided in the embodiments of the present application has high scalability.

[0151] To simplify the first knowledge graph constructed based on the images in the data set, reference is made to Figure 11 , Figure 11 Another method for constructing a first knowledge graph provided by the embodiments of the present application is shown in the flowchart. The method is executed by any computing device (or apparatus) with computing processing capability, which is not limited by the embodiments of the present application. In one example, the method can be executed by the image processing apparatus described above. As shown in Figure 11 After steps 101, 103-105 are executed, the method can further execute steps 106-107.

[0152] In step 106, the computing device merges the directed paths with added label nodes to obtain a simplified first knowledge graph by merging the nodes representing the same object and the same type of object on the directed paths with added label nodes.

[0153] The same type of object refers to the same type of object. The directed path with added label nodes is the directed path containing the label nodes obtained after the computing device executes the method shown in Figure 5 For simplicity of description, the directed path with added label nodes is referred to as a rule path (for example, the directed path 1 and the directed path 4 shown in Figure 10 ).

[0154] In one possible implementation, the computing device merges the multiple rule paths obtained from the knowledge graph constructed based on the images in the data set by merging the nodes representing the same object on the rule paths to obtain a simplified first knowledge graph. For example, in combination with Figure 10 When the rule paths obtained after the computing device executes the method shown in Figure 5 include the directed path 1 and the directed path 4 shown in Figure 10 Since the directed path 1 and the directed path 4 contain multiple nodes representing the same object, such as the node “person”, the node “screen”, and the node “bright”, the computing device merges the nodes representing the same object in the directed path 1 and the directed path 4 into one node based on the nodes representing the same object in the directed path 1 and the directed path 4, such as merging the node “person” in the directed path 1 and the directed path 4 into the node “person” shown in Figure 12 , and merging the node “screen” in the directed path 1 and the directed path 4 into the node “screen” shown in Figure 12The node "screen" shown, and the node "light" in directed path 1 and directed path 4 are merged into... Figure 12 The node shown is "lit," thus enabling the merging of directed path 1 and directed path 4, thereby obtaining... Figure 12 The first knowledge graph shown.

[0155] In another possible implementation, the computing device merges multiple rule paths obtained from the knowledge graph constructed from images in the dataset, using nodes representing the same or similar objects on the rule paths, to obtain a first knowledge graph. Merging rule paths based on nodes representing similar objects can also be understood as the computing device using a minimal generalization algorithm from related technologies to merge the rule paths. For example, continuing with... Figure 10 When the computing device executes Figure 5 The rule paths obtained after the method shown include Figure 10 The directed paths 1 and 4 shown contain multiple nodes representing the same object, such as the node "person," the node "screen," and the node "light." They also contain nodes representing similar objects, such as the node "phone" and the node "tablet." Since both phones and tablets are electronic devices, the computing device merges these nodes representing the same object in directed paths 1 and 4 into a single node. For example, it merges the node "person" in directed paths 1 into a single node. Figure 13 The node "person" shown merges the nodes "screen" in directed path 1 and directed path 4 into a single node. Figure 13 The node "screen" shown merges the "lit" nodes in directed path 1 and directed path 4 into a single node. Figure 13 The node shown is "lit". Furthermore, the computing device, based on the nodes representing similar objects in directed path 1 and directed path 4, merges these nodes into a single node that can represent that similar object. For example, it merges the node "phone" in directed path 1 and the node "tablet" in directed path 4 into a single node. Figure 13 The node "electronic device" in the diagram allows for the merging of directed path 1 and directed path 4, thus obtaining... Figure 13 The first knowledge graph shown.

[0156] It should be noted that when the plurality of rule paths are merged based on the nodes representing the same object and the same type of object, the directed edges used to connect the same nodes in different rule paths are deleted, so that in the first knowledge graph obtained after the rule paths are merged, there is only one directed edge between any two nodes connected by a directed edge. This process is also called merging of the same directed edges in different rule paths. Among them, the same directed edge refers to the directed edge connecting the nodes representing the same object or the same type of object in different rule paths. For example, in the example of merging the rule paths based on the nodes representing the same object in step 106, Figure 10 The directed edge used to connect the node "screen" and the node "bright" in the directed path 1 and the directed edge used to connect the node "screen" and the node "bright" in the directed path 4 are the same directed edges. For another example, in combination with Figure 10 Since the node "mobile phone" in the directed path 1 and the node "tablet computer" in the directed path 4 represent the same type of object, i.e., electronic devices, the directed edge used to connect the node "person" and the node "mobile phone" in the directed path 1 and the directed edge used to connect the node "person" and the node "tablet computer" in the directed path 4 are the same directed edges, and the directed edge used to connect the node "mobile phone" and the node "screen" in the directed path 1 and the directed edge used to connect the node "tablet computer" and the node "screen" in the directed path 4 are also the same directed edges.

[0157] It should be further noted that after the computing device obtains the plurality of rule paths from the knowledge graph constructed based on the images in the data set, the plurality of rule paths can be merged into at least one knowledge graph. Moreover, when there is an independent rule path in the plurality of rule paths. Among them, the independent rule path refers to the objects represented by the nodes in the rule path and the objects represented by the nodes in other rule paths are all different and are not the same type of object. That is, the independent rule path cannot be merged with other rule paths. In this case, the first knowledge graph constructed by the computing device based on the data set includes at least one knowledge graph and at least one independent rule path, and each graph in the at least one knowledge graph is obtained by merging at least two rule paths.

[0158] In this way, the expression of a large number of redundant nodes and edges in the rule paths constructed based on the images in the data set can be removed, thereby saving storage resources for storing the first knowledge graph.

[0159] Step 107, the computing device records the rule paths to which the merged nodes and the merged directed edges in the first knowledge graph belong.

[0160] Among them, the merged node is a node obtained by merging a plurality of nodes belonging to different rule paths. For example, in combination with the example of merging the rule paths based on the nodes representing the same object in step 106, Figure 12The nodes "person", "screen" and "bright" in the knowledge graph obtained after the merging rule path shown are merged nodes. For another example, in combination with the example of merging rule paths based on nodes representing the same object and the same type of object in step 106, Figure 13 The nodes "person", "screen", "bright" and "electronic device" in the knowledge graph obtained after the merging of the directed path 1 and the directed path 4 shown are merged nodes.

[0161] The merged directed edge is a directed edge obtained by merging multiple directed edges belonging to different rule paths. Specifically, the merged directed edge is a directed edge obtained by merging multiple same directed edges belonging to different rule paths, and the same directed edge is described in detail with reference to the related description in step 106, which will not be repeated. For example, in combination with the example of the same directed edge in step 106, in the process of merging the directed path 1 and the directed path 4 shown, Figure 10 The merged directed edge is a directed edge obtained by merging multiple directed edges belonging to different rule paths. Specifically, the merged directed edge is a directed edge obtained by merging multiple same directed edges belonging to different rule paths, and the same directed edge is described in detail with reference to the related description in step 106, which will not be repeated. For example, in combination with the example of the same directed edge in step 106, in the process of merging the directed path 1 and the directed path 4 shown, Figure 12 In the knowledge graph shown, the directed edge connecting the node "screen" and the node "bright", and the directed edge connecting the node "bright" and the label node for recording the object "screen" are merged directed edges. For another example, in combination with the example of the same directed edge in step 106, in the process of merging the directed path 1 and the directed path 4 shown, Figure 10 The merged directed edge is a directed edge obtained by merging multiple directed edges belonging to different rule paths. Specifically, the merged directed edge is a directed edge obtained by merging multiple same directed edges belonging to different rule paths, and the same directed edge is described in detail with reference to the related description in step 106, which will not be repeated. For example, in combination with the example of the same directed edge in step 106, in the process of merging the directed path 1 and the directed path 4 shown, Figure 13 In the knowledge graph shown, the directed edge connecting the node "screen" and the node "bright", the directed edge connecting the node "bright" and the label node for recording the object "screen", the directed edge connecting the node "person" and the node "electronic device", and the directed edge connecting the node "electronic device" and the node "screen" are merged directed edges.

[0162] In the embodiments of the present application, one rule path represents one rule, and when the computing device merges multiple rule paths into the first knowledge graph in step 106, it is necessary to record the rule paths to which the merged nodes and the merged directed edges in the first knowledge graph originally belong, and it is necessary to record the rule paths to which each directed edge in the first knowledge graph originally belongs. The rule paths to which the merged nodes and the merged directed edges in the first knowledge graph recorded by the computing device belong, and the rule paths to which each directed edge originally belongs, can be associated with the first knowledge graph as description information of the first knowledge graph, so as to distinguish the rule paths matched with the to-be-processed image when the to-be-processed image is subsequently inferred.

[0163] For example, in combination with the example of the same directed edge in step 106, Figure 10 and Figure 12 Because Figure 12 The merged nodes: the node "person", the node "screen" and the node "bright" shown respectively belong to Figure 10Given directed paths 1 and 4, the computing device records that the node "person" belongs to both directed paths 1 and 4, the node "screen" belongs to both directed paths 1 and 4, and the node "light" belongs to both directed paths 1 and 4. Similarly, because... Figure 12 The merged directed edges shown are: the directed edge connecting the node "Screen" and the node "Light", and the directed edge connecting the node "Light" and the label node used to record the object "Screen". These belong to... Figure 10 As shown in directed path 1 and directed path 4, the computing device records that the directed edge connecting node "screen" and node "light" belongs to directed path 1 and directed path 4, and the directed edge connecting node "light" and the label node used to record object "screen" belongs to directed path 1 and directed path 4.

[0164] For example, combining Figure 10 and Figure 13 ,because Figure 13 The merged nodes shown are: node "person", node "screen", node "light", and node "electronic device", which belong to... Figure 10 Given directed paths 1 and 4, the computing device records: node "person" belongs to both directed paths 1 and 4, node "screen" belongs to both directed paths 1 and 4, node "light" belongs to both directed paths 1 and 4, and node "electronic device" belongs to both directed paths 1 and 4. Similarly, due to... Figure 13 The merged directed edges shown are: the directed edge connecting the node "Screen" and the node "On", the directed edge connecting the node "On" and the label node used to record the object "Screen", the directed edge connecting the node "Person" and the node "Electronic Device", and the directed edge connecting the node "Electronic Device" and the node "Screen", which respectively belong to Figure 10 As shown in directed paths 1 and 4, the computing device records: the directed edge connecting node "screen" and node "light" belongs to directed path 1 and directed path 4; the directed edge connecting node "light" and the label node used to record the object "screen" belongs to directed path 1 and directed path 4; the directed edge connecting node "person" and node "electronic device" belongs to directed path 1 and directed path 4; and the directed edge connecting node "electronic device" and node "screen" belongs to directed path 1 and directed path 4.

[0165] In some examples, steps 104 to 107 can also be understood as follows: for each image in dataset D (such as image I), the computing device draws a knowledge graph G for image I based on the process described in step 104 above. I Next, the computing device is based on G. IStep 105 is executed to obtain at least one directed path (i.e., a rule path) containing a label node. Then, the computing device uses a minimal generalization algorithm to generalize the knowledge graph {G} corresponding to each image in the dataset. I Multiple rule paths obtained from {rule_i, i∈[1,n]} are merged to obtain n (n≤D) directed paths representing rule {rule_i, i∈[1,n]}.

[0166] Each Rule_i can be considered a directed path (rule:body→head) (see above description), where the nodes of the path are objects, and the directed edges represent the relationships between objects. By executing steps 106-107, the computing device can merge identical nodes in all paths {Rule_i, i∈[1,n]} into a new node and record the set of rule paths to which this new node originally belonged, thus obtaining the first knowledge graph (denoted as directed graph DG'). The computing device also records the rule path Rule_j, j∈[1,n] to which each directed edge in DG' originally belonged. Since there may be cases where two nodes in DG' after merging rule paths have multiple directed edges, the computing device merges edges representing the same relationship between two nodes into one edge and records the set of rule paths to which the merged edge originally belonged, thereby simplifying multiple rule paths. In this way, the first knowledge graph G representing the rule base can be obtained. As can be seen, the first knowledge graph G includes at least one graph obtained after merging rule paths, and includes at least one independent rule path (refer to the description of "independent rule path" above).

[0167] Furthermore, when it is passed Figure 4 The rule base constructed by the method is deployed in an image processing device, or it can be used through... Figure 5 or Figure 11 The first knowledge graph constructed by the method is deployed in the image processing device. The image processing device can then detect the target object to be operated on in the image to be processed using the rule base or the first knowledge graph, and complete the processing of the image after performing the target operation on the target object. When the target operation is a desensitization operation, it indicates that the target object is a privacy object in the image to be processed. Therefore, by performing a desensitization operation on the privacy object in the image to be processed, privacy protection of the image to be processed can be achieved. Since the rules in the rule base and the knowledge graph are interpretable, the detection of the target object to be operated on in the image to be processed based on the rule base or the first knowledge graph also has strong interpretability, and the accuracy of detecting the target object in the image to be processed can be improved.

[0168] The process of processing the to-be-processed image based on the rule base or the first knowledge graph constructed above is described below.

[0169] Reference Figure 14 , Figure 14 A method flow diagram for image processing is shown. Optionally, the method can be applied to the implementation environment shown in Figure 2 or Figure 3 , and executed by the image processing device in the implementation environment shown in Figure 2 or Figure 3 , and the image processing device is deployed with a rule base constructed by the method shown in Figure 4 , or deployed with a first knowledge graph constructed by the method shown in Figure 5 or Figure 11 . As shown in Figure 14 , the method includes the following steps 201-203.

[0170] Step 201, the image processing device acquires the to-be-processed image.

[0171] In one possible case, the to-be-processed image is stored locally in the image processing device, so the image processing device acquires the to-be-processed image, including: the image processing device acquires the to-be-processed image stored locally.

[0172] For example, in combination with the application scenario shown in Figure 2 , when the image processing device is applied to a terminal device, i.e., the method is executed by the terminal device, it means that the terminal device is deployed with the rule base or the first knowledge graph described above. In this case, the image processing device acquires the to-be-processed image, including: the terminal device acquires the to-be-processed image stored locally in the terminal device.

[0173] In another possible case, the image processing device acquires the to-be-processed image, including: the image processing device receives the to-be-processed image sent by other equipment.

[0174] For example, in combination with the application scenario shown in Figure 2 , when the image processing device is applied to a terminal device, i.e., the method is executed by the terminal device, it means that the terminal device is deployed with the rule base or the first knowledge graph described above. In this case, the image processing device acquires the to-be-processed image, including: the terminal device receives the to-be-processed image sent by other equipment.

[0175] For another example, in combination with the application scenario shown in Figure 3 , when the image processing device is applied to any application server, i.e., the method is executed by the application server, it means that the application server is deployed with the rule base or the first knowledge graph described above. In this case, the image processing device acquires the to-be-processed image, including: receiving the to-be-processed image uploaded / published by the terminal device.

[0176] In step 202, the image processing apparatus determines the target object in the image to be processed according to the image to be processed and the rule base.

[0177] The target object is an object in the image to be processed that is to be subjected to a target operation. The rule base is extracted based on images in a data set, and the data set includes a plurality of images, each of which contains an object labeled with a target label, and the object labeled with the target label is an object subjected to the target operation. For any rule in the rule base, such as a first rule, the first rule is extracted based on a first image in the data set, and the first rule is used to indicate the object labeled with the target label in the first image under the condition that the first image includes the object and the relationship between the objects. The construction process of the rule base and the related description can be referred to the description of steps 101-102, and will not be repeated here.

[0178] In one possible implementation, the image processing apparatus first detects and identifies the objects in the image to be processed and the relationship between the objects (for details, refer to the related description of step 102), and then queries the rule base according to the objects and the relationship between the objects detected from the image to be processed, to find a target rule in the rule base that includes the objects in the image to be processed and the relationship between the objects. Then, the image processing apparatus determines the object labeled with the target label indicated by the target rule as the target object in the image to be processed. Since the rules in the rule base provided by the embodiments of the present application are interpretable rules, the image processing method provided by the embodiments of the present application has strong interpretability when detecting the target object subjected to the target operation in the image to be processed.

[0179] In another possible implementation, when the rule base is implemented as the first knowledge graph described above, that is, the rules in the rule base are represented by the directed paths in the first knowledge graph, the image processing apparatus determines the target object in the image to be processed according to the image to be processed and the rule base, including: the image processing apparatus determines the target object in the image to be processed according to the image to be processed and the first knowledge graph. For details, refer to the description of steps 2021-2023. Figure 15 , Figure 15 A process diagram for determining a target object in an image to be processed according to the image to be processed and the first knowledge graph is shown. As shown in Figure 15 , the process includes steps 2021-2023.

[0180] In step 2021, the image processing apparatus constructs a second knowledge graph according to the image to be processed.

[0181] In this second knowledge graph, nodes represent objects contained in the image to be processed, and directed edges describe the relationships between objects in the image to be processed. For a detailed explanation of how the image processing device constructs the second knowledge graph based on the image to be processed, please refer to steps 103-104, which describe how the computing device constructs the knowledge graph corresponding to the image based on the images in the dataset; further details will not be repeated here.

[0182] Step 2022: The image processing device queries the first knowledge graph for the target path, which includes nodes and directed edges in the second knowledge graph.

[0183] The target path is one of multiple directed paths contained in the first knowledge graph. The nodes representing objects and the directed edges describing the relationships between objects on the target path exist in both the first and second knowledge graphs. It should be understood that the second knowledge graph includes all nodes on the target path except for the label nodes.

[0184] In some examples, the second knowledge graph constructed by the image processing device based on the image to be processed is G. Img For example, refer to Figure 16 The process by which the image processing device queries the first knowledge graph for a target path that includes nodes and directed edges in the second knowledge graph can be implemented by the following steps S11 to S14. Wherein, G Img = <N Img E Img >,N Img For G Img The set of nodes in E Img For G Img The set of directed edges in the array.

[0185] S11, The image processing device, based on the second knowledge graph G Img The root node is determined in the first knowledge graph to identify the target root node.

[0186] In this context, the root node of the graph refers to a node that includes only outgoing edges, for example... Figure 10 The nodes “table” and “cabinet” shown are both root nodes.

[0187] Optionally, the image processing device can be based on G Img The root node is determined in the first knowledge graph using either method 1 or method 2.

[0188] Method 1, the image processing device according to G Img The root node is traversed through each root node in the first knowledge graph. i ,i∈[1,m]}, thus connecting the first knowledge graph with G ImgThe root node of the first knowledge graph is determined as the target root node. Wherein, i and m are integers, and m represents the number of root nodes in the first knowledge graph.

[0189] In mode 2, the image processing device sequentially points to each root node in the first knowledge graph by a preset pointer, and sequentially determines whether the root node currently pointed to by the pointer is a root node of the second knowledge graph G Img , and determines the root node currently pointed to by the pointer as the target root node when the root node currently pointed to by the pointer is a root node of the second knowledge graph G Img . Wherein, the pointer represents a logical calling sequence, which can be understood as a pointer in the code for implementing the processes described in S11-S14.

[0190] S12, the image processing device determines whether the out-edge e of the node currently pointed to by the pointer in the first knowledge graph exists in the out-edge set E Img of the second knowledge graph G Img .

[0191] Wherein, e E p , E p is the set of all out-edges of the node currently pointed to by the pointer.

[0192] It should be understood that when the image processing device determines the target root node by the above-mentioned mode 1, the image processing device points to the target root node by a preset pointer in S12. When the image processing device determines the target root node by the above-mentioned mode 2, the pointer has already pointed to the target root node.

[0193] It should also be understood that any out-edge e ij of the node can be represented as <o i ,r i,j ,o j >. Wherein, the detailed description of o i ,r i,j ,o j can be referred to the above, and will not be repeated here.

[0194] When the image processing device determines that the out-edge set E p of the node currently pointed to by the pointer contains the out-edge e Img that exists in the out-edge set of the root node of the second knowledge graph G ij =<o i ,r i,j ,o j >, the image processing device executes S13.

[0195] When the image processing device determines that the out-edge set E p of the node currently pointed to by the pointer does not contain the out-edge e Img that exists in the out-edge set of the root node of the second knowledge graph G ij =<oi ,r i,j ,o j If the image processing device determines that it has failed to find the target path in the first knowledge graph, that is, the image processing device has failed to detect the target object in the image to be processed according to the first knowledge graph, and at this time the image processing device ends the process.

[0196] S13, The image processing device moves the pointer to the outgoing edge e. ij The tail node is denoted as N1.

[0197] Here, the tail node of the outgoing edge refers to the node that the outgoing edge points to. That is, the image processing device processes the data according to the outgoing edge e. ij The direction in which the pointer is pointed controls the movement of the pointer to the next node after the currently pointed node, i.e., node N1.

[0198] When the edge e ij If the number is multiple, the image processing device will process each outgoing edge e. ij Execution of S13 to S14 will not be repeated here.

[0199] S14. The image processing device determines whether node N1 is a label node.

[0200] Optionally, the image processing device can determine whether node N1 is a label node based on whether node N1 contains a label marker (refer to the description above). For example, if the image processing device determines that node N1 contains a label marker, then node N1 is determined to be a label node. Conversely, if the image processing device determines that node N1 does not contain a label marker, then node N1 is determined not to be a label node.

[0201] Optionally, since the label node is the last node on the directed path in the first knowledge graph, the image processing device can determine whether node N1 is a label node by determining whether node N1 contains outgoing edges pointing to other nodes. For example, when the image processing device determines that node N1 does not contain outgoing edges pointing to other nodes, it determines that node N1 is the last node of the current path, and thus determines that node N1 is a label node. Conversely, when the image processing device determines that node N1 contains outgoing edges pointing to other nodes, it determines that node N1 is not the last node of the current path, and thus determines that node N1 is not a label node.

[0202] When the image processing device determines that node N1 is a label node, the image processing device determines that the directed path to which the label node belongs is the target path.

[0203] When the image processing device determines that node N1 is not a label node, the image processing device returns to step S12 and re-executes S12 to S14 for the node N1 currently pointed to by the pointer until the target path is determined or the process ends.

[0204] When the image processing apparatus fails to determine the target path in the first knowledge graph, it indicates that the query fails. Alternatively, the image processing apparatus can subsequently add the target path in the first knowledge graph based on the user's processing requirements for the to-be-processed image, that is, add the user's required rules in the rule library. For details, refer to the description of adding rules in the first knowledge graph below.

[0205] In step 2023, the image processing apparatus determines the object recorded by the label node of the target path as the target object in the to-be-processed image.

[0206] It can be understood that when the label node of the target path records multiple objects, it indicates that the number of target objects is multiple. When the label node of the target path only records one object, it indicates that the number of target objects is one.

[0207] In step 203, the image processing apparatus performs a target operation on the target object in the to-be-processed image to complete the processing of the to-be-processed image.

[0208] Alternatively, when the target label annotated in the image of the data set for constructing the first knowledge graph is a privacy label, it indicates that the target object determined based on the first knowledge graph is a privacy object in the to-be-processed image, and therefore the target operation performed by the image processing apparatus on the target object in the to-be-processed image is a desensitization operation. Further, the image processing apparatus performs the desensitization operation on the target object in the to-be-processed image to complete the privacy protection processing of the to-be-processed image.

[0209] Alternatively, when the target label annotated in the image of the data set for constructing the first knowledge graph is a special attention label, it indicates that the target object determined based on the first knowledge graph is an object that the user pays special attention to in the to-be-processed image. Therefore, the target operation performed by the image processing apparatus on the target object in the to-be-processed image is a highlight display operation, and further, the image processing apparatus performs the highlight display operation on the target object in the to-be-processed image to complete the highlight display of the object that the user pays special attention to in the to-be-processed image, thereby improving the user experience.

[0210] Through steps 201-203, the purpose of detecting the target object to be executed in the to-be-processed image based on the first knowledge graph is achieved, and after the target object is executed, the corresponding processing of the to-be-processed image can be realized. For example, when the target operation is a desensitization operation, it indicates that the target object is a privacy object in the to-be-processed image, and thus by performing the desensitization operation on the privacy object in the to-be-processed image, the privacy protection processing of the to-be-processed image can be realized.

[0211] And, since the rules in the rule base provided by the embodiments of the present application are explainable rules, and the embodiments of the present application express and fuse the common rules contained in the image set in the form of a knowledge graph, use a visual way to show the rule logic, and use a graph path matching way to infer the target object of the target operation to be performed in the image to be processed, the image processing method provided by the embodiments of the present application has strong explainability when detecting the target object of the target operation to be performed. Or, it can be understood that any node and edge in the first knowledge graph provided by the embodiments of the present application has a real semantic that can be understood by humans, so this representation method has explainability, and when detecting the target object of the target operation to be performed in the image to be processed based on the first knowledge graph, logical reasoning is used, so the explainability of detecting the target object of the target operation to be performed in the image to be processed based on the first knowledge graph is strong.

[0212] In addition, when detecting the target object of the target operation to be performed in the image to be processed based on the rule base or the first knowledge graph, the accuracy of detecting the target object in the image to be processed can be improved. Compared with the accuracy of detecting the target object of the target operation to be performed in the image to be processed based on the deep learning model in the related art, the scheme provided by the embodiments of the present application has higher accuracy in detecting the target object of the target operation to be performed in the image to be processed.

[0213] For example, Table 1 shows the comparison of the detection accuracy of the related art and the scheme provided by the embodiments of the present application when detecting the target object of the target operation to be performed in the image to be processed. As shown in Table 1, when detecting the target object of the target operation to be performed in the image to be processed by the deep learning model in the related art, the correct rate of the detection result is 0.510, and when detecting the target object of the target operation to be performed in the image to be processed by the scheme provided by the embodiments of the present application, the correct rate of the detection result is 0.807.

[0214] Table 1

[0215] Related Art The Present Invention Accuracy of Detection Results 0.510 0.807

[0216] In addition, since the reasoning of the graph structure only includes logical implication calculation, that is, the reasoning based on the graph structure is to infer the target object through logical operation, without the need for large-scale neural network calculation as in the related art, the complexity of the reasoning is theoretically strictly limited, saving computing resources, so the reasoning speed is faster than the deep learning model. That is, compared with the speed of detecting the target object of the target operation to be performed in the image to be processed based on the deep learning model, the scheme provided by the embodiments of the present application has faster speed in detecting the target object of the target operation to be performed in the image to be processed.

[0217] For example, Table 2 shows a comparison of the detection time of the related art and the scheme provided by the embodiment of the present application when detecting the target object to be executed in the image to be processed. As shown in Table 2, when the deep learning model in the related art is used to detect the target object to be executed in the image to be processed, the detection time is 218 milliseconds, and when the scheme provided by the embodiment of the present application is used to detect the target object to be executed in the image to be processed, the detection time is 201 milliseconds.

[0218] Table 2

[0219] Related Art The Present Invention Time to Complete Detection 218 milliseconds 201 milliseconds

[0220] In some embodiments, the rule library constructed above needs to be updated.

[0221] For example, new rules need to be added in the rule library according to the newly added data set image.

[0222] For another example, rules need to be deleted from the rule library that has been constructed.

[0223] For another example, rules need to be modified in the rule library in response to the user's feedback after using the rule library to detect the image to be processed. The implementation of modifying the rules can be to delete the rules used to detect the results that do not match the user's expected detection results, and then add rules that match the user's expected detection results. In one example, when the image processing device determines that the target object determined in the image to be processed based on the rule library is not the object that the image processing device itself expects to execute the target operation, the image processing device can modify the rules in the rule library. In this case, the image processing device can update the rule library according to the image to be processed in response to the first indication of the user's feedback. The first indication is used to indicate that the target operation is performed on other objects in the image to be processed, which are the objects that the user expects to execute the target operation.

[0224] Optionally, when the rule library is constructed by steps 101-102, the computing device or the image processing device applying the rule library can add rules in the rule library according to new images labeled with target tags, or delete rules in the rule library according to requirements (such as user requirements).

[0225] Optionally, when the rule base is represented by the first knowledge graph described above, the computing device or the image processing apparatus applying the first knowledge graph can add a directed path representing a new rule in the first knowledge graph according to the image newly labeled with the target label, or delete a directed path representing a rule in the first knowledge graph according to a requirement (e.g., a user requirement). It should be understood that when a directed path is deleted or added in the first knowledge graph, the structure and content of other directed paths in the first knowledge graph need to be ensured.

[0226] In one example, when the image processing apparatus performs steps 201-202 described above, the image processing apparatus determines that the target object determined in step 202 is not the object that the user expects to perform the target operation on, and the image processing apparatus can update the first knowledge graph according to the image to be processed in response to the first indication fed back by the user and used to indicate that the target operation is performed on other objects different from the target object in the image to be processed. Specifically, the image processing apparatus can first delete the target path in the first knowledge graph including the label node used to record the target object, and then add a directed path in the first knowledge graph according to the image to be processed and the first indication. The label nodes on the added directed path are used to record the aforementioned other objects.

[0227] The process of deleting the target path in the first knowledge graph including the label node used to record the target object by the image processing apparatus is described below. For example, referring to Figure 17 , the process of deleting the target path in the first knowledge graph by the image processing apparatus can be implemented by S21-S25 described below.

[0228] S21, the image processing apparatus finds whether there is a node Np same as the node pointed to by the pointer in the target path in the first knowledge graph.

[0229] Specifically, when the image processing apparatus performs S21 for the first time, the image processing apparatus can point a preset pointer to the root node in the target path, and then find whether there is a node Np same as the root node pointed to by the pointer in the first knowledge graph.

[0230] The pointer represents a logical calling sequence, for example, it can be understood as a pointer in the code implementing the processes described in S21-S25.

[0231] When the image processing apparatus determines that there is no node Np in the first knowledge graph, it means that the first knowledge graph does not include a path same as the target path, and thus the process of deleting the target path ends.

[0232] When the image processing apparatus determines that there is a node Np in the first knowledge graph, the image processing apparatus records the node Np and adds the node Np to the finding rule set F.

[0233] S22, the image processing apparatus finds whether an edge Ep same as the out edge of the node currently pointed to by the pointer in the target path is contained in all the out edges of the node Np in the first knowledge graph.

[0234] Since the target path is a directed path, if the node on the target path includes an out edge, only one out edge is included. Therefore, it can also be understood that the image processing apparatus finds whether the out edge Ep of the node currently pointed to by the pointer in the target path is included in all the out edges of the node Np in the first knowledge graph. It should be understood that when S22 is performed for the first time, the node currently pointed to by the pointer in the target path is the root node of the target path, and therefore the out edge Ep is the out edge of the root node of the target path.

[0235] When the image processing apparatus determines that all the out edges of the node Np in the first knowledge graph do not contain Ep, it means that the first knowledge graph does not include a path same as the target path, and therefore the process of deleting the target path ends.

[0236] When the image processing apparatus determines that all the out edges of the node Np in the first knowledge graph contain Ep, it records the Ep and adds the Ep to the search rule set F.

[0237] S23, the image processing apparatus moves the pointer to the node pointed to by Ep in the target path in the direction indicated by Ep, denoted as Np_next.

[0238] S24, the image processing apparatus determines whether the node Np_next in the target path is a label node.

[0239] The detailed description of whether the image processing apparatus determines whether the node Np_next is a label node can be referred to the related description in S14 described above, and will not be repeated here.

[0240] When the image processing apparatus determines that the node Np_next is a label node, it means that the process of searching for the object to be deleted in the first knowledge graph in this deletion operation has been completed, and the image processing apparatus continues to perform S25.

[0241] When the image processing apparatus determines that the node Np_next is not a label node, it means that the process of searching for the object to be deleted in the first knowledge graph in this deletion operation has not been completed, and the image processing apparatus performs S21-S24 with respect to the node Np_next currently pointed to by the pointer, and so on, until the process of searching for the object to be deleted in the first knowledge graph is completed.

[0242] S25, the image processing apparatus deletes the target path in the first knowledge graph according to the nodes and edges recorded in the search rule set F obtained by S21-S24.

[0243] Specifically, for any node (or edge) recorded in the rule set F, for example, for node 1 (or edge 1) recorded in the rule set F, the image processing apparatus deletes the identifier (ID) of the target path from the rule path set to which the node 1 (or edge 1) belongs according to the description information of the first knowledge graph. After the ID of the target path is deleted from the rule path set to which the node 1 (or edge 1) belongs, the set is empty, indicating that the node 1 (or edge 1) does not belong to any rule path, at this time, the image processing apparatus deletes the node 1 (or edge 1) in the first knowledge graph.

[0244] In this way, by S21-S25, the purpose of deleting the target path in the first knowledge graph can be achieved, and at the same time of deleting the target path, other rule paths in the first knowledge graph are not affected.

[0245] The process of adding a directed path in the first knowledge graph by the image processing apparatus according to the to-be-processed image and the first indication is described below. For example, referring to Figure 18 The process can be implemented by S31-S34 described below.

[0246] S31, the image processing apparatus generates a to-be-added path according to the to-be-processed image and the first indication.

[0247] The first indication is used to indicate that a target operation is performed on other objects (denoted as expected objects) in the to-be-processed image different from the target object, and the target object is the target object to be executed in the to-be-processed image. The target operation determined in step 202.

[0248] In response to the first indication, the image processing apparatus can determine that the expected object in the to-be-processed image is the object to be executed on the target operation, and therefore the image processing apparatus labels the expected object in the to-be-processed image with a target label corresponding to the target operation. Further, the image processing apparatus generates a rule path containing a label node according to the to-be-processed image labeled with the target label, and the detailed process can refer to the description of steps 103-105 above, which will not be described again. In this way, the image processing apparatus generates a rule path containing a label node according to the to-be-processed image labeled with the target label, which is the to-be-added path that needs to be added to the first knowledge graph.

[0249] S32, the image processing apparatus finds whether the root node currently pointed to by the pointer in the to-be-added path exists in the root node of the first knowledge graph according to the root node currently pointed to by the pointer in the to-be-added path, denoted as Np.

[0250] The pointer indicates a logical calling sequence, for example, can be understood as a pointer in the code for implementing the processes described in S32-S34. It should be understood that when S32 is performed for the first time, the image processing apparatus sets the pointer to point to the root node in the path to be added.

[0251] When the image processing apparatus determines that the node Np does not exist in the root node of the first knowledge graph, it indicates that the root node of the path to be added does not exist in the first knowledge graph, and therefore the image processing apparatus adds the root node Np in the first knowledge graph, and adds the edges contained in the path to be added for the node Np in the first knowledge graph.

[0252] When the image processing apparatus determines that the node Np exists in the root node of the first knowledge graph, S33 is performed.

[0253] S33, the image processing apparatus moves the pointer to the node pointed by the outgoing edge of the node Np in the path to be added, denoted as Np_next.

[0254] S34, the image processing apparatus determines whether the node Np_next in the path to be added is a label node.

[0255] The detailed description of whether the image processing apparatus determines that the node Np_next is a label node can refer to the related description in S14 described above, and will not be repeated here.

[0256] When the image processing apparatus determines that the node Np_next in the path to be added is a label node, it indicates that all the nodes in the path to be added except the label node have been added to the first knowledge graph, and therefore the image processing apparatus adds the label node Np_next after the node Np in the first knowledge graph, thereby completing the addition of the path to be added in the first knowledge graph.

[0257] When the image processing apparatus determines that the node Np_next in the path to be added is not a label node, for the node Np_next currently pointed by the pointer, the image processing apparatus determines whether there is a node Np_next currently pointed by the pointer in the node pointed by the outgoing edge of the node Np in the first knowledge graph. Further, the image processing apparatus performs S33-S34 for the node Np_next currently pointed by the pointer, and repeats until the addition of the path to be added in the first knowledge graph is completed.

[0258] In this way, through S31-S34, the purpose of adding a directed path representing a rule (i.e. a rule path) in the first knowledge graph can be achieved.

[0259] It can be seen that, by Figure 17 and Figure 18The process of updating the first knowledge graph can yield a first knowledge graph that meets the user's personalized needs. Furthermore, the update of the first knowledge graph based on user-personalized data only involves adding or deleting paths within the graph. Therefore, it not only efficiently meets the needs of rule personalization and dynamic changes, but also, compared to related technologies where deep learning models are fine-tuned based on user-personalized data to detect target objects in images to be processed, the efficiency of updating the first knowledge graph based on user-personalized data in this embodiment is higher.

[0260] For example, Table 3 shows a comparison of the time required to fine-tune the deep learning model in the relevant technology based on user-personalized data and to update the first knowledge graph provided in the embodiments of this application. As shown in Table 3, the time required to fine-tune the deep learning model in the relevant technology based on user-personalized data is 5 minutes, while the time required to update the first knowledge graph based on user-personalized data is only 209 milliseconds.

[0261] Table 3

[0262] Related Art The Present Invention Fine Tuning / Update Time 5 minutes 209 milliseconds

[0263] To enhance understanding of the solutions described in the embodiments of this application, further explanation is provided below with specific examples.

[0264] refer to Figure 19 , Figure 19 A schematic diagram of another image processing method provided in an embodiment of this application is shown. For example... Figure 19 As shown, the computing device first acquires the dataset, and based on... Figure 5 or Figure 11 The method constructs a first knowledge graph. Then, the first knowledge graph is deployed on user devices (such as...). Figure 2 The computing device shown or Figure 3 (e.g., application servers shown). Thus, when the user device acquires the image to be processed, it executes... Figure 14 The method shown implements image detection based on a first knowledge graph, processes the image, and outputs the processing result to the user. When the processing result does not meet the user's expectations, the user device receives user feedback and executes... Figure 17 and / or Figure 18 The process described achieves the updating of the first knowledge graph.

[0265] The above mainly describes the solution provided by the embodiments of this application from a methodological perspective.

[0266] To achieve the above functions, this application also provides an image processing apparatus, which includes one or more units / modules for performing the image processing method described above and / or the method for constructing a rule base / first knowledge graph.

[0267] In one example, referring to Figure 20 , Figure 20 A structure diagram of an image processing apparatus provided by an embodiment of the present application is shown. The image processing apparatus 2000 is configured to perform the image processing method described above, for example, to perform the method shown in Figure 4 , Figure 5 , Figure 11 and Figures 14-18 . The image processing apparatus 2000 can include an acquisition unit 2001, a determination unit 2002, and a processing unit 2003.

[0268] The acquisition unit 2001 is configured to acquire a to-be-processed image. The determination unit 2002 is configured to determine a target object in the to-be-processed image according to the to-be-processed image and a rule base. The processing unit 2003 is configured to perform a target operation on the target object in the to-be-processed image. The target object is an object in the to-be-processed image that is to be subjected to the target operation. The rule base is extracted based on images in a data set. For a first rule in the rule base, the first rule is any rule in the rule base, and the first rule is extracted based on a first image in the data set. The first rule is used to indicate an object and a relationship between objects in the first image that are labeled with a target label under the condition that the first image includes the object and the relationship between the objects. The data set includes a plurality of images, and each image in the plurality of images includes an object labeled with the target label. The object labeled with the target label is the object that is subjected to the target operation.

[0269] As an example, in combination with Figure 14 , the acquisition unit 2001 can be configured to perform step 201, the determination unit 2002 can be configured to perform step 202, and the processing unit 2003 can be configured to perform step 203.

[0270] Optionally, when the target label is a label indicating privacy, the target operation is a desensitization operation.

[0271] Optionally, when the target label is a label indicating special attention of a user, the target operation is a highlight display operation.

[0272] Optionally, the rules in the rule base are represented by directed paths in a first knowledge graph, the first knowledge graph is constructed according to the plurality of images, the directed paths in the first knowledge graph include nodes and directed edges, the nodes of the directed paths represent objects included in the plurality of images, the directed edges of the directed paths are used to describe relationships between the objects included in the plurality of images, a last node in a direction of a directed edge in the directed paths is a label node, the label node is used to record objects represented by at least one node on the directed path, and the objects represented by the at least one node are all objects labeled with the target label in the plurality of images.

[0273] Optionally, the processing unit 2003 is further configured to construct a second knowledge graph according to the to-be-processed image, and query the target path including the nodes and directed edges in the second knowledge graph in the first knowledge graph, where the nodes in the second knowledge graph represent objects contained in the to-be-processed image, the directed edges in the second knowledge graph are used to describe relationships between the objects contained in the to-be-processed image, and the target path is one of the multiple directed paths. The determining unit 2002 is specifically configured to determine the object recorded by the label node of the target path as the target object.

[0274] As an example, in combination with Figure 15 The processing unit 2003 can be configured to perform steps 2021-2022, and the determining unit 2002 can be configured to perform step 2023.

[0275] Optionally, after determining the target object in the to-be-processed image according to the to-be-processed image and the rule base, the processing unit 2003 is further configured to update the rule base according to the to-be-processed image in response to a first indication of user feedback. The first indication is used to indicate that the target operation is performed on other objects different from the target object in the to-be-processed image.

[0276] Optionally, when the first knowledge graph is a knowledge graph constructed according to multiple images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the processing unit 2003 is specifically configured to delete the target path in the first knowledge graph, and add a directed path in the first knowledge graph according to the to-be-processed image and the first indication. Wherein the nodes for representing objects and the directed edges for describing relationships between the objects on the target path exist in the second knowledge graph, and the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed according to the to-be-processed image; the label node on the added directed path is used to record other objects.

[0277] As an example, in combination with Figure 17 The processing unit 2003 can be configured to perform S21-S25. In combination with Figure 18 The processing unit 2003 can be configured to perform S31-S34.

[0278] Optionally, the image processing apparatus 2000 is applied to a terminal device, and the terminal device is deployed with a rule base. Therefore, the acquisition unit 2001 is specifically configured to acquire the to-be-processed image stored in the terminal device locally or sent to the terminal by other devices.

[0279] As an example, in combination with Figure 14 The acquisition unit 2001 can be configured to perform step 201.

[0280] Optionally, the image processing apparatus 2000 is applied to an application server, the application server is deployed with a rule base, and the obtaining unit 2001 is specifically configured to receive the to-be-processed image uploaded by the terminal device.

[0281] As an example, in combination with Figure 14 , the obtaining unit 2001 can be configured to perform step 201.

[0282] Optionally, the processing unit 2003 is further configured to, before determining the target object in the to-be-processed image according to the to-be-processed image and the rule base, construct a plurality of knowledge graphs according to a plurality of images; and add a label node at a tail end of a directed path containing a preset node in the plurality of knowledge graphs to obtain a first knowledge graph. The plurality of knowledge graphs have the same number as the number of images in the data set. The plurality of knowledge graphs include a plurality of directed paths containing the preset node. The nodes on the directed paths represent objects contained in the plurality of images. The directed edges in the directed paths are used to describe relationships between the objects contained in the plurality of images. The preset node contained in the directed path is a node located on the directed path and used to represent an object in the plurality of images that is labeled with a target label. The tail end of the directed path is an end where a terminal point of a direction of the directed edge of the directed path is located. The label node added at the tail end of the directed path is used to record the object represented by the preset node on the directed path.

[0283] As an example, in combination with Figure 5 , the processing unit 2003 can be configured to perform steps 104-105.

[0284] Optionally, the processing unit 2003 is further configured to, after adding the label node at the tail end of the directed path containing the preset node in the plurality of knowledge graphs, merge the directed paths with the added label nodes by using nodes representing the same object and the same type of object on the directed paths with the added label nodes to obtain the first knowledge graph; and record a merged node and a directed path to which a merged directed edge belongs in the first knowledge graph. The same type of object refers to the same type of object. The merged node is a node obtained by merging a plurality of nodes belonging to different directed paths. The merged directed edge is a directed edge obtained by merging a plurality of directed edges belonging to different directed paths.

[0285] As an example, in combination with Figure 11 , the processing unit 2003 can be configured to perform steps 106-107.

[0286] Optionally, the rule base comprises at least one of the following first rule to sixth rule: the first rule: under the condition that the image comprises an outdoor scene containing a person, the person wears a mask, and the person contains a face, the object labeled with the privacy label in the image is a face; the second rule: under the condition that the image comprises a bedroom / dormitory scene containing a person, and the person contains a face, the object labeled with the privacy label in the image is a bedroom / dormitory and a face; the third rule: under the condition that the image comprises an office scene containing a screen, and the attribute of the screen is bright, the object labeled with the privacy label in the image is a screen; the fourth rule: under the condition that the image comprises an outdoor / garage scene containing a car, and the car contains a license plate, the object labeled with the privacy label in the image is a license plate; the fifth rule: under the condition that the image is a chat screenshot, and the chat screenshot contains a left-side avatar / left-side nickname, the object labeled with the privacy label in the image is a left-side avatar / left-side nickname; the sixth rule: under the condition that the image comprises a card containing an ID number of an ID card / name / age / address, the object labeled with the privacy label in the image is the ID number / name / age / address.

[0287] For specific description of the above optional mode, please refer to the foregoing method embodiments, which will not be repeated here. In addition, the explanation and beneficial effect of any one of the image processing apparatuses 2000 provided above can refer to the corresponding method embodiments described above, which will not be repeated here.

[0288] As an example, in combination with the following description Figure 22 , the functions implemented by the determination unit 2002 and the processing unit 2003 in the image processing apparatus 2000 can be implemented by the processor 2201 in Figure 22 executing program codes in the memory 2202 in Figure 22 . The functions implemented by the acquisition unit 2001 in the image processing apparatus 2000 can be implemented by the processor 2201 in Figure 22 executing program codes in the memory 2202 in Figure 22 , or by the communication interface 2203 shown in Figure 22 .

[0289] In another example, taking the target label as a privacy label and the target operation as a desensitization operation as an example, referring to Figure 21 , Figure 21 shows another structure of an image processing apparatus provided by the embodiments of the present application. The image processing apparatus 2100 is used to execute the image processing method described above, for example, to execute the methods shown in Figure 4 , Figure 5 , Figure 11 and Figures 14-18 , to realize the protection processing of the privacy object in the image to be processed.

[0290] As Figure 21As shown, the image processing device 2100 deploys the rule base or first knowledge graph described above. The image processing device 2100 also includes an image privacy detection module 2101, an image desensitization module 2102, and a user feedback module 2103. The image privacy detection module 2101 is used to detect privacy objects in the image to be processed based on the rule base or first knowledge graph (implemented through step 202). The image desensitization module 2102 is used to desensitize the detected privacy objects in the image to be processed (implemented through step 203). The user feedback module 2103 is used to obtain user feedback on the image processing results and update the rule base or first knowledge graph based on the feedback (e.g., through...). Figure 17 and / or Figure 18 The update process is implemented as described above. The image processing result is the result after the image privacy detection module 2101 detects privacy objects in the image to be processed, and the image desensitization module 2102 desensitizes the privacy objects in the image to be processed.

[0291] It can be seen that, compared to Figure 1 The image privacy detection framework provided by the related technologies shown in this application includes an image privacy detection module and an image desensitization module, as well as a user feedback module in the image processing device provided in this application embodiment. Therefore, the method provided in this application embodiment supports updating the rule base or the first knowledge graph based on personalized privacy rules based on user feedback. That is, the solution provided in this application embodiment has the ability to automatically detect and update the rule base or the first knowledge graph.

[0292] Those skilled in the art will readily recognize that, based on the units and algorithm steps described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0293] It should be noted that, Figure 20 or Figure 21 The module / unit division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented either in hardware or as software functional modules.

[0294] This application provides a computing device for implementing some or all of the functions in the image processing method provided in this application, or for implementing some or all of the functions in the method for constructing a rule base or a first knowledge graph provided in this application. For example, the computing device is implemented as the image processing apparatus described above.

[0295] refer to Figure 22 , Figure 22 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 22 As shown, the computing device 2200 includes a processor 2201, a memory 2202, a communication interface 2203, and a bus 2204. The processor 2201, memory 2202, and communication interface 2203 are interconnected via the bus 2204. The computing device 2200 also includes an input / output interface 2205, which is interconnected with the processor 2201, memory 2202, and communication interface 2203 via the bus 2204.

[0296] The processor 2201 can include a general-purpose processor and / or a special-purpose hardware chip. The general-purpose processor can include a central processing unit (CPU), a microprocessor, or a graphics processing unit (GPU). The CPU is, for example, a single-CPU, and is, for example, a multi-CPU. The special-purpose hardware chip is a hardware module with high processing performance. The special-purpose hardware chip includes at least one of a digital signal processor (DSP), a data processing unit (DPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, a neural processing unit (NPU), a tensor processing unit (TPU), an artificial intelligent chip, or a network processor (NP). The processor 2201 can also be an integrated circuit chip with processing capability of signals. In the implementation process, part or all of the functions of the method provided by the embodiments of the present application can be completed by integrated logic circuits or software forms of instructions in the processor 2201.

[0297] The memory 2202 is configured to store computer programs, including an operating system 2202a and executable code (i.e., program instructions) 2202b. The memory 2202 is, for example, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), a flash memory, or another type of static storage device that can store static information and instructions, or is, for example, a static RAM (SRAM), a dynamic random access memory (DRAM), a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a synchlink DRAM (SLDRAM), or another type of dynamic storage device that can store information and instructions, or is, for example, a read-only optical disc or another optical disc storage, an optical disc storage (including a compact disc, a laser disc, an optical disc, a digital versatile disc, a Blu-ray disc, etc.), a magnetic disk storage medium or another magnetic storage device, or is any other medium that can be used to carry or store desired executable codes in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. For example, the memory 2202 is configured to store a rule base or a first knowledge graph, etc. The memory 2202 is, for example, independent and connected to the processor 2201 through the bus 2204. Alternatively, the memory 2202 and the processor 2201 are integrated together. The memory 2202 can store executable codes, and when the executable codes stored in the memory 2202 are executed by the processor 2201, the processor 2201 is configured to perform part or all of the functions of the method provided in the embodiments of the present application. For implementation of the processor 2201 to perform the process, please refer to the related description in the foregoing embodiments. The memory 2202 can further include software modules and data required by other running processes such as an operating system.

[0298] The communication interface 2203 uses, for example but is not limited to, a transceiver module such as a transceiver to realize communication with other devices or communication networks. For example, the communication interface 2203 can be any one or any combination of the following devices: a communication interface (such as an Ethernet interface), a wireless network card, etc. A device with network access function. Among them, the communication interface 2203 includes a receiving unit for receiving data / packets, and a sending unit for sending data / packets.

[0299] Bus 2204 is any type of communication bus, for example, a system bus, for interconnecting the internal devices (e.g., memory 2202, processor 2201, communication interface 2203) of computing device 2200. The above-mentioned devices inside computing device 2200 are interconnected by bus 2204 in the embodiments of the present application. Alternatively, the above-mentioned devices inside computing device 2200 can also be communicatively connected to each other in other ways in addition to bus 2204, for example, the above-mentioned devices inside computing device 2200 are interconnected by internal logical interfaces.

[0300] Input / output interface 2205 is used to realize the human-computer interaction between the user and computing device 2200, for example, to realize the text interaction or voice interaction between the user and computing device 2200. Input / output interface 2205 includes an input interface for realizing the input of information by the user to computing device 2200, and an output interface for realizing the output of information by computing device 2200 to the user. As an example, the input interface includes but is not limited to a touch screen, a keyboard, a mouse, a microphone, etc., and the output interface includes but is not limited to a display screen, a speaker, etc. The touch screen, the keyboard, or the mouse are used to input text / image information, the microphone is used to input voice information, and the display screen is used to output text / image information, and the speaker is used to output voice information.

[0301] In one example, the output interface in input / output interface 2205 is used to output the processing result of the to-be-processed image to the user. In another example, the input interface in input / output interface 2205 is used to receive the opinion of the user according to the processing result of the to-be-processed image.

[0302] It should be noted that the above-mentioned devices can be respectively arranged on independent chips, or at least partially or entirely arranged on the same chip. Whether to arrange each device independently on different chips or to integrate on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures each have their own emphasis, and the parts not described in detail in a certain process can be referred to the related description of other processes.

[0303] In the above-mentioned embodiments, all or part of the implementation can be realized by software, hardware, firmware or any combination thereof. When realized by software, all or part of the implementation can be realized in the form of a computer program product. The computer program product providing a program development platform includes one or more computer instructions, which, when loaded and executed on computing device 2200, realize all or part of the functions of the image processing method or the method of constructing the rule library / first knowledge graph provided by the embodiments of the present application.

[0304] Also, the computer instructions can be stored in or transferred from one computer-readable storage medium to another computer-readable storage medium, such as from one website, computer, server or data center to another website, computer, server or data center over a wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) connection. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0305] The embodiments of the present application further provide a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and includes computer program instructions. When the computer program instructions are executed by a computing device, a computer system or a processor, the computing device, the computer system or the processor performs the image processing method and / or the method for constructing a rule base / first knowledge graph provided by the embodiments of the present application.

[0306] The embodiments of the present application further provide a computer program product including instructions, which, when executed by a computing device, a computer system or a processor, cause the computing device, the computer system or the processor to implement the image processing method and / or the method for constructing a rule base / first knowledge graph provided by the embodiments of the present application.

[0307] The computer system is a system with computing processing capability. The computer system generally includes a processor and a memory. The processor is configured to call and run instructions stored in the memory, so that the computer system implements the image processing method and / or the method for constructing a rule base / first knowledge graph described above. Optionally, the computer system can further include at least one of an input interface or an output interface. The processor, the memory, the input interface and the output interface of the computer system are connected through an internal connection path.

[0308] Those skilled in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or by a program instructing related hardware to complete, and the program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk.

[0309] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data for analysis, stored data, displayed data, etc.) and signals involved in the present application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0310] The chip comprises a processor for running program instructions or codes, and the chip or a device comprising the chip can be used to execute the image processing method and / or the method for constructing a rule library / first knowledge graph provided in the embodiments of the present application. Illustratively, the chip further comprises an input interface, an output interface, and a memory. The input interface, the output interface, the processor, and the memory of the chip are connected through an internal connection path of the chip, and the memory in the chip is used to store program instructions or codes run by the processor, and the input interface and the output interface of the chip are used for connection and communication of the chip with other chips or devices.

[0311] In the embodiments of the present application, the terms "first", "second" and "third" are only used for descriptive purposes and should not be construed as indicating or implying relative importance. The term "at least one" refers to one or more, and the term "multiple" refers to at least two, unless otherwise explicitly limited.

[0312] In the present application, the term "and / or" is only used to describe the association relationship of the associated objects, which means that there can be three kinds of relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " in this paper generally represents that the front and rear associated objects are a "or" relationship.

[0313] It should be understood that the terms used in the description of various described examples herein are only for the purpose of describing specific examples and are not intended to be limiting. As used in the description of various described examples and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0314] It should be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information.

[0315] It should be understood that the term "comprise" (also referred to as "includes", "including", "comprises" and / or "comprising") when used in the present specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0316] It should also be understood that in various embodiments of the present application, the size of the serial number of each process does not mean the order of execution, and the execution order of each process should be determined by its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0317] The above merely describes optional embodiments of the present application and is not intended to limit the present application. Any modification, equivalent replacement, improvement, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

[0318] Finally, it should be noted that the above embodiments are used to illustrate the technical solutions of the present application, but not to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalent ones. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a to-be-processed image; determining a target object in the to-be-processed image according to the to-be-processed image and a rule base, the target object being an object in the to-be-processed image that is to be subjected to a target operation; the rule base is extracted based on images in a data set, the data set comprising a plurality of images, each of the plurality of images containing an object labeled with a target label, the object labeled with the target label being an object subjected to the target operation; for a first rule in the rule base, the first rule being any rule in the rule base, the first rule being extracted based on a first image in the data set, the first rule being used to indicate an object and a relationship between objects included in the first image under the condition of the first image labeled with the target label; performing the target operation on the target object in the to-be-processed image.

2. The method of claim 1, wherein, When the target label is a label representing privacy, the target operation is a desensitization operation.

3. The method of claim 1, wherein, When the target label is a label representing special attention of a user, the target operation is a highlighting operation.

4. The method of any one of claims 1 to 3, wherein, The rules in the rule base are represented by directed paths in a first knowledge graph, the first knowledge graph being constructed according to the plurality of images, the directed paths comprising nodes and directed edges, the nodes of the directed paths representing objects contained in the plurality of images, the directed edges of the directed paths being used to describe relationships between the objects contained in the plurality of images; the last node in the direction of the directed edge in the directed path is a label node, the label node being used to record objects represented by at least one node on the directed path, the objects represented by the at least one node all being objects labeled with the target label in the plurality of images.

5. The method of claim 4, wherein, The determining of the target object in the to-be-processed image according to the to-be-processed image and the rule base comprises: constructing a second knowledge graph according to the to-be-processed image, the nodes in the second knowledge graph representing objects contained in the to-be-processed image, the directed edges in the second knowledge graph being used to describe relationships between the objects contained in the to-be-processed image; querying a target path comprising nodes and directed edges in the second knowledge graph in the first knowledge graph, the target path being one of a plurality of directed paths; determining the object recorded by the label node of the target path as the target object.

6. The method of any one of claims 1 to 3, 5, wherein, After the determining of the target object in the to-be-processed image according to the to-be-processed image and the rule base, the method further comprises: in response to a first indication of user feedback, updating the rule base according to the to-be-processed image; wherein the first indication is used to indicate that the target operation is performed on other objects in the to-be-processed image different from the target object.

7. The method of claim 6, wherein, When the first knowledge graph is a knowledge graph constructed according to the plurality of images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the updating of the rule base according to the to-be-processed image comprises: delete a target path in the first knowledge graph; wherein the nodes on the target path for representing objects and the directed edges for describing the relationships between the objects exist in a second knowledge graph, and the object represented by a label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed according to the to-be-processed image; add a directed path in the first knowledge graph according to the to-be-processed image and the first indication, wherein the label nodes on the added directed path are used to record the other objects.

8. The method of any one of claims 1 to 3, 5, 7, wherein, The method is executed by a terminal device, and the terminal device is deployed with the rule library. The method includes: obtaining the to-be-processed image stored in the terminal device locally or sent to the terminal device by another device.

9. The method of any one of claims 1 to 3, 5, 7, wherein, The method is executed by an application server, and the application server is deployed with the rule library. The method includes: receiving the to-be-processed image uploaded by the terminal device.

10. The method of claim 5, wherein, Before determining the target object in the to-be-processed image according to the to-be-processed image and the rule library, the method further includes: constructing a plurality of knowledge graphs according to the plurality of images, wherein the number of the plurality of knowledge graphs is the same as the number of the images in the data set; adding a label node at the tail end of a directed path containing a preset node in the plurality of knowledge graphs to obtain the first knowledge graph; the plurality of knowledge graphs include a plurality of directed paths containing a preset node, wherein the nodes on the directed paths represent the objects contained in the plurality of images, the directed edges in the directed paths are used to describe the relationships between the objects contained in the plurality of images, the preset node contained in the directed path is a node located on the directed path and used to represent an object in the plurality of images labeled with the target label, and the tail end of the directed path is an end where the terminal point of the direction of the directed edge of the directed path is located, and the label node added at the tail end of the directed path is used to record the object represented by the preset node on the directed path.

11. The method of claim 10, wherein, After adding the label node at the tail end of the directed path containing the preset node in the plurality of knowledge graphs, the method further includes: merging the directed paths added with the label nodes to obtain the first knowledge graph through the nodes on the directed paths added with the label nodes representing the same object and the same type of object; wherein the same type of object refers to the same type of object; recording the merged nodes in the first knowledge graph and the directed paths to which the merged directed edges belong, wherein the merged nodes are nodes obtained by merging a plurality of nodes belonging to different directed paths, and the merged directed edges are directed edges obtained by merging a plurality of directed edges belonging to different directed paths.

12. The method of any one of claims 1 to 3, 5, 7, 10, 11, wherein, The rule library includes at least one of the following first rule to sixth rule: First rule: under the condition that an image includes an outdoor scene containing a person, the person wearing a mask, and the person containing a face, the object in the image labeled with a privacy label is a face; Second rule: under the condition that an image includes a bedroom / dormitory scene containing a person and the person containing a face, the object in the image labeled with a privacy label is a bedroom / dormitory and a face; Third rule: in the condition that the image includes an office scene containing a screen, the attribute of the screen is bright, the object labeled with a privacy label in the image is the screen; Fourth rule: in the condition that the image includes an outdoor / garage scene containing a car, the car contains a license plate, the object labeled with a privacy label in the image is the license plate; Fifth rule: in the condition that the image is a chat screenshot, and the chat screenshot contains a left avatar / left nickname, the object labeled with a privacy label in the image is the left avatar / left nickname; Sixth rule: in the condition that the image includes a card containing an ID / name / age / address, the object labeled with a privacy label in the image is the ID / name / age / address.

13. An image processing apparatus characterized by comprising: Comprising: One or more units / modules for performing the method of any of claims 1-12.

14. An image processing apparatus characterized by comprising: Comprising: A memory, a communication interface, and one or more processors that receive or send data through the communication interface, the one or more processors being configured to read program instructions stored in the memory to perform the method of any of claims 1-12.

15. A computer readable storage medium characterized by: Computer program instructions that, when executed by a computing device or processor, cause the computing device or processor to perform the method of any of claims 1-12.

16. A computer program product comprising instructions, characterized in that, Computer program instructions that, when executed by a computing device or processor, cause the computing device or processor to perform the method of any of claims 1-12.

Images