Image processing method and device

Abstract

The invention discloses an image processing method and device, and belongs to the technical field of image processing. The method comprises the following steps: acquiring a to-be-processed image; determining a target object in the to-be-processed image according to the to-be-processed image and a rule base; and executing the target operation on the target object in the to-be-processed image. Wherein the target object is an object of a to-be-executed target operation in the to-be-processed image; the rule base is obtained based on image extraction in the data set, and for a first rule obtained based on first image extraction in the data set in the rule base, the first rule is used for indicating an object marked with a target label in the first image under the condition of objects included in the first image and the relation between the objects. According to the method, the rules in the rule base are interpretable rules, so that when the target object of the to-be-executed target operation in the to-be-processed image is detected through the method, and the corresponding operation is executed on the target object to complete image processing, high interpretability is achieved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] With the development of the information age and the popularity of smartphones, people are accustomed to sharing and communicating on social platforms through the Internet. According to incomplete statistics, hundreds of millions of photos and hundreds of millions of hours of video viewing are generated every day on some social applications. In addition, personal pictures are also perceived and collected by the outside world in many other different ways, such as mobile application access, cloud storage, etc.

[0003] However, although sharing and disseminating pictures on social applications through the Internet can bring many conveniences to people, there is a risk of leakage of a large amount of personal information contained in the shared and disseminated pictures, that is, there is a risk of user privacy leakage through sharing and disseminating pictures. For example, the background in the picture will reveal the user's geographical location and whereabouts, and the people appearing in the picture will reveal the user's social relationships, etc. Therefore, it is necessary to perform privacy protection processing on the pictures posted and disseminated by users on the Internet.

[0004] In the related art, deep learning models or support vector machine algorithm models are used to classify images into images containing private objects and images not containing private objects, and images containing private objects are desensitized to achieve privacy protection of images. However, deep learning models and support vector machine algorithm models are dependent on training sets and lack interpretability, so they are not of usable value. Therefore, there is an urgent need for an image processing method with strong interpretability to achieve privacy protection of images. Summary of the invention

[0005] The present application provides an image processing method and device. The rules in the rule library provided in the method are interpretable rules. Therefore, when the method is used to detect the target object for the target operation to be performed in the image to be processed, and to perform the corresponding operation on the target object to complete the image processing, it has strong interpretability.

[0006] The technical solutions provided by this application are as follows:

[0007] In a first aspect, the present application provides an image processing method, the method comprising: obtaining an image to be processed; determining a target object in the image to be processed according to the image to be processed and a rule base; and performing a target operation on the target object in the image to be processed. The target object is the object on which the target operation is to be performed in the image to be processed. The rule base is extracted based on images in a data set. For a first rule in the rule base, the first rule is any rule in the rule base, and the first rule is extracted based on a first image in the data set. The first rule is used to indicate an object in the first image that is marked with a target label under the condition that the objects included in the first image and the relationship between the objects. The data set includes multiple images, each of the multiple images contains an object marked with a target label, and the object marked with the target label is the object on which the target operation is to be performed.

[0008] Through the present application, the purpose of detecting the target object of the target operation to be performed in the image to be processed based on the rule base can be achieved, and after the target operation is performed on the target object, the corresponding processing of the image to be processed can be achieved. For example, when the target operation is a desensitization operation, it means that the target object is a privacy object in the image to be processed, and thus by performing a desensitization operation on the privacy object in the image to be processed, the privacy protection processing of the image to be processed can be achieved. Since the rules in the rule base provided by the present application are interpretable rules, when the target object of the target operation to be performed in the image to be processed is detected by the present application, and the corresponding operation is performed on the target object to complete the image processing, it has strong interpretability.

[0009] In a possible design, when the target label is a label indicating privacy, the target operation is a desensitizing operation.

[0010] Through this possible design, it is possible to detect and process the privacy objects in the image, thereby achieving the privacy protection of the image.

[0011] In another possible design manner, when the target tag is a tag that the user particularly pays attention to, the target operation is a highlighting operation.

[0012] Through this possible design, it is possible to detect and process the object that the user is interested in in the image, thereby improving the user experience.

[0013] In another possible design, the rules in the rule base are represented by directed paths in the first knowledge graph, the first knowledge graph is constructed based on the multiple images, the directed paths in the first knowledge graph include nodes and directed edges, the nodes of the directed paths represent the objects contained in the multiple images, the directed edges of the directed paths are used to describe the relationship between the objects contained in the multiple images, the last node in the direction indicated by the directed edge in the directed path is a label node, the label node is used to record the object represented by at least one node on the directed path, and the objects represented by at least one node are all objects marked with target labels in the multiple images.

[0014] Through this possible design, by expressing and integrating the common rules contained in the image collection in the form of a knowledge graph, the rule logic is displayed in a visual way, and any node and edge in the knowledge graph has real semantics that can be understood by humans. Therefore, when the image processing method provided by the present application is used to detect the target object to be operated on, it has strong interpretability and high detection accuracy.

[0015] In another possible design, the above-mentioned determining the target object in the image to be processed based on the image to be processed and the rule base includes: constructing a second knowledge graph based on the image to be processed; querying the target path including nodes and directed edges in the second knowledge graph in the first knowledge graph; and determining the object recorded by the label node of the target path as the target object. The nodes in the second knowledge graph represent the objects contained in the image to be processed, the directed edges in the second knowledge graph are used to describe the relationship between the objects contained in the image to be processed, and the target path is one of multiple directed paths.

[0016] Through this possible design, since the present application uses the graph path matching method to infer the target object of the target operation to be performed in the image to be processed, and logical reasoning is used when detecting the target object of the target operation to be performed in the image to be processed based on the knowledge graph, the target object of the target operation to be performed in the image to be processed based on the first knowledge graph is highly interpretable and has high detection accuracy. In addition, since the reasoning of the graph structure only includes logical implications (i.e. path matching) calculations, that is, the reasoning based on the graph structure is to infer the target object through logical operations, without the need for large-scale neural network calculations as in the related art, the complexity of reasoning is strictly limited in theory, saving computing resources, so that the reasoning speed is faster than the deep learning model.

[0017] In another possible design, after determining the target object in the image to be processed according to the image to be processed and the rule base, the method further includes: updating the rule base according to the image to be processed in response to a first instruction fed back by the user, wherein the first instruction is used to instruct to perform a target operation on other objects in the image to be processed that are different from the target object.

[0018] Through this possible design, the purpose of automatically detecting and updating the rule base based on user feedback is achieved.

[0019] In another possible design, when the first knowledge graph is a knowledge graph constructed based on multiple images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the above-mentioned updating of the rule base according to the image to be processed includes: deleting the target path in the first knowledge graph; adding a directed path in the first knowledge graph according to the image to be processed and the first indication. Among them, the nodes on the target path used to represent the object and the directed edges used to describe the relationship between the objects exist in the second knowledge graph, and the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed based on the image to be processed; the label node on the newly added directed path is used to record other objects.

[0020] Through this possible design, since the update of user personalized data in the knowledge graph only involves the addition and deletion of paths in the graph, it can not only efficiently meet the needs of rule personalization and dynamic changes, but also compared with the related art of fine-tuning the deep learning model for detecting the target object to be performed in the target operation to be performed in the image to be processed according to the user's personalized data, the efficiency of updating the knowledge graph according to the user's personalized data in the present application is higher.

[0021] In another possible design, the method is executed by a terminal device, and the terminal device is deployed with a rule library. Then, the obtaining of the image to be processed includes: obtaining the image to be processed stored locally on the terminal device or sent to the terminal by other devices.

[0022] In another possible design, the method is executed by an application server, and the application server is deployed with a rule library. Then, the step of obtaining the image to be processed includes: receiving the image to be processed uploaded by a terminal device.

[0023] Through these two possible designs, the present application can be flexibly applied to different scenarios.

[0024] In another possible design, before determining the target object in the image to be processed based on the image to be processed and the rule base, the method also includes: constructing multiple knowledge graphs based on the multiple images; adding a label node at the end of the directed path containing preset nodes in the multiple knowledge graphs to obtain a first knowledge graph; wherein the number of the multiple knowledge graphs is the same as the number of images in the data set; the multiple knowledge graphs include multiple directed paths containing preset nodes, wherein the nodes on the directed paths represent the objects contained in the multiple images, the directed edges in the directed paths are used to describe the relationship between the objects contained in the multiple images, and the preset nodes contained in the directed paths are nodes located on the directed paths and used to represent the objects marked with target labels in the multiple images; the tail end of the directed path is the end of the direction indicated by the directed edge of the directed path, and the label node added at the tail end of the directed path is used to record the objects represented by the preset nodes on the directed path.

[0025] Through this possible design approach, it is possible to construct a first knowledge graph based on the data set.

[0026] In another possible design, after adding a label node to the end of the directed path containing the preset node in the above-mentioned multiple knowledge graphs, the above-mentioned method further includes: merging the directed paths with the label node added by the nodes representing the same object and the same type of object on the directed paths with the label node added, to obtain the first knowledge graph; recording the directed paths to which the merged nodes and merged directed edges in the first knowledge graph belong. Among them, the same type of object refers to the object of the same type; the merged node is a node formed by merging multiple nodes belonging to different directed paths, and the merged directed edge is a directed edge formed by merging multiple directed edges belonging to different directed paths.

[0027] Through this possible design, the first knowledge graph can be simplified.

[0028] In another possible design, the rule library includes at least one of the following first to sixth rules: First Rule: Under the condition that the image includes an outdoor scene containing a person, the person is wearing a mask, and the person includes a face, the object marked with a privacy label in the image is a face; Second Rule: Under the condition that the image includes a bedroom / dormitory scene containing a person, and the person includes a face, the object marked with a privacy label in the image is a bedroom / dormitory and a face; Third Rule: Under the condition that the image includes an office scene containing a screen, and the attribute of the screen is bright, the object marked with a privacy label in the image is the screen; Fourth Rule: Under the condition that the image includes an outdoor / garage scene containing a car, and the car includes a license plate, the object marked with a privacy label in the image is the license plate; Fifth Rule: Under the condition that the image is a chat screenshot and the chat screenshot includes a left avatar / left nickname, the object marked with a privacy label in the image is a left avatar / left nickname; Sixth Rule: Under the condition that the image includes a card containing an identity document (ID) number / name / age / address, the object marked with a privacy label in the image is ID / name / age / address.

[0029] Through this possible design, typical privacy rules are given.

[0030] In a second aspect, the present application provides an image processing device, which includes one or more units / modules, and the one or more units / modules are used to execute the method provided in the first aspect and any possible design method in the first aspect.

[0031] In a possible design, the above-mentioned device includes: an acquisition unit, which is used to acquire an image to be processed. A determination unit, which is used to determine a target object in the image to be processed based on the image to be processed and a rule base. A processing unit, which is used to perform a target operation on the target object in the image to be processed. The target object is an object on which the target operation is to be performed in the image to be processed. The rule base is extracted based on images in a data set. For a first rule in the rule base, the first rule is any rule in the rule base, and the first rule is extracted based on a first image in the data set. The first rule is used to indicate an object in the first image that is marked with a target label under the condition that the objects included in the first image and the relationship between the objects. The data set includes multiple images, each of the multiple images contains an object marked with a target label, and the object marked with the target label is an object for performing the target operation.

[0032] In another possible design, when the target label is a label indicating privacy, the target operation is a desensitizing operation.

[0033] In yet another possible design, when the target tag is a tag that the user is particularly concerned about, the target operation is a highlighting operation.

[0034] In another possible design, the rules in the rule base are represented by directed paths in the first knowledge graph, and the first knowledge graph is constructed based on multiple images. The directed path in the first knowledge graph includes nodes and directed edges, the nodes of the directed path represent objects contained in the multiple images, the directed edges of the directed path are used to describe the relationship between the objects contained in the multiple images, the last node in the direction indicated by the directed edge in the directed path is a label node, and the label node is used to record the object represented by at least one node on the directed path, and the object represented by at least one node is an object marked with a target label in the multiple images.

[0035] In another possible design, the processing unit is further used to construct a second knowledge graph based on the image to be processed, and to query the first knowledge graph for a target path including nodes and directed edges in the second knowledge graph. The determination unit is specifically used to determine the object recorded by the label node of the target path as the target object. The nodes in the second knowledge graph represent the objects contained in the image to be processed, the directed edges in the second knowledge graph are used to describe the relationship between the objects contained in the image to be processed, and the target path is one of the multiple directed paths.

[0036] In another possible design, the processing unit is further configured to update the rule base according to the image to be processed in response to a first indication fed back by the user after determining the target object in the image to be processed according to the image to be processed and the rule base, wherein the first indication is used to instruct to perform a target operation on other objects in the image to be processed that are different from the target object.

[0037] In another possible design, when the first knowledge graph is a knowledge graph constructed based on multiple images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the processing unit is specifically used to delete the target path in the first knowledge graph, and, based on the image to be processed and the first indication, add a directed path to the first knowledge graph. Among them, the nodes on the target path used to represent the object and the directed edges used to describe the relationship between the objects exist in the second knowledge graph, and the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed based on the image to be processed. The label nodes on the newly added directed path are used to record other objects.

[0038] In another possible design, the above-mentioned apparatus is applied to a terminal device, and the terminal device is deployed with a rule library, and the acquisition unit is specifically used to acquire the to-be-processed image stored locally in the terminal device or sent to the terminal by other devices.

[0039] In another possible design, the above-mentioned device is applied to an application server, and the application server is deployed with a rule library, and the acquisition unit is specifically used to receive the image to be processed uploaded by the terminal device.

[0040] In another possible design, the processing unit is also used to construct multiple knowledge graphs based on multiple images before determining the target object in the image to be processed based on the image to be processed and the rule base, and to add a label node to the tail end of the directed path containing preset nodes in the multiple knowledge graphs to obtain a first knowledge graph. The number of the multiple knowledge graphs is the same as the number of images in the data set. The multiple knowledge graphs include multiple directed paths containing preset nodes, wherein the nodes on the directed paths represent the objects contained in the multiple images, the directed edges in the directed paths are used to describe the relationship between the objects contained in the multiple images, the preset nodes contained in the directed paths are nodes located on the directed paths and used to represent the objects marked with target labels in the multiple images, the tail end of the directed path is the end of the end point of the direction indicated by the directed edge of the directed path, and the label node added at the tail end of the directed path is used to record the objects represented by the preset nodes on the directed path.

[0041] In another possible design, the processing unit is further used to add a label node to the tail end of a directed path containing a preset node in multiple knowledge graphs, and then merge the directed paths with the label node added by the nodes representing the same object and the same type of object on the directed paths with the label node added to obtain a first knowledge graph; and record the merged nodes and the directed paths to which the merged directed edges in the first knowledge graph belong. Among them, the same type of object refers to objects of the same type. The merged node is a node that is formed by merging multiple nodes belonging to different directed paths, and the merged directed edge is a directed edge that is formed by merging multiple directed edges belonging to different directed paths.

[0042] In another possible design, the rule library includes at least one of the following first to sixth rules: First Rule: When the image includes an outdoor scene containing a person, the person is wearing a mask, and the person includes a face, the object marked with a privacy label in the image is a face; Second Rule: When the image includes a bedroom / dormitory scene containing a person, and the person includes a face, the object marked with a privacy label in the image is a bedroom / dormitory and a face; Third Rule: When the image includes an office scene containing a screen, and the attribute of the screen is bright, the object marked with a privacy label in the image is the screen; Fourth Rule: When the image includes an outdoor / garage scene containing a car, and the car includes a license plate, the object marked with a privacy label in the image is the license plate; Fifth Rule: When the image is a chat screenshot, and the chat screenshot includes a left avatar / left nickname, the object marked with a privacy label in the image is a left avatar / left nickname; Sixth Rule: When the image includes a card containing an ID card number ID / name / age / address, the object marked with a privacy label in the image is ID / name / age / address.

[0043] It should be understood that the description of possible technical solutions and beneficial effects executed by the above-mentioned various functional modules / units can refer to the solutions provided by the above-mentioned first aspect and any possible design method in the first aspect, and will not be repeated here.

[0044] In a third aspect, the present application provides an image processing device. The device includes: a memory, a communication interface, and one or more processors. The one or more processors receive or send data through the communication interface, and the one or more processors are configured to read program instructions stored in the memory to execute the method provided in the first aspect and any possible design method in the first aspect.

[0045] The image processing device described in the second aspect and the third aspect may be, for example, any computing device / apparatus with computing and processing capabilities, such as mobile phones, tablets, vehicles, laptops, general-purpose computers, artificial intelligence (AI) terminals, wearable devices, or servers such as application servers and storage servers. Or it may be a component in the aforementioned computing device / apparatus, such as a single board, or it may be a chip for implementing the method described in any of the above aspects and part or all of the operations in any possible design.

[0046] In a fourth aspect, the present application provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by a computing device or a processor, the computing device or the processor executes the method provided in the first aspect and any possible design method in the first aspect.

[0047] In a fifth aspect, the present application provides a computer program product comprising instructions, which, when executed by a computing device or a processor, causes the computing device or the processor to execute the method provided in the first aspect and any possible design method in the first aspect.

[0048] In a sixth aspect, the present application provides a chip, which includes a processor for running program instructions or codes, and the chip or a device including the chip can be used to execute the method provided in the first aspect and any possible design method in the first aspect. Exemplarily, the chip also includes: an input interface, an output interface, and a memory. Among them, the input interface, output interface, processor, and memory of the chip are connected through the internal connection path of the chip, the memory in the chip is used to store the program instructions or codes run by the processor, and the input interface and output interface of the chip are used for the connection and communication between the chip and other chips or devices.

[0049] It can be understood that any of the image processing devices, computer-readable storage media, computer program products or chips provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods and will not be repeated here.

[0050] In this application, the names of the above-mentioned image processing devices, etc. do not limit the devices or functional modules themselves. In actual implementation, these devices or functional modules may appear with other names. As long as the functions of each device or functional module are similar to those of this application, they all fall within the scope of protection of this application. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram showing a system framework for implementing image privacy protection;

[0052] Figure 2 This is a schematic diagram of an application scenario of the method provided in the embodiment of the present application;

[0053] Figure 3 This is another schematic diagram of an application scenario of the method provided in the embodiment of the present application;

[0054] Figure 4 This is a schematic diagram of a method for building a rule base provided in an embodiment of the present application;

[0055] Figure 5 It is a schematic diagram of a method flow for constructing a first knowledge graph provided in an embodiment of the present application;

[0056] Figure 6 is a schematic diagram of a knowledge graph provided in an embodiment of the present application;

[0057] Figure 7 is a schematic diagram of another knowledge graph provided in an embodiment of the present application;

[0058] Figure 8 is a schematic diagram of another knowledge graph provided in an embodiment of the present application;

[0059] Fig. 9 is a schematic diagram of another knowledge graph provided in an embodiment of the present application;

[0060] Fig.10 is a schematic diagram of a first knowledge graph provided in an embodiment of the present application;

[0061] Fig.11 It is a flowchart of another method for constructing a first knowledge graph provided in an embodiment of the present application;

[0062] Fig.12 is a schematic diagram of another first knowledge graph provided in an embodiment of the present application;

[0063] Fig.13 is a schematic diagram of another first knowledge graph provided in an embodiment of the present application;

[0064] Fig.14 It is a flowchart of an image processing method provided by an embodiment of the present application;

[0065] Fig.15 It is a schematic diagram of a process for determining a target object in an image to be processed according to the image to be processed and a first knowledge graph provided by an embodiment of the present application;

[0066] Fig.16 It is a schematic diagram of a process of searching a target path including nodes and directed edges in a second knowledge graph in a first knowledge graph provided by an embodiment of the present application;

[0067] Fig.17 It is a schematic diagram of a process of deleting a target path in a first knowledge graph provided by an embodiment of the present application;

[0068] Fig.18 It is a schematic diagram of a process of adding a directed path in a first knowledge graph according to an image to be processed and a first indication provided by an embodiment of the present application;

[0069] Fig.19 is a schematic diagram of another image processing method provided in an embodiment of the present application;

[0070] Fig. 20 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0071] Fig.21 is a structural schematic diagram of another image processing device provided in an embodiment of the present application;

[0072] Fig. 22 It is a structural diagram of a computing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0074] To facilitate understanding, the technology and background involved in the embodiments of the present application are first explained below.

[0075] 1) Graph structure expression

[0076] Graph structure expression refers to the use of directed graphs to express image content. For example, the information in the image is used to form the nodes of the directed graph, and the positional relationship and / or semantic relationship between the elements / objects (such as background, objects, etc.) in the image is used to form the edges of the directed graph. Among them, the directed graph used to express image content is also called the knowledge graph used to describe image content.

[0077] In the embodiment of the present application, the objects in the image include but are not limited to items, people, etc.

[0078] 2) Desensitization

[0079] Desensitizing information / data means deforming certain sensitive information / data according to certain rules, thereby achieving reliable protection of sensitive privacy data.

[0080] Common methods of data desensitization include but are not limited to: a. Replacement: Replace sensitive data with other values, such as replacing sensitive data with specific symbols or random numbers; b. Encryption: Encrypt sensitive data so that only authorized personnel can decrypt it; c. Shielding: Hide part of the sensitive data, such as displaying only some characters; d. Hashing: Convert sensitive data to hash values, and prohibit reverse restoration; e. Data deformation: Change the format or structure of data; f. Generalization: Replace specific sensitive data with more general information; g. Date offset: Offset date-related sensitive data; h. Field truncation: Only retain part of the sensitive data; i. Noise addition: Add random noise to sensitive data.

[0081] Data desensitization technology has been widely used, including the safe use of desensitized real data sets in development, testing, and outsourcing environments to achieve data availability while protecting sensitive privacy data.

[0082] In the field of image processing, methods for desensitizing an image in a certain area of ​​a picture include but are not limited to: a. blurring the image area that needs to be desensitized; b. mosaicing the image area that needs to be desensitized; c. setting the pixel value of the image area that needs to be desensitized to 0 or 255; and so on.

[0083] In the related art, in order to achieve image privacy protection, reference Figure 1 , Figure 1 A schematic diagram of a system framework for implementing image privacy protection is shown. Figure 1As shown, the system includes an image privacy detection module and a desensitization module. The image privacy detection module is used to detect the privacy objects in the image to be published on the end side, and the desensitization module is used to desensitize the image of the privacy object in the detected image. Then, the end side publishes the desensitized image to the network side, such as to a social network or the cloud. Among them, the objects in the image include but are not limited to objects, people, places, etc. in the image. The privacy object is an object in the image with user privacy information, such as a license plate in the image.

[0084] The process of detecting the privacy object in the image to be published on the end side by the image privacy detection module generally includes: firstly extracting the features of the object from the image to be published through the visual feature extraction network to obtain the probability that the object corresponding to the feature has a privacy label, then weighting the probability that the object corresponding to the feature has a privacy label through the attention mechanism, and determining whether the object corresponding to the feature has a privacy label or not according to the weighted probability, and then determining the object with the privacy label as a privacy object, thereby achieving the purpose of detecting the privacy object in the image. The visual feature extraction network includes but is not limited to the visual geometry group network (VGG) and the residual network (RESNET).

[0085] It can be seen that Figure 1 The system shown uses the binary classification function of the deep learning model to achieve the purpose of detecting private objects in images. However, in this method, the data set used to train the deep learning model lacks interpretability, and the privacy rules of the objects in the images included in the data set are hidden in the model, so it is not intuitive. In addition, deep learning models usually require a large number of data sets to be trained, so the computational cost is high. In addition, a fixed single deep learning model cannot meet the user's personalized privacy rule requirements. When the privacy rules contained in the deep learning model do not match the user's privacy requirements, the deep learning model needs to be trained again based on the user data, so the cost of personalization is high.

[0086] Based on this, an embodiment of the present application provides an image processing method, which can detect the target object of the target operation to be performed (or understood to be required to be performed) in the image to be processed through a rule base extracted from the data set image, and then perform the target operation on the target object, so as to complete the processing of the image to be processed. Among them, the rules in the rule base are used to indicate the object marked with a target label in the image under the condition that the image includes objects and the relationship between objects. The object marked with a target label in the image is the object for performing the target operation. Therefore, when the target object of the target operation to be performed in the image to be processed is detected by the rules in the rule base, when the objects in the image to be processed and the relationship between the objects meet the relationship between the objects indicated by a certain rule in the rule base, the object marked with the target label indicated by the rule is determined as the target object of the target operation to be performed in the image to be processed. It can be seen that the method has strong interpretability, and then the method can improve the accuracy of detecting the target object in the image to be processed.

[0087] The objects in the image may be understood as elements in the image, including but not limited to objects, texts, places, etc. contained in the image, but not limited thereto.

[0088] Optionally, when the target label is a label indicating privacy (referred to as a privacy label), the target operation is a desensitization operation. In this case, after executing the method provided in the embodiment of the present application, the privacy object in the image to be processed can be detected, and the privacy object can be desensitized, thereby achieving privacy protection of the image to be processed, and the method has strong interpretability and can improve the accuracy of detecting the target object in the image to be processed.

[0089] Optionally, when the target tag is a tag that indicates special attention of the user (recorded as a special attention tag), the target operation is a highlighting operation. In this case, after executing the method provided in the embodiment of the present application, it is possible to determine the object that the user is particularly concerned about in the image to be processed, and perform highlighting on the object, thereby meeting the user's personalized needs for the image to be processed, improving the user experience, and the method has strong interpretability and can improve the accuracy of detecting the target object in the image to be processed.

[0090] refer to Figure 2 , Figure 2 A schematic diagram of an application scenario of the method provided in the embodiment of the present application is shown. Figure 2 As shown, the application scenario includes a computing device 20. The computing device 20 includes an image processing device. The image processing device is used to obtain an image to be processed from a memory of the computing device 20, or receive an image to be processed from other devices, and execute the method provided in the embodiment of the present application on the image to be processed to complete the processing of the image to be processed, for example, to complete the processing of privacy protection of the image to be processed.

[0091] In some examples, the computing device 20 is implemented as a terminal device, which includes but is not limited to a mobile phone, a tablet, a car-mounted device, a laptop computer, a general-purpose computer, an AI terminal, a wearable device, etc.

[0092] In some other examples, the computing device 20 is implemented as a server, etc. The server is, for example, an application server, a storage server, etc. The application server is, for example, a social application server, etc., but is not limited thereto.

[0093] refer to Figure 3 , Figure 3 FIG. 2 is a schematic diagram showing another application scenario of the method provided in the embodiment of the present application. Figure 3 As shown, the application scenario includes an application server 30 and a terminal device 31. The terminal device 31 communicates with the application server 30 through a network (such as the Internet). When the application server 30 performs the method provided in the embodiment of the present application on the image uploaded / published by the terminal device, the image uploaded / published by the terminal device can be processed, for example, the privacy protection of the image uploaded / published by the terminal device can be processed.

[0094] It should be understood that the above content is an illustrative description of the application scenarios of the method provided in the embodiments of the present application, and does not constitute a limitation on the application scenarios of the method. A person of ordinary skill in the art can know that as business needs change, the application scenarios can be adjusted according to application needs, and the embodiments of the present application do not list them one by one.

[0095] The embodiment of the present application also provides an image processing device, which is used to execute the image processing method provided in the embodiment of the present application. Exemplarily, the device can be implemented as a computing device or a functional module in a computing device, and the computing device is, for example, a mobile phone, a tablet, a car, a laptop, a general-purpose computer, an artificial intelligence (AI) terminal, a wearable device, or a computing device is, for example, an application server, a storage server, or other server. This is not limited.

[0096] The implementation process of the image processing method provided in the embodiment of the present application is described below.

[0097] Before executing the image processing method provided in the embodiment of the present application, it is necessary to construct a rule base based on the data set in advance so that the image processing device can detect the target object to be executed in the image to be processed according to the rule base. The rule base is obtained based on the image extraction in the data set. The process of constructing the rule base described in the embodiment of the present application is first described in detail below.

[0098] refer to Figure 4 , Figure 4The following is a flow chart of a method for constructing a rule base provided by an embodiment of the present application. The method is executed by any computing device (or apparatus) with computing processing capabilities, and the present application embodiment does not limit this. In one example, the method can be executed by the image processing apparatus described above. Figure 4 As shown, the method includes the following steps.

[0099] Step 101: A computing device obtains a data set.

[0100] The data set includes a plurality of images. In the embodiment of the present application, each image in the data set includes an object marked with a target label, and the object marked with the target label is an object for performing a target operation.

[0101] Optionally, the embodiment of the present application does not specifically limit the number of objects marked with target labels in the image. That is, in the embodiment of the present application, for any image in the data set, there may be only one object marked with a target label in the image, or there may be multiple objects.

[0102] In a possible implementation, the computing device downloads multiple images containing objects labeled with target labels from the network to obtain the above dataset. Optionally, a third-party device downloads multiple images containing objects labeled with target labels from the network to obtain the above dataset, and then the third-party device sends the dataset to the computing device, or the third-party device copies the downloaded dataset to the computing device through any storage device, which is not limited. In response, the computing device obtains the dataset.

[0103] In one example, for an image downloaded from the Internet that contains an object marked with a target tag, the object marked with the target tag in the image is the object on which the target operation is performed in the image. Taking the desensitization operation in which the target tag is a privacy tag and the target operation is to blur the image as an example, for any image downloaded from the Internet, when the image contains a car, and the car displayed in the image contains a blurred license plate. Among them, the blurred license plate is a specific implementation form of the desensitization operation performed on the license plate image. Since the desensitization operation is an operation performed on an object with a privacy tag, the blurred license plate in the image is the object marked with the privacy tag in the image. It can be seen that the license plate in the image is the user's private information.

[0104] In another possible implementation, the computing device acquires multiple images in advance, and annotates target labels for objects in the images on which target operations are expected to be performed according to user requirements, thereby obtaining the above-mentioned data set. The multiple images may be downloaded from the Internet, or taken by the user, etc., and there is no limitation on this.

[0105] For example, taking the case where the target label is a privacy label and the target operation is a desensitization operation, for any image acquired by a computing device, when the image contains an office scene, and the office scene displayed in the image contains a lit display screen, that is, there is content displayed on the display screen contained in the image. Since the content displayed on the display screen in the office scene is internal data of the enterprise, when the content displayed on the display screen is not privacy-protected, that is, it is published on a social network or uploaded to the cloud, it will cause corporate secrets to be leaked, that is, the content displayed on the display screen in the office scene is the private information of the enterprise, therefore, the computing device annotates (such as manually annotates) a privacy label for the lit display screen contained in the image.

[0106] Step 102: The computing device extracts rules from the images in the data set to obtain a rule base.

[0107] The computing device may first detect and identify objects in the image through a target detection algorithm, and identify the relationship between the objects in the image. Optionally, the computing device also identifies the state and / or attribute of the objects in the image. The relationship between the objects may be a logical relationship and / or a spatial position relationship, which is not limited.

[0108] Optionally, for each image in the data set, the computing device may first pre-classify the image according to the object to be identified in the image, and then use a corresponding target detection algorithm to perform detection and identification on the pre-classified image.

[0109] For example, when an object is included in an image, the computing device divides the image into a first category of objects that need to be identified in the image. Then the computing device uses a target detection algorithm that can detect objects in the image to detect and identify objects in the first category of images, and uses a related algorithm to detect the relationship between each object in the image and the state and / or attribute of each object. Exemplarily, the computing device uses a deep learning model in the relevant technology (such as a faster region with convolutional neural network feature (faster-RCNN) model) to detect and identify objects in the first category of images, and uses a deep learning model in the relevant technology to identify the relationship between objects in the image. It should be understood that when the relationship between objects in the image is identified by a deep learning model, a confidence threshold can be set. When the confidence of the relationship between objects detected by the model exceeds the confidence threshold, the model outputs the relationship between the detected objects.

[0110] For another example, when the image contains text, the computing device divides the image into a second category in which text in the image needs to be recognized. The computing device then uses an object detection algorithm capable of detecting text in the image to detect and recognize text in the second category image, and uses a related algorithm to recognize the relationship between the text in the image and the state and / or attribute of the text. In one example, the relationship between the text in the image can be a positional relationship.

[0111] For another example, when an image contains both objects and text, the computing device divides the image into a third category that requires first identifying the objects in the image and then identifying the text in the image. In this case, the computing device uses a target detection algorithm that can detect objects in the image to detect and identify the objects in the third category image, and then uses a target detection algorithm that can detect text in the image to detect and identify the text in the third category image, and uses a related algorithm to identify the relationship between each object in the image and the state and / or attribute of each object. Among them, the relationship between objects is, for example, the position relationship between objects and text in the image, the relationship between objects and objects, the relationship between text and text, and so on. Alternatively, when an image contains both objects and text, the computing device divides the image into a fourth category that requires first identifying the text in the image and then identifying the objects in the image. In this case, the computing device uses a target detection algorithm that can detect text in the image to detect and identify the text in the third category image, and then uses a target detection algorithm that can detect objects in the image to detect and identify the objects in the third category image, and uses a related algorithm to identify the relationship between each object in the image and the state and / or attribute of each object.

[0112] Further, the computing device records the objects detected from each image of the data set, the relationships between the objects, the states and / or properties of the objects, and the objects annotated with the target tags as rules extracted from the images.

[0113] In one example, when the objects detected by the computing device from the first image of a data set include outdoor scenes, people, masks, and human faces, and the relationship between the detected objects includes: the outdoor scene contains people, people wear masks, and people contain human faces, and the object marked with a privacy label in the first image is a human face, the computing device records a first rule extracted from the first image: under the conditions that the first image includes an outdoor scene containing people, people wear masks, and people contain human faces, the object marked with a privacy label in the first image (i.e., a privacy object) is a human face.

[0114] In some embodiments, the first rule can also be expressed as: <outdoor scene, including, person> ∧ <person, wearing, mask> ∧ <person, including, face> → privacy object: face. Among them, "∧" is a conjunction symbol, which is a concept in logic, indicating that when two or more propositions are true at the same time, the entire compound proposition is true. "→" represents logical implication, which is a logical relationship, specifically indicating a conditional relationship. For example, when A logically implies B, it is recorded as A→B, and specifically indicates that when condition A is true, B is also true.

[0115] In another example, when the objects detected by the computing device from the second image of the data set include bedroom / dormitory scenes, people, and faces, and the relationship between the detected objects includes: bedroom / dormitory scenes contain people, people contain faces, and objects marked with privacy tags in the second image include bedroom / dormitory scenes and faces, the computing device records a second rule extracted from the second image: under the condition that the second image includes bedroom / dormitory scenes containing people, and people contain faces, the objects marked with privacy tags in the second image are bedrooms / dormitory and faces. In some embodiments, the second rule can also be expressed as: <bedroom / dormitory scene, contains, people> ∧ <people, contains, faces> → privacy objects: bedroom / dormitory scene, faces.

[0116] In another example, when the objects detected by the computing device from the third image of the data set include an office scene and a screen, and the relationship between the detected objects includes: the office scene includes a screen, and the state of the detected object includes: the state of the screen is bright (for example, the state of the screen is determined to be bright by the text / image displayed on the screen, or the state of the screen is determined to be bright by the pixel value of the area where the screen is located in the image), and the object marked with a privacy label in the third image includes a screen, then the computing device records the third rule extracted from the third image: under the condition that the third image includes an office scene including a screen, and the state of the screen is bright, the object marked with a privacy label in the third image is a screen. In some embodiments, the third rule can also be expressed as: <office, includes, screen> ∧ <screen, state, bright> → privacy object: screen.

[0117] In another example, when the objects detected by the computing device from the fourth image of the data set include an outdoor / garage scene, a car, and a license plate, and the relationship between the detected objects includes: the outdoor / garage scene contains a car, the car contains a license plate, and the object marked with a privacy label in the fourth image is a license plate, the computing device records a fourth rule extracted from the fourth image: under the condition that the fourth image includes an outdoor / garage scene containing a car, and the car contains a license plate, the object marked with a privacy label in the fourth image is a license plate. In some embodiments, the fourth rule can also be expressed as: <outdoor / garage scene, contains, car> ∧ <car, contains, license plate> → privacy object: license plate.

[0118] In another example, when the computing device detects that the object in the fifth image of the data set includes a left avatar / left nickname, and detects that the fifth image is a chat screenshot, and detects that the relationship between the fifth image and the object included in the fifth image includes: the chat screenshot contains a left avatar / left nickname, and the object marked with a privacy label in the fifth image is a left avatar / left nickname, then the computing device records the fifth rule extracted from the fifth image: under the condition that the fifth image is a chat screenshot and the chat screenshot contains a left avatar / left nickname, the object marked with a privacy label in the fifth image is a left avatar / left nickname. In some embodiments, the fifth rule can also be expressed as: <chat screenshot, containing, left avatar / left nickname, etc.> → privacy object: left avatar / left nickname, etc.

[0119] In another example, when the objects detected by the computing device from the sixth image of the data set include a card, an identity document (ID) / name / age / address, and the relationship between the detected objects includes: the card contains ID / name / age / address, and the object marked with a privacy tag in the sixth image is ID / name / age / address, then the computing device records the sixth rule extracted from the sixth image: under the condition that the sixth image includes the card containing the identity document ID / name / age / address, the object marked with a privacy tag in the sixth image is ID / name / age / address. In some embodiments, the sixth rule can also be expressed as: <card, contains, ID / name / age / address> → privacy object: ID / name / age / address, etc.

[0120] In this way, the rule set composed of the rules extracted by the computing device based on each image in the data set (such as the first rule to the sixth rule mentioned above) constitutes the rule base described in the embodiment of the present application. It can be seen that for any rule in the rule base, such as the first rule, the first rule is extracted based on the first image in the data set, and the first rule is used to indicate the objects in the first image that are marked with the target label under the condition that the objects included in the first image and the relationship between the objects.

[0121] In some embodiments, the above rule base can be represented as a first knowledge graph (or a rule knowledge graph), and the rules in the rule base can be represented by directed paths in the first knowledge graph. Among them, the first knowledge graph is a directed graph that expresses the image content through a graph structure. The first knowledge graph is constructed based on multiple images in the data set. The directed path in the first knowledge graph includes nodes and directed edges. The nodes in the directed path represent the objects contained in the images in the data set, and the directed edges of the directed path are used to describe the relationship between the objects contained in the images in the data set. In addition, the last node in the direction indicated by the directed edge in the directed path is a label node, and the label node is used to record the object represented by at least one node on the directed path containing the label node. The objects represented by the at least one node are all objects marked with the target label in the image of the data set.

[0122] In this case, refer to Figure 5 , Figure 5 A schematic flow chart of a method for constructing a first knowledge graph provided by an embodiment of the present application is shown. The method is executed by any computing device (or apparatus) with computing processing capabilities, and the embodiment of the present application does not limit this. In one example, the method can be executed by the image processing apparatus described above. Figure 5 As shown, after executing step 101, the method executes the following steps 103 to 105.

[0123] Step 103: The computing device detects objects in each image in the data set and identifies the relationships between the objects.

[0124] For a detailed description of how the computing device detects objects in the images in the data set and identifies the relationships between the objects, please refer to the relevant description in step 102 and will not be repeated here.

[0125] In one example, the set of objects detected by the computing device from any image in the data set (such as image I, where I is a positive integer) can be expressed as: I ={o1,o2,…,o n}, and any two objects in the image (such as o i ,o j ) can be recorded as r i,j . Where n is a positive integer greater than 1, and I, i, and j are integers between [1, n]. i,j It can be a logical relationship or a spatial position relationship. i ,o i and r j,j The triple T i,j = <o i ,r i,j ,o j >.

[0126] Step 104: The computing device constructs multiple knowledge graphs based on the multiple images included in the data set.

[0127] The computing device can construct a knowledge graph corresponding to each image in the data set, using the objects detected in each image as nodes and the relationships between the objects detected in the images as directed edges. Therefore, the number of knowledge graphs constructed by the computing device based on each image in the data set is the same as the number of images in the data set.

[0128] In conjunction with the example described in step 103, the computing device sets the object set O in the image I to I As a knowledge graph (denoted as G I ) nodes, and the relationship between objects in image I r i,j As a knowledge graph G I The directed edges of the image I and the multiple triples T corresponding to the image I i,j By connecting the same nodes as common nodes, we can get the knowledge graph G corresponding to image I. I Therefore, the knowledge graph G corresponding to image I I Can be viewed as multiple triples T i,j The set composed of: G I ={T i,j ,i≠j,o i ,o j ∈O I}. Since the knowledge graph G I contains all the objects in image I and the relationships between them, so the knowledge graph G I Defined as the relationship graph of image I.

[0129] In the embodiment of the present application, for the scene objects and objects / people detected in the image, the direction of the directed edge is usually: the scene object points to the object / person.

[0130] In one example, in combination with the example described in step 103, when the objects detected by the computing device from the first image include outdoor scenes, people, masks, and faces, and the relationships between the detected objects include: the outdoor scene includes people, the people wear masks, and the people include faces, then the computing device uses the objects in the first image as nodes and the relationships between the objects as directed edges to draw the following: Figure 6 The knowledge graph 60 is shown.

[0131] In an embodiment of the present application, when a computing device draws a knowledge graph corresponding to an image, the knowledge graph also includes nodes for representing the state and / or attributes of objects detected in the image, and directed edges from nodes representing objects to nodes representing the state and / or attributes of the objects.

[0132] In one example, in combination with the example described in step 103, when the objects detected by the computing device from the third image include an office scene and a screen, and the relationship between the detected objects includes: the office scene includes a screen, and the detected state of the screen is bright, the computing device uses the objects and the states of the objects in the third image as nodes, and uses the relationship between the objects and the relationship between the objects and the states as directed edges, and draws the following: Figure 7 The knowledge graph 70 is shown.

[0133] In another example, when the objects detected by the computing device from the seventh image of the data set include a table, a person, a water cup, a mobile phone, and a screen, and the relationships between the detected objects include: the table is near the person, the person is holding a water cup and a mobile phone, and the mobile phone includes a screen, and the states and / or attributes of the detected objects include: the state of the person is standing, the color of the water cup is white, and the state of the screen is bright, the computing device uses the objects in the seventh image and the states / attributes of the objects as nodes, and uses the relationships between the objects and the relationships between the objects and the states (or attributes) as directed edges, and draws Figure 8 The knowledge graph 80 is shown.

[0134] In another example, when the objects detected by the computing device from the eighth image of the data set include a cabinet, a person, a tablet computer, and a screen, and the relationship between the detected objects includes: the cabinet is near the person, the person is holding the tablet computer, and the tablet computer includes the screen, and the states and / or attributes of the detected objects include: the state of the person is lying down, and the state of the screen is bright, the computing device uses the objects in the eighth image and the states / attributes of the objects as nodes, and uses the relationship between the objects and the relationship between the objects and the states (or attributes) as directed edges, and draws Fig. 9 The knowledge graph 90 is shown.

[0135] Optionally, for the knowledge graph constructed by the computing device for each image, the computing device may use a minimum generalization algorithm to merge nodes of the knowledge graph to simplify the knowledge graph. For example, when the knowledge graph includes a node representing a mobile phone and a node representing a tablet computer, since both the mobile phone and the tablet computer are electronic devices with screens, the computing device may merge the two nodes into one node representing the electronic device.

[0136] Step 105: The computing device adds a label node to the tail end of a directed path containing a preset node in multiple knowledge graphs to obtain a first knowledge graph.

[0137] Among them, multiple knowledge graphs are knowledge graphs constructed based on multiple images included in the data set. It should be understood that a knowledge graph includes at least one directed path. A directed path in a knowledge graph refers to: starting from a node that only includes outgoing edges, along the direction indicated by the directed edges, to the path passed by a node that only includes incoming edges. Among them, the outgoing edges included in the node refer to directed edges pointing from the node to other nodes, and the incoming edges included in the node are directed edges pointing from other nodes to the node. In addition, the nodes on the directed paths in the knowledge graph represent the objects contained in the image that constructs the knowledge graph, and the directed edges in the directed paths in the knowledge graph are used to describe the relationship between the objects contained in the image that constructs the knowledge graph.

[0138] Exemplary, combined Figure 8 , the outgoing edge of the node "table" in the knowledge graph 80 refers to the directed edge from the node "table" to the node "person", and the incoming edge of the node "bright" refers to the directed edge from the node "screen" to the node "bright". Figure 8 It can be seen that the node "table" only includes outgoing edges, and the nodes "bright", "white" and "standing" only include incoming edges, so Figure 8 The knowledge graph 80 shown includes three directed paths, namely directed path 1: node "table" → node "person" → node "mobile phone" → node "screen" → node "bright", directed path 2: node "table" → node "person" → node "cup" → node "white", directed path 3: node "table" → node "person" → node "standing".

[0139] In an embodiment of the present application, the computing device constructs multiple knowledge graphs based on multiple images included in the data set, including multiple directed paths containing preset nodes. The nodes on the directed paths represent objects included in the aforementioned multiple images, the directed edges in the directed paths are used to describe the relationship between the objects included in the aforementioned multiple images, and the preset nodes included in the directed paths are nodes located on the directed paths and used to represent objects marked with target labels in the aforementioned multiple images.

[0140] For example, for any of the multiple knowledge graphs constructed by the computing device based on the images in the data set, such as the first graph, when the first graph is constructed based on the first image in the data set, the first graph includes at least one directed path containing preset nodes. The nodes on the directed path represent the objects contained in the first image, the directed edges in the directed path are used to describe the relationship between the objects contained in the first image, and the preset nodes contained in the directed path are located in the directed path and are used to represent the objects marked with the target label in the first image. For example, combined with Figure 6 , the preset nodes included in the directed path in the knowledge graph 60 are nodes representing the faces in the first image. Figure 7 , the preset node included in the directed path in the knowledge graph 70 is the node representing the screen in the third image.

[0141] It should be understood that since the number of objects annotated with the target label in the first image used to construct the first atlas is one or more, the number of preset nodes in the first atlas is also one or more, and the first atlas includes at least one directed path including the preset node. Each directed path including the preset node includes at least one preset node.

[0142] Specifically, for the first graph constructed based on the first image in the data set, the computing device adds a label node to the tail end of the directed path containing the preset node in the multiple knowledge graphs to obtain the first knowledge graph, including: the computing device adds a label node to the tail end of the directed path containing the preset node in the first graph to obtain the directed path containing the label node included in the first knowledge graph. Among them, the tail end of the directed path refers to the end where the end point of the direction indicated by the directed edge on the directed path is located. And, the label node added to the tail end of the directed path containing the preset node is used to record the object represented by the preset node on the directed path. For the first graph constructed based on the first image, the label node added to the tail end of the directed path containing the preset node in the first graph is used to record the object represented by the preset node on the directed path and marked with the target label in the first image. Optionally, the computing device can set a label flag for the label node so that the computing device or image processing device can identify the label node.

[0143] Exemplary, combined Figure 8 ,refer to Fig.10 , when used to construct Figure 8 In the seventh image of the knowledge graph 80 shown, the object marked with the target label includes the screen, that is, the node "screen" in the knowledge graph 80 is a preset node, and the directed path including the preset node in the knowledge graph 80 is the directed path 1 mentioned above, so the computing device adds a label node at the end of the directed path 1 to record the object represented by the preset node in the directed path 1, thereby obtaining Fig.10 The first knowledge graph shown includes a directed path 1 containing labeled nodes.

[0144] Another exemplary method, combined with Fig. 9 ,refer to Fig.10 , when used to construct Fig. 9The objects marked with the target label in the eighth image of the knowledge graph 90 shown include screens, that is, the node "screen" in the knowledge graph 90 is a preset node, and the directed path including the preset node in the knowledge graph 90 is the directed path 4 described above: node "cabinet" → node "person" → node "tablet" → node "screen" → node "bright", so the computing device adds a label node at the end of the directed path 4 to record the object represented by the preset node in the directed path 4, thereby obtaining Fig.10 The first knowledge graph shown includes a directed path 4 containing labeled nodes.

[0145] It should be understood that when a directed path containing preset nodes contains multiple preset nodes, the label node added at the tail end of the directed path is used to record the objects represented by the multiple preset nodes on the directed path, that is, the label node records the set of objects represented by the multiple preset nodes.

[0146] It should also be understood that when the knowledge graph corresponding to an image includes multiple directed paths containing preset nodes, correspondingly, the computing device can construct a first knowledge graph based on the knowledge graph corresponding to the image, including multiple directed paths containing label nodes.

[0147] Optionally, the computing device may use a minimum general generalization algorithm to generalize and merge multiple directed paths containing labeled nodes obtained from a knowledge graph constructed based on images in a data set. For detailed description, please refer to the description of step 106 below and will not be repeated here.

[0148] In this way, based on the above process, the computing device can obtain the directed paths with added labeled nodes based on the knowledge graph constructed based on each image, thereby obtaining a first knowledge graph consisting of a set of directed paths with added labeled nodes.

[0149] It can also be seen from the above that when the directed path in the knowledge graph corresponding to the image (the conjunction of the triples corresponding to the image) is defined as the rule body of the rule, and the label node used to record the set of objects represented by the preset nodes in the directed path (that is, the set of objects marked with the target label) is defined as the rule head of the rule, then the rule can be expressed as a logical implication expression: body→head. Among them, body represents a series of conditions, and head represents the set of objects marked with the target label in the image, so that when body is true as a condition, head is true. In an embodiment of the present application, a logical implication expression represents a rule.

[0150] Since the expression of the graph structure of the first knowledge graph is linked in the form of triples through nodes and edges, the granularity is fine and easy to change, and the nodes and edges have semantic relationships, so it can be used for downstream tasks in a variety of applications. Therefore, when the rule base is represented by the first knowledge graph and used to detect the target object to be performed in the image to be processed in the image processing method described below, the solution provided by the embodiment of the present application is highly scalable.

[0151] To simplify the first knowledge graph built based on the images in the dataset, refer to Fig.11 , Fig.11 A schematic diagram of another method for constructing a first knowledge graph provided by an embodiment of the present application is shown. The method is executed by any computing device (or apparatus) with computing processing capabilities, and the present application embodiment does not limit this. In one example, the method can be executed by the image processing apparatus described above. Fig.11 As shown, after executing steps 101 and 103 to 105, the method may further execute steps 106 to 107.

[0152] Step 106: The computing device merges the directed paths to which the label nodes are added by the nodes representing the same object and the same type of objects on the directed paths to which the label nodes are added, to obtain a simplified first knowledge graph.

[0153] Here, the same type of object refers to the same type of object. The directed path with the label node added is the computing device executing Figure 5 The directed path containing the label node obtained by the method shown. For the sake of simplicity, the directed path with the label node added is recorded as a regular path (e.g. Fig.10 Directed path 1 and directed path 4 are shown).

[0154] In a possible implementation, the computing device merges multiple rule paths obtained from the knowledge graph constructed based on the images in the data set by representing the same object nodes on the rule paths to obtain a simplified first knowledge graph. Fig.10 , when the computing device executes Figure 5 The rule paths obtained after the method shown include Fig.10 As shown in the directed path 1 and directed path 4, since there are multiple nodes representing the same object in the directed path 1 and directed path 4, such as the node “person”, the node “screen” and the node “bright”, the computing device merges the nodes representing the same object in the directed path 1 and directed path 4 into one node based on the nodes representing the same object in the directed path 1 and directed path 4, such as merging the node “person” in the directed path 1 and directed path 4 into Fig.12 The node "person" shown in the figure merges the nodes "screen" in directed paths 1 and 4 into Fig.12The node "screen" shown in FIG. 1 and the node "bright" in directed path 1 and directed path 4 are merged into Fig.12 The node shown is "bright", so the directed path 1 and the directed path 4 can be merged to obtain Fig.12 The first knowledge graph shown.

[0155] In another possible implementation, the computing device merges multiple rule paths obtained from the knowledge graph constructed based on the images in the data set by using nodes on the rule paths that represent the same object and the same type of object, thereby obtaining a first knowledge graph. The merging of rule paths based on nodes that represent the same type of object can also be understood as the computing device merging rule paths using the minimum general generalization algorithm in the related art. Fig.10 , when the computing device executes Figure 5 The rule paths obtained after the method shown include Fig.10 As shown in the directed path 1 and directed path 4, there are multiple nodes representing the same object in the directed path 1 and directed path 4, such as the node "person", the node "screen" and the node "bright", and there are also nodes representing the same type of objects in the directed path 1 and directed path 4, such as the node "mobile phone" and the node "tablet computer", wherein the mobile phone and the tablet computer are both electronic devices. Therefore, the computing device merges the nodes representing the same object in the directed path 1 and directed path 4 into one node based on the nodes representing the same object in the directed path 1 and directed path 4, such as merging the node "person" in the directed path 1 and directed path 4 into Fig.13 The node "person" shown in the figure merges the nodes "screen" in directed paths 1 and 4 into Fig.13 The node "screen" shown in the figure merges the nodes "bright" in directed paths 1 and 4 into Fig.13 Furthermore, the computing device merges the nodes representing the same type of objects in directed path 1 and directed path 4 into a node that can represent the same type of objects, such as merging the node "mobile phone" in directed path 1 and the node "tablet computer" in directed path 4 into a node that can represent the same type of objects. Fig.13 , so that directed path 1 and directed path 4 can be merged to obtain Fig.13 The first knowledge graph shown.

[0156] It should be noted that when merging multiple rule paths based on nodes representing the same object or the same type of object, the directed edges connecting the same nodes in different rule paths will be deleted, so as to ensure that in the first knowledge graph obtained after merging the rule paths, there is only one directed edge between any two nodes connected by directed edges. This process is also called merging the same directed edges in different rule paths. The same directed edges refer to the directed edges connecting nodes representing the same object or the same type of object in different rule paths. For example, Fig.10 , the directed edge in directed path 1 connecting the node "screen" and the node "bright" and the directed edge in directed path 4 connecting the node "screen" and the node "bright" are the same directed edge. Fig.10 , since the node “mobile phone” in directed path 1 and the node “tablet computer” in directed path 4 represent the same type of objects, namely electronic devices, the directed edge in directed path 1 used to connect the node “person” and the node “mobile phone” and the directed edge in directed path 4 used to connect the node “person” and the node “tablet computer” are the same directed edge, and the directed edge in directed path 1 used to connect the node “mobile phone” and the node “screen” and the directed edge in directed path 4 used to connect the node “tablet computer” and the node “screen” are also the same directed edge.

[0157] It should also be noted that after the computing device obtains multiple rule paths based on the knowledge graph constructed by the images in the data set, the multiple rule paths can be merged into at least one knowledge graph. And, when there are independent rule paths among the multiple rule paths. Among them, an independent rule path means that the objects represented by the nodes in the rule path are different from the objects represented by the nodes in other rule paths, and are not the same type of objects. That is, an independent rule path cannot be merged with other rule paths. In this case, the first knowledge graph constructed by the computing device based on the data set includes at least one knowledge graph and at least one independent rule path, and each graph in the at least one knowledge graph is obtained by merging at least two rule paths.

[0158] In this way, a large number of redundant node and edge expressions in the regular path constructed based on the image in the data set can be removed, thereby saving storage resources for storing the first knowledge graph.

[0159] Step 107: The computing device records the regular paths to which the merged nodes and merged directed edges belong in the first knowledge graph.

[0160] The merged node is a node formed by merging multiple nodes belonging to different rule paths. For example, in combination with the example of merging rule paths based on nodes representing the same object in step 106, Fig.12The nodes "person", "screen" and "bright" in the knowledge graph obtained after the merge rule path shown are merged nodes. For another example, combined with the example of merging rule paths based on nodes representing the same object and the same type of object in step 106, Fig.13 The nodes “person”, “screen”, “bright” and “electronic device” in the knowledge graph obtained after the merge rule path shown are merged nodes.

[0161] The merged directed edge is a directed edge formed by merging multiple directed edges belonging to different regular paths. Specifically, the merged directed edge is a directed edge formed by merging multiple identical directed edges belonging to different regular paths. For detailed description of identical directed edges, refer to the relevant description in step 106 and will not be repeated here. Fig.10 The directed path 1 and directed path 4 shown are combined to obtain Fig.12 In the knowledge graph shown, the directed edge connecting the node "screen" and the node "bright", as well as the directed edge connecting the node "bright" and the label node for recording the object "screen", are all merged directed edges. Fig.10 The directed path 1 and directed path 4 shown are combined to obtain Fig.13 In the knowledge graph shown, the directed edge connecting the node "screen" and the node "bright", the directed edge connecting the node "bright" and the label node used to record the object "screen", the directed edge connecting the node "person" and the node "electronic device", and the directed edge connecting the node "electronic device" and the node "screen" are all merged directed edges.

[0162] In the embodiment of the present application, a rule path represents a rule. When the computing device merges multiple rule paths into the first knowledge graph in step 106, it is necessary to record the rule paths to which the merged nodes and merged directed edges in the first knowledge graph originally belong, and it is necessary to record the rule paths to which each directed edge in the first knowledge graph originally belongs. The rule paths to which the merged nodes and merged directed edges in the first knowledge graph, as well as the rule paths to which each directed edge originally belongs, recorded by the computing device, can be associated with the first knowledge graph as descriptive information of the first knowledge graph, so as to distinguish the rule paths that match the image to be processed when the image to be processed is subsequently inferred.

[0163] Exemplary, combined Fig.10 and Fig.12 ,because Fig.12 The merged nodes shown are: node "person", node "screen" and node "bright", which belong to Fig.10The directed path 1 and directed path 4 are shown, so the computing device records: the node "person" belongs to the directed path 1 and directed path 4, the node "screen" belongs to the directed path 1 and directed path 4, and the node "bright" belongs to the directed path 1 and directed path 4. Similarly, due to Fig.12 The merged directed edges shown in the figure: the directed edge connecting the node "screen" and the node "bright", the directed edge connecting the node "bright" and the label node used to record the object "screen", belong to Fig.10 As shown, directed path 1 and directed path 4, the computing device records: the directed edge connecting the node “screen” and the node “bright” belongs to directed path 1 and directed path 4, and the directed edge connecting the node “bright” and the label node used to record the object “screen” belongs to directed path 1 and directed path 4.

[0164] Another exemplary method, combined with Fig.10 and Fig.13 ,because Fig.13 The merged nodes shown: node "person", node "screen", node "bright", node "electronic equipment" belong to Fig.10 The directed path 1 and directed path 4 shown in FIG. 1 , therefore, the computing device records: the node “person” belongs to the directed path 1 and directed path 4, the node “screen” belongs to the directed path 1 and directed path 4, the node “bright” belongs to the directed path 1 and directed path 4, and the node “electronic device” belongs to the directed path 1 and directed path 4. Similarly, due to Fig.13 The merged directed edges shown: the directed edge connecting the node "screen" and the node "bright", the directed edge connecting the node "bright" and the label node for recording the object "screen", the directed edge connecting the node "person" and the node "electronic device", and the directed edge connecting the node "electronic device" and the node "screen", belong to Fig.10 As shown, the directed path 1 and directed path 4, therefore, the computing device records: the directed edge connecting the node "screen" and the node "bright" belongs to directed path 1 and directed path 4, the directed edge connecting the node "bright" and the label node used to record the object "screen" belongs to directed path 1 and directed path 4, the directed edge connecting the node "person" and the node "electronic device" belongs to directed path 1 and directed path 4, and the directed edge connecting the node "electronic device" and the node "screen" belongs to directed path 1 and directed path 4.

[0165] In some examples, the process of steps 104 to 107 can also be understood as: for each image in the data set D (such as image I), the computing device draws the knowledge graph G of image I based on the process of step 104 above. I Next, the computing device is based on G IExecute step 105 to obtain at least one directed path (i.e., a regular path) containing a label node. Then, the computing device uses the least general generalization algorithm to map the knowledge graph {G I ,I∈D} are merged to obtain n (n≤D) directed paths representing the rule {rule_i,i∈[1,n]}.

[0166] Among them, each Rule_i can be regarded as a directed path (rule: body→head) (see the above description), in which the nodes of the path are objects and the directed edges are the relationships between objects. By executing steps 106 to 107, the computing device can merge the same nodes in all paths {Rule_i, i∈[1,n]} into a new node, and record the set of rule paths to which the new node originally belongs, thereby obtaining the first knowledge graph (recorded as a directed graph DG'). The computing device also records the rule path Rule_j, j∈[1,n] to which each directed edge in DG' originally belongs. Since there may be a situation in which there are multiple directed edges between two nodes in DG' obtained after merging the rule paths, the computing device merges the edges representing the same relationship between the two nodes into one edge, and records the set of rule paths to which the merged edges originally belong, thereby achieving the purpose of simplifying multiple rule paths. In this way, the first knowledge graph G representing the rule base can be obtained. It can be seen that the first knowledge graph G includes at least one graph obtained by merging rule paths, and includes at least one independent rule path (refer to the description of "independent rule path" above).

[0167] Furthermore, when passing Figure 4 The rule base constructed by the method is deployed in the image processing device, or Figure 5 or Fig.11 The first knowledge graph constructed by the method is deployed in an image processing device, and the image processing device can detect the target object of the target operation to be performed in the image to be processed through the rule base or the first knowledge graph, and complete the processing of the image to be processed after performing the target operation on the target object. When the target operation is a desensitization operation, it means that the target object is a privacy object in the image to be processed, so that the privacy protection processing of the image to be processed can be achieved by performing the desensitization operation on the privacy object in the image to be processed. Since the rules and knowledge graphs in the rule base are interpretable, when the target object of the target operation to be performed in the image to be processed is detected based on the rule base or the first knowledge graph, it also has strong interpretability, and when the target object of the target operation to be performed in the image to be processed is detected based on the rule base or the first knowledge graph, the accuracy of detecting the target object in the image to be processed can be improved.

[0168] The following describes the process of processing the image to be processed based on the rule base or the first knowledge graph constructed above.

[0169] refer to Fig.14 , Fig.14 The following is a flow chart of an image processing method provided by an embodiment of the present application. Optionally, the method can be applied to Figure 2 or Figure 3 The implementation environment shown in Figure 2 or Figure 3 The image processing device in the implementation environment shown in the figure is executed, and the image processing device is deployed with Figure 4 The rule base constructed by the method described above, or deployed with Figure 5 or Fig.11 The first knowledge graph constructed by the method. Fig.14 As shown, the method includes the following steps 201 to 203.

[0170] Step 201: The image processing apparatus obtains an image to be processed.

[0171] In a possible situation, the image to be processed is stored locally in the image processing device, and therefore, the image processing device acquires the image to be processed, including: the image processing device acquires the image to be processed stored locally.

[0172] Exemplary, combined Figure 2 In the application scenario shown, when the image processing apparatus is applied to a terminal device, that is, when the method is executed by the terminal device, it means that the terminal device is deployed with the above rule base or the first knowledge graph. In this case, the image processing apparatus obtains the image to be processed, including: the terminal device obtains the image to be processed stored locally on the terminal device.

[0173] In another possible situation, the image processing apparatus obtains the image to be processed, including: the image processing apparatus receives the image to be processed sent by other devices.

[0174] Exemplary, combined Figure 2 In the application scenario shown, when the image processing apparatus is applied to a terminal device, that is, when the method is executed by the terminal device, it means that the terminal device is deployed with the above rule base or the first knowledge graph. In this case, the image processing apparatus obtains the image to be processed, including: the terminal device receives the image to be processed sent by other devices.

[0175] Another exemplary method is to combine Figure 3 In the application scenario shown, when the image processing device is applied to any application server, that is, when the method is executed by the application server, it means that the application server is deployed with the above rule base or the first knowledge graph. In this case, the image processing device obtains the image to be processed, including: receiving the image to be processed uploaded / published by the terminal device.

[0176] Step 202: The image processing apparatus determines a target object in the image to be processed according to the image to be processed and the rule library.

[0177] The target object is the object on which the target operation is to be performed in the image to be processed. The rule base is extracted in advance based on the images in the data set, and the data set includes multiple images, each of which contains an object marked with a target label, and the object marked with the target label is the object for performing the target operation. For any rule in the rule base, such as the first rule, the first rule is extracted based on the first image in the data set, and the first rule is used to indicate the object marked with the target label in the first image under the condition that the first image includes the objects and the relationship between the objects. The construction process of the rule base and related instructions can refer to the description of steps 101 to 102, and will not be repeated here.

[0178] In a possible implementation, the image processing device first detects and identifies the objects and the relationships between the objects in the image to be processed (for detailed description, refer to the relevant description of step 102), and then the image processing device queries the rule base based on the objects and the relationships between the objects detected from the image to be processed, so as to find the target rule including the objects and the relationships between the objects in the image to be processed in the rule base. Then, the image processing device determines the object marked with the target label indicated by the target rule as the target object in the image to be processed. Since the rules in the rule base provided in the embodiment of the present application are interpretable rules, the image processing method provided in the embodiment of the present application has strong interpretability when detecting the target object to be executed in the image to be processed.

[0179] In another possible implementation, when the rule base is implemented as the first knowledge graph mentioned above, that is, when the rules in the rule base are represented by directed paths in the first knowledge graph, the image processing device determines the target object in the image to be processed according to the image to be processed and the rule base, including: the image processing device determines the target object in the image to be processed according to the image to be processed and the first knowledge graph. Fig.15 , Fig.15 A schematic diagram of a process for determining a target object in an image to be processed based on the image to be processed and the first knowledge graph provided by an embodiment of the present application is shown. Fig.15 As shown, the process includes steps 2021 to 2023.

[0180] Step 2021: The image processing device constructs a second knowledge graph based on the image to be processed.

[0181] The nodes in the second knowledge graph represent the objects contained in the image to be processed, and the directed edges in the second knowledge graph are used to describe the relationship between the objects contained in the image to be processed. The detailed description of the image processing apparatus constructing the second knowledge graph according to the image to be processed can refer to the description of the computing device constructing the knowledge graph corresponding to the image according to the image in the data set in steps 103 to 104, which will not be repeated here.

[0182] Step 2022: The image processing device queries the first knowledge graph for a target path including nodes and directed edges in the second knowledge graph.

[0183] The target path is one of the multiple directed paths included in the first knowledge graph, and the nodes used to represent objects and the directed edges used to describe the relationship between objects on the target path exist in both the first knowledge graph and the second knowledge graph. It should be understood that the second knowledge graph includes nodes other than the label nodes on the target path.

[0184] In some examples, the second knowledge graph constructed by the image processing device according to the image to be processed is G Img For example, refer to Fig.16 The process of the image processing device searching the first knowledge graph for a target path including nodes and directed edges in the second knowledge graph can be implemented by the following steps S11 to S14. Img = <N Img ,E Img >,N Img G Img The node set in E Img G Img The set of directed edges in .

[0185] S11, the image processing device performs image processing according to the second knowledge graph G Img The root node of the first knowledge graph determines the target root node.

[0186] The root node of the graph refers to the node that only includes outgoing edges, such as Fig.10 The nodes "table" and "cabinet" shown are both root nodes.

[0187] Optionally, the image processing device may be configured to process the image according to G Img The root node of the target knowledge graph is determined by using the following method 1 or method 2.

[0188] Method 1: The image processing device uses G Img The root node traverses each root node {root i ,i∈[1,m]}, thus combining the first knowledge graph with G ImgThe root node with the same root node as the target root node is determined as the target root node. Wherein, i and m are both integers, and m represents the number of root nodes in the first knowledge graph.

[0189] Method 2: The image processing device points the preset pointer to each root node in the first knowledge graph in turn, and determines in turn whether the root node currently pointed to by the pointer is G Img The root node of G Img When the root node of the pointer is strongly pointed to, the root node is determined as the target root node. The pointer represents a logical call sequence, for example, it can be understood as a pointer in the code that implements the process described in S11 to S14.

[0190] S12, the image processing device determines whether the outgoing edge e of the node in the first knowledge graph currently pointed to by the pointer exists in the second knowledge graph G Img The outgoing edge set E Img middle.

[0191] Where, e∈E p , E p is the set of all outgoing edges of the node that the pointer currently points to.

[0192] It should be understood that when the image processing device determines the target root node using the above method 1, the image processing device points the preset pointer to the target root node in S12. When the image processing device determines the target root node using the above method 2, the pointer already points to the target root node.

[0193] It should also be understood that any outgoing edge e of a node ij It can be expressed as <o i ,r i,j ,o j >. Among them, o i ,r i,j ,o j The detailed description can be found above and will not be repeated here.

[0194] When the image processing device determines the outgoing edge set E of the node currently pointed by the pointer p In G, Img Outgoing edge e in the outgoing edge set of the root node ij = <o i ,r i,j ,o j >, the image processing device executes S13.

[0195] When the image processing device determines the outgoing edge set E of the node currently pointed by the pointer p does not contain the Img Outgoing edge e in the outgoing edge set of the root node ij = <oi ,r i,j ,o j >, the image processing device determines that the search for the target path in the first knowledge graph has failed, that is, the image processing device fails to detect the target object in the image to be processed according to the first knowledge graph, and the image processing device ends the process.

[0196] S13, the image processing device moves the pointer to point to the outgoing edge e ij The tail node is denoted as N1.

[0197] The tail node of the outgoing edge refers to the node to which the outgoing edge points. ij The control pointer moves to the next node of the currently pointed node, that is, node N1.

[0198] When the edge ij If the number is multiple, the image processing device performs a ij Execute S13 to S14, which will not be described in detail.

[0199] S14. The image processing apparatus determines whether the node N1 is a label node.

[0200] Optionally, the image processing device may determine whether the node N1 is a label node according to whether the node N1 contains a label flag (refer to the above description). For example, when the image processing device determines that the node N1 contains a label flag, it determines that the node N1 is a label node. Correspondingly, when the image processing device determines that the node N1 does not contain a label flag, it determines that the node N1 is not a label node.

[0201] Optionally, since the label node is the last node on the directed path in the first knowledge graph, the image processing device can determine whether the node N1 is a label node by determining whether the node N1 contains outgoing edges pointing to other nodes. For example, when the image processing device determines that the node N1 does not contain outgoing edges pointing to other nodes, it determines that the node N1 is the last node of the current path, and then determines that the node N1 is a label node. Correspondingly, when the image processing device determines that the node N1 contains outgoing edges pointing to other nodes, it determines that the node N1 is not the last node of the current path, and then determines that the node N1 is not a label node.

[0202] When the image processing apparatus determines that the node N1 is a label node, the image processing apparatus determines that the directed path to which the label node belongs is the target path.

[0203] When the image processing device determines that the node N1 is not a label node, the image processing device returns to step S12 and re-executes S12 to S14 for the node N1 currently pointed to by the pointer until the target path is determined or the process ends.

[0204] When the image processing device fails to determine the target path in the first knowledge graph, it indicates that the query fails. Optionally, the image processing device can subsequently add the target path to the first knowledge graph based on the user's processing requirements for the image to be processed, that is, add the rules required by the user to the rule base. For detailed instructions, please refer to the description of adding rules to the first knowledge graph below.

[0205] Step 2023: The image processing device determines the object recorded in the label node of the target path as the target object in the image to be processed.

[0206] It can be understood that when the label node of the target path records multiple objects, it means that the number of target objects is multiple. When the label node of the target path records only one object, it means that the number of target objects is one.

[0207] Step 203: The image processing apparatus performs a target operation on the target object in the image to be processed to complete processing of the image to be processed.

[0208] Optionally, when the target label annotated in the image of the data set for constructing the first knowledge graph is a privacy label, it means that the target object determined based on the first knowledge graph is a privacy object in the image to be processed, so the target operation performed by the image processing device on the target object in the image to be processed is a desensitization operation. Furthermore, the image processing device performs a desensitization operation on the target object in the image to be processed to complete the privacy protection processing of the image to be processed.

[0209] Optionally, when the target label annotated in the image of the data set for constructing the first knowledge graph is a special attention label, it means that the target object determined based on the first knowledge graph is an object of special attention to the user in the image to be processed. Therefore, the target operation performed by the image processing device on the target object in the image to be processed is a highlighting operation, and then, the image processing device performs a highlighting operation on the target object in the image to be processed to complete the highlighting of the object of special attention to the user in the image to be processed, thereby improving the user experience.

[0210] Through steps 201 to 203, the purpose of detecting the target object of the target operation to be performed in the image to be processed based on the first knowledge graph is achieved, and after performing the target operation on the target object, the corresponding processing of the image to be processed can be achieved. For example, when the target operation is a desensitization operation, it means that the target object is a private object in the image to be processed, so by performing the desensitization operation on the private object in the image to be processed, the privacy protection processing of the image to be processed can be achieved.

[0211] Moreover, since the rules in the rule base provided by the embodiment of the present application are interpretable rules, and the embodiment of the present application uses the form of knowledge graph to express and fuse the common rules contained in the image set, uses a visual way to display the rule logic, and uses the method of graph path matching to infer the target object of the target operation to be performed in the image to be processed, the image processing method provided by the embodiment of the present application has strong interpretability when detecting the target object of the target operation to be performed. Or it can be understood that any node and edge in the first knowledge graph provided by the embodiment of the present application has real semantics that can be understood by humans, so this representation method has interpretability, and when detecting the target object of the target operation to be performed in the image to be processed based on the first knowledge graph, logical reasoning is used, so the interpretability of detecting the target object of the target operation to be performed in the image to be processed based on the first knowledge graph is strong.

[0212] In addition, when the target object of the target operation to be performed in the image to be processed is detected based on the rule base or the first knowledge graph, the accuracy of detecting the target object in the image to be processed can be improved. Compared with the accuracy of detecting the target object of the target operation to be performed in the image to be processed based on the deep learning model in the related art, the solution provided in the embodiment of the present application has higher accuracy in detecting the target object of the target operation to be performed in the image to be processed.

[0213] Exemplarily, Table 1 shows the comparison of the detection accuracy of the target object to be performed in the image to be processed by using the relevant technology and the solution provided by the embodiment of the present application. As shown in Table 1, when the target object to be performed in the image to be processed is detected by the deep learning model in the relevant technology, the detection result accuracy is 0.510, while when the target object to be performed in the image to be processed is detected by the solution provided by the embodiment of the present application, the detection result accuracy is 0.807.

[0214] Table 1

[0215] Related technologies This application Accuracy of test results 0.510 0.807

[0216] In addition, since the reasoning of the graph structure only includes logical implication calculations, that is, the reasoning based on the graph structure is to infer the target object through logical operations, without the need for large-scale neural network calculations as in the related art, the complexity of the reasoning is strictly limited in theory, saving computing resources, so that the reasoning speed is faster than the deep learning model. That is, compared with the speed of detecting the target object to be executed in the image to be processed based on the deep learning model, the solution provided by the embodiment of the present application detects the target object to be executed in the image to be processed faster.

[0217] Exemplarily, Table 2 shows the comparison of the detection time of the related technology and the solution provided by the embodiment of the present application when the target object to be performed in the image to be processed is detected by the related technology and the solution provided by the embodiment of the present application. As shown in Table 2, when the target object to be performed in the image to be processed is detected by the deep learning model in the related technology, the time to complete the detection is 218 milliseconds, while when the target object to be performed in the image to be processed is detected by the solution provided by the embodiment of the present application, the time to complete the detection is 201 milliseconds.

[0218] Table 2

[0219] Related technologies This application Time to complete the test 218 ms 201 ms

[0220] In some embodiments, the rule base constructed above needs to be updated.

[0221] For example, it is necessary to add new rules to the rule base based on the newly added dataset images.

[0222] For another example, you need to delete rules from a rule base that has been built.

[0223] For another example, it is necessary to modify the rules in the rule base in response to the feedback from the user after using the rule base to detect the image to be processed. Among them, the implementation method of modifying the rules can be to delete the rules used to detect the results that do not match the user's expected detection results, and then add new rules that match the user's expected detection results. In one example, when the image processing device determines that the target object determined in the image to be processed based on the rule base is not the object on which it expects to perform the target operation, the image processing device can modify the rules in the rule base. In this case, the image processing device can update the rule base according to the image to be processed in response to the first indication of the user's feedback. Among them, the first indication is used to indicate the execution of the target operation on other objects in the image to be processed that are different from the target object, and the other objects are the objects on which the user expects to perform the target operation.

[0224] Optionally, when the rule base is a rule base constructed through steps 101 to 102, the computing device or the image processing device that applies the rule base can add new rules to the rule base according to new images marked with target tags, or delete rules from the rule base according to needs (such as user needs).

[0225] Optionally, when the rule base is represented by the first knowledge graph described above, the computing device or the image processing device using the first knowledge graph can add a directed path representing the rule in the first knowledge graph according to a new image annotated with a target tag, or delete the directed path representing the rule in the first knowledge graph according to demand (such as user demand). It should be understood that when deleting a directed path or adding a directed path in the first knowledge graph, it is necessary to ensure that the structure and content of other directed paths in the first knowledge graph are not affected.

[0226] In one example, after the image processing device executes the above steps 201 to 202, if the image processing device determines that the target object determined in step 202 is not the object that the user expects to perform the target operation on, the image processing device can respond to the first instruction fed back by the user and used to instruct to perform the target operation on other objects in the image to be processed that are different from the target object, and update the first knowledge graph according to the image to be processed. Specifically, the image processing device can first delete the target path including the label node for recording the target object in the first knowledge graph, and then add a directed path in the first knowledge graph according to the image to be processed and the first instruction, and the label node on the newly added directed path is used to record the aforementioned other objects.

[0227] The following describes a process in which the image processing apparatus deletes a target path including a label node for recording a target object in the first knowledge graph. Fig.17 , the process of the image processing device deleting the target path in the first knowledge graph can be implemented by the following S21 to S25.

[0228] S21. The image processing device searches the first knowledge graph for a node that is the same as the node pointed to by the pointer in the target path, denoted as Np.

[0229] Specifically, when the image processing device executes S21 for the first time, it can point the preset pointer to the root node in the target path, and then search whether there is a node Np identical to the root node in the first knowledge graph based on the root node currently pointed to by the pointer.

[0230] The pointer represents a logical calling sequence, for example, it can be understood as a pointer in the code that implements the process described in S21 to S25.

[0231] When the image processing device determines that the node Np does not exist in the first knowledge graph, it means that the first knowledge graph does not include a path identical to the target path, and thus the process of deleting the target path ends.

[0232] When the image processing device determines that there is a node Np in the first knowledge graph, the image processing device records the node Np and adds the node Np to the search rule set F.

[0233] S22. The image processing device searches among all outgoing edges of the node Np in the first knowledge graph whether an edge is included that is the same as the outgoing edge of the node currently pointed to by the pointer in the target path, denoted as Ep.

[0234] Since the target path is a directed path, if the node on the target path includes an outgoing edge, only one outgoing edge is included. Therefore, it can also be understood that the image processing device searches for all outgoing edges of the node Np in the first knowledge graph to see whether the outgoing edge Ep of the node currently pointed to by the pointer in the target path is included. It should be understood that when S22 is executed for the first time, the node currently pointed to by the pointer in the target path is the root node of the target path, so the outgoing edge Ep is the outgoing edge of the root node of the target path.

[0235] When the image processing device determines that all outgoing edges of the node Np in the first knowledge graph do not include Ep, it means that the first knowledge graph does not include a path identical to the target path, and thus the process of deleting the target path ends.

[0236] When the image processing device determines that all outgoing edges of the node Np in the first knowledge graph contain Ep, it records Ep and adds Ep to the search rule set F.

[0237] S23. The image processing device moves the pointer along the direction indicated by Ep to the node pointed by Ep in the target path, which is recorded as Np_next.

[0238] S24: The image processing apparatus determines whether the node Np_next in the target path is a label node.

[0239] The detailed description of how the image processing apparatus determines whether the node Np_next is a label node can be referred to the relevant description in the above S14 and will not be repeated here.

[0240] When the image processing device determines that the node Np_next is a label node, it means that the process of searching for the object to be deleted in the first knowledge graph in this deletion operation has been completed, and the image processing device continues to execute S25.

[0241] When the image processing device determines that the node Np_next is not a label node, it means that the process of searching for the object to be deleted in the first knowledge graph in this deletion operation has not been completed. The image processing device executes S21 to S24 for the node Np_next currently pointed to by the pointer, and repeats this cycle until the process of searching for the object to be deleted in the first knowledge graph is completed.

[0242] S25. The image processing device deletes the target path in the first knowledge graph according to the nodes and edges recorded in the search rule set F obtained in S21 to S24.

[0243] Specifically, for any node (or edge) recorded in the search rule set F, for example, for node 1 (or edge 1) recorded in the search rule set F, the image processing device deletes the ID of the target path from the set of rule paths to which node 1 (or edge 1) belongs, which is recorded in the description information of the first knowledge graph, based on the identifier (ID) of the target path. When the ID of the target path is deleted from the set of rule paths to which node 1 (or edge 1) belongs, the set is empty, indicating that node 1 (or edge 1) does not belong to any rule path. At this time, the image processing device deletes node 1 (or edge 1) in the first knowledge graph.

[0244] In this way, through S21 to S25, the purpose of deleting the target path in the first knowledge graph can be achieved, and when the target path is deleted, other rule paths in the first knowledge graph will not be affected.

[0245] The following describes a process in which the image processing device adds a directed path to the first knowledge graph according to the image to be processed and the first instruction. Fig.18 , this process can be implemented through the following S31~S34.

[0246] S31. The image processing device generates a path to be added according to the image to be processed and the first instruction.

[0247] The first instruction is used to instruct to perform a target operation on other objects (referred to as expected objects) in the image to be processed that are different from the target object. The target object is the target object for the target operation to be performed determined in step 202 in the image to be processed.

[0248] In response to the first indication, the image processing device can determine that the expected object in the image to be processed is the object of the target operation to be performed, so the image processing device labels the expected object in the image to be processed with a target label corresponding to the target operation. Furthermore, the image processing device generates a regular path containing label nodes based on the image to be processed labeled with the target label. The detailed process can be referred to the description of steps 103 to 105 above and will not be repeated here. In this way, the regular path containing label nodes generated by the image processing device based on the image to be processed labeled with the target label is the path to be added that needs to be added to the first knowledge graph.

[0249] S32. The image processing device searches the root nodes of the first knowledge graph for the root node currently pointed to by the pointer in the path to be added, based on the root node currently pointed to by the pointer in the path to be added, and records it as Np.

[0250] The pointer represents a logic calling sequence, for example, it can be understood as a pointer in the code implementing the process described in S32 to S34. It should be understood that when S32 is executed for the first time, the image processing device sets the pointer to point to the root node in the path to be added.

[0251] When the image processing device determines that the node Np does not exist in the root node of the first knowledge graph, it means that the root node of the path to be added does not appear in the first knowledge graph. Therefore, the image processing device adds the root node Np to the first knowledge graph, and adds the edges contained in the path to be added to the node Np in the first knowledge graph.

[0252] When the image processing device determines that there is a node Np in the root node of the first knowledge graph, S33 is executed.

[0253] S33: The image processing device moves the pointer to the node pointed to by the outgoing edge of the node Np in the path to be added, which is recorded as Np_next.

[0254] S34: The image processing apparatus determines whether the node Np_next in the path to be added is a label node.

[0255] The detailed description of how the image processing apparatus determines whether the node Np_next is a label node can be referred to the relevant description in the above S14 and will not be repeated here.

[0256] When the image processing device determines that the node Np_next in the path to be added is a label node, it means that all nodes except the label node in the path to be added have been added to the first knowledge graph. Therefore, the image processing device adds the label node Np_next after the node Np in the first knowledge graph, thereby completing the addition of the path to be added in the first knowledge graph.

[0257] When the image processing device determines that the node Np_next in the path to be added is not a label node, then for the node Np_next currently pointed to by the pointer, the image processing device searches for the node Np_next currently pointed to by the pointer in the path to be added, and finds whether the node Np_next currently pointed to by the pointer in the first knowledge graph exists in the node pointed to by the outgoing edge of the node Np in the first knowledge graph. Then, the image processing device executes S33 to S34 for the node Np_next currently pointed to by the pointer, and repeats this until the path to be added in the first knowledge graph is completed.

[0258] In this way, through S31 to S34, the purpose of adding a directed path (ie, rule path) representing the rule in the first knowledge graph can be achieved.

[0259] It can be seen that through Fig.17 and Fig.18The process updates the first knowledge graph, and can obtain the first knowledge graph that meets the personalized needs of the user. Moreover, the update of the first knowledge graph for the personalized data of the user only involves the addition and deletion of the paths in the graph, so it can not only efficiently meet the needs of rule personalization and dynamic changes, but also, compared with the deep learning model used to fine-tune the target object to be executed in the image to be processed according to the personalized data of the user in the related art, the efficiency of updating the first knowledge graph according to the personalized data of the user in the embodiment of the present application is higher.

[0260] Exemplarily, Table 3 shows a comparison of the time required to fine-tune the deep learning model in the related technology according to the user personalized data and update the first knowledge graph provided by the embodiment of the present application. As shown in Table 3, the time required to fine-tune the deep learning model in the related technology according to the user personalized data is 5 minutes, while the time to update the first knowledge graph according to the user personalized data is only 209 milliseconds.

[0261] Table 3

[0262] Related technologies This application Fine-tune / update time 5 minutes 209 ms

[0263] In order to deepen the understanding of the scheme described in the embodiment of the present application, it is further explained below with reference to specific examples.

[0264] refer to Fig.19 , Fig.19 FIG. 2 is a schematic diagram showing another image processing method provided by an embodiment of the present application. Fig.19 As shown, the computing device first obtains the data set and then Figure 5 or Fig.11 The method constructs a first knowledge graph. Then, the first knowledge graph is deployed on a user device (such as Figure 2 The computing device shown or Figure 3 In this way, when the user device obtains the image to be processed, it executes Fig.14 The method shown in the figure detects the image to be processed based on the first knowledge graph, processes the image to be processed, and outputs the processing result to the user. When the processing result does not meet the user's expectations, the user device receives user feedback and performs Fig.17 and / or Fig.18 The process realizes the update of the first knowledge graph.

[0265] The above mainly introduces the solution provided in the embodiment of the present application from the perspective of method.

[0266] In order to achieve the above functions, an embodiment of the present application also provides an image processing device, which includes one or more units / modules, and the one or more units / modules are used to execute the image processing method and / or the method for constructing a rule base / first knowledge graph described above.

[0267] In an example, refer to Fig. 20 , Fig. 20 FIG. 2 is a schematic diagram showing the structure of an image processing device provided in an embodiment of the present application. The image processing device 2000 is used to execute the image processing method described above, for example, to execute Figure 4 , Figure 5 , Fig.11 as well as Figures 14 to 18 The image processing apparatus 2000 may include an acquisition unit 2001 , a determination unit 2002 and a processing unit 2003 .

[0268] The acquisition unit 2001 is used to acquire the image to be processed. The determination unit 2002 is used to determine the target object in the image to be processed according to the image to be processed and the rule base. The processing unit 2003 is used to perform a target operation on the target object in the image to be processed. The target object is the object on which the target operation is to be performed in the image to be processed. The rule base is extracted based on the image in the data set. For the first rule in the rule base, the first rule is any rule in the rule base, and the first rule is extracted based on the first image in the data set, and the first rule is used to indicate the object marked with the target label in the first image under the condition that the objects included in the first image and the relationship between the objects. The data set includes multiple images, each of the multiple images contains an object marked with the target label, and the object marked with the target label is the object to perform the target operation.

[0269] As an example, combining Fig.14 , the acquisition unit 2001 can be used to execute step 201, the determination unit 2002 can be used to execute step 202, and the processing unit 2003 can be used to execute step 203.

[0270] Optionally, when the target label is a label indicating privacy, the target operation is a desensitization operation.

[0271] Optionally, when the target tag is a tag that the user is particularly concerned about, the target operation is a highlighting operation.

[0272] Optionally, the rules in the rule base are represented by directed paths in the first knowledge graph, and the first knowledge graph is constructed based on multiple images. The directed paths in the first knowledge graph include nodes and directed edges. The nodes of the directed paths represent objects contained in the multiple images. The directed edges of the directed paths are used to describe the relationship between the objects contained in the multiple images. The last node in the direction indicated by the directed edge in the directed path is a label node, and the label node is used to record the object represented by at least one node on the directed path, and the objects represented by at least one node are all objects marked with target labels in the multiple images.

[0273] Optionally, the processing unit 2003 is further used to construct a second knowledge graph based on the image to be processed, and to query the target path including nodes and directed edges in the second knowledge graph in the first knowledge graph. The nodes in the second knowledge graph represent objects contained in the image to be processed, the directed edges in the second knowledge graph are used to describe the relationship between the objects contained in the image to be processed, and the target path is one of multiple directed paths. The determination unit 2002 is specifically used to determine the object recorded by the label node of the target path as the target object.

[0274] As an example, combining Fig.15 The processing unit 2003 can be used to execute steps 2021 to 2022, and the determination unit 2002 can be used to execute step 2023.

[0275] Optionally, the processing unit 2003 is further configured to, after determining the target object in the image to be processed according to the image to be processed and the rule base, update the rule base according to the image to be processed in response to a first indication fed back by the user, wherein the first indication is used to indicate that a target operation is performed on other objects in the image to be processed that are different from the target object.

[0276] Optionally, when the first knowledge graph is a knowledge graph constructed based on multiple images, and the rules in the rule base are represented by directed paths in the first knowledge graph, the processing unit 2003 is specifically used to delete the target path in the first knowledge graph, and, according to the image to be processed and the first indication, add a directed path in the first knowledge graph. Among them, the nodes on the target path used to represent the object and the directed edges used to describe the relationship between the objects exist in the second knowledge graph, and the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed based on the image to be processed; the label node on the newly added directed path is used to record other objects.

[0277] As an example, combining Fig.17 , the processing unit 2003 can be used to execute S21 to S25. Fig.18 , the processing unit 2003 can be used to execute S31~S34.

[0278] Optionally, the image processing apparatus 2000 is applied to a terminal device, and the terminal device is deployed with a rule library, and the acquisition unit 2001 is specifically used to acquire the to-be-processed image stored locally in the terminal device or sent to the terminal by other devices.

[0279] As an example, combining Fig.14 , the acquisition unit 2001 can be used to execute step 201.

[0280] Optionally, the image processing apparatus 2000 is applied to an application server, and the application server is deployed with a rule library, and the acquisition unit 2001 is specifically used to receive an image to be processed uploaded by a terminal device.

[0281] As an example, combining Fig.14 , the acquisition unit 2001 can be used to execute step 201.

[0282] Optionally, the processing unit 2003 is also used to construct multiple knowledge graphs based on multiple images before determining the target object in the image to be processed based on the image to be processed and the rule base; and add a label node at the end of the directed path containing preset nodes in the multiple knowledge graphs to obtain a first knowledge graph. Wherein, the number of multiple knowledge graphs is the same as the number of images in the data set; the multiple knowledge graphs include multiple directed paths containing preset nodes, wherein the nodes on the directed paths represent the objects contained in the multiple images, the directed edges in the directed paths are used to describe the relationship between the objects contained in the multiple images, the preset nodes contained in the directed paths are nodes located on the directed paths and used to represent the objects marked with target labels in the multiple images, the tail end of the directed path is the end where the end point of the direction indicated by the directed edge of the directed path is located, and the label node added at the tail end of the directed path is used to record the object represented by the preset node on the directed path.

[0283] As an example, combining Figure 5 , the processing unit 2003 can be used to execute steps 104 to 105.

[0284] Optionally, the processing unit 2003 is further used to add a label node to the tail end of a directed path containing a preset node in multiple knowledge graphs, merge the directed paths with the label node added by the nodes representing the same object and the same type of object on the directed paths with the label node added, and obtain a first knowledge graph; and record the directed paths to which the merged nodes and merged directed edges in the first knowledge graph belong. Among them, the same type of object refers to an object of the same type, the merged node is a node obtained by merging multiple nodes belonging to different directed paths, and the merged directed edge is a directed edge obtained by merging multiple directed edges belonging to different directed paths.

[0285] As an example, combining Fig.11 , the processing unit 2003 can be used to execute steps 106 to 107.

[0286] Optionally, the rule library includes at least one of the following first to sixth rules: First rule: When the image includes an outdoor scene containing people, people wearing masks, and people including faces, the object marked with a privacy label in the image is a face; Second rule: When the image includes a bedroom / dormitory scene containing people, and people including faces, the object marked with a privacy label in the image is a bedroom / dormitory and a face; Third rule: When the image includes an office scene containing a screen, and the attribute of the screen is bright, the object marked with a privacy label in the image is the screen; Fourth rule: When the image includes an outdoor / garage scene containing a car, and the car includes a license plate, the object marked with a privacy label in the image is the license plate; Fifth rule: When the image is a chat screenshot and the chat screenshot contains a left avatar / left nickname, the object marked with a privacy label in the image is a left avatar / left nickname; Sixth rule: When the image includes a card containing an identity card identification number ID / name / age / address, the object marked with a privacy label in the image is ID / name / age / address.

[0287] For the detailed description of the above optional methods, please refer to the above method embodiments, which will not be repeated here. In addition, the explanation and description of the beneficial effects of any of the above image processing devices 2000 can refer to the above corresponding method embodiments, which will not be repeated here.

[0288] As an example, in combination with the following Fig. 22 The functions implemented by the determination unit 2002 and the processing unit 2003 in the image processing device 2000 can be realized by Fig. 22 Processor 2201 in the execution Fig. 22 The functions implemented by the acquisition unit 2001 in the image processing device 2000 can be realized by Fig. 22 Processor 2201 in the execution Fig. 22 2202 in the memory, or by Fig. 22 The communication interface 2203 is shown implemented.

[0289] In another example, take the target label as a privacy label and the target operation as a desensitization operation. Fig.21 , Fig.21 FIG. 2 is a schematic diagram showing the structure of another image processing device provided in an embodiment of the present application. The image processing device 2100 is used to execute the image processing method described above, for example, to execute Figure 4 , Figure 5 , Fig.11 as well as Figures 14 to 18 The method shown is used to realize the protection processing of privacy objects in the image to be processed.

[0290] like Fig.21As shown, the rule base or the first knowledge graph described above is deployed in the image processing device 2100. The image processing device 2100 also includes an image privacy detection module 2101, an image desensitization module 2102 and a user feedback module 2103. Among them, the image privacy detection module 2101 is used to detect privacy objects in the image to be processed based on the rule base or the first knowledge graph (implemented by step 202). The image desensitization module 2102 is used to desensitize the privacy objects detected in the image to be processed (implemented by step 203). The user feedback module 2103 is used to obtain user feedback on the image processing results, and update the rule base or the first knowledge graph according to the feedback (such as by Fig.17 and / or Fig.18 The update process is implemented as described above. The image processing result is the result of detecting the privacy object in the image to be processed by the image privacy detection module 2101 and desensitizing the privacy object in the image to be processed by the image desensitization module 2102.

[0291] It can be seen that compared with Figure 1 The image privacy detection framework provided by the related technology shown, the image processing device provided in the embodiment of the present application includes not only an image privacy detection module and an image desensitization module, but also a user feedback module, so that the method provided in the embodiment of the present application supports updating the rule base or the first knowledge graph based on personalized privacy rules based on user feedback, that is, the solution provided in the embodiment of the present application has the ability to automatically detect and update the rule base or the first knowledge graph.

[0292] Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0293] It should be noted that Fig. 20 or Fig.21 The division of modules / units in the above is schematic and is only a logical function division. There may be other division methods in actual implementation. For example, two or more functions may be integrated into one processing module. The above integrated modules may be implemented in the form of hardware or software function modules.

[0294] The present application embodiment provides a computing device, which is used to implement some or all of the functions in the image processing method provided in the present application embodiment, or to implement some or all of the functions in the method for building a rule base or a first knowledge graph provided in the present application embodiment. For example, the computing device is implemented as the image processing apparatus described above.

[0295] refer to Fig. 22 , Fig. 22 Schematic diagram of a computing device provided in an embodiment of the present application. Fig. 22 As shown, the computing device 2200 includes a processor 2201, a memory 2202, a communication interface 2203, and a bus 2204. The processor 2201, the memory 2202, and the communication interface 2203 are connected to each other through the bus 2204. The computing device 2200 also includes an input / output interface 2205, and the input / output interface 2205, the processor 2201, the memory 2202, and the communication interface 2203 are connected to each other through the bus 2204.

[0296] Processor 2201 may include a general processor and / or a dedicated hardware chip. A general processor may include: a central processing unit (CPU), a microprocessor or a graphics processing unit (GPU). The CPU is, for example, a single-core processor (single-CPU), or a multi-core processor (multi-CPU). A dedicated hardware chip is a hardware module for high-performance processing. Dedicated hardware chips include digital signal processors (DSP), data processors (DPU), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, neural processing units (NPU), tensor processing units (TPU), artificial intelligence (artificial intelligent) chips or network processors (NP). Processor 2201 may also be an integrated circuit chip with signal processing capabilities. During the implementation process, part or all of the functions of the method provided in the embodiment of the present application can be completed through the hardware integrated logic circuit in the processor 2201 or the instructions in the form of software.

[0297] The memory 2202 is used to store computer programs, and the computer programs include an operating system 2202a and executable codes (i.e., program instructions) 2202b. The memory 2202 is, for example, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, or other types of static storage devices that can store static information and instructions, and is also a random access memory (static RAM, SRAM), a dynamic random access memory (dynamic random access memory, DRAM), a synchronous dynamic random access memory (synchronous DRAM, SDRAM), a double data rate synchronous dynamic random access memory (double data rate SDRAM, DDRSDRAM), an enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), a synchronous link dynamic random access memory (synchlink DRAM, SLDRAM) or other types of dynamic storage devices that can store information and instructions, such as read-only optical disks or other optical disk storage, optical disk storage (including compressed optical disks, laser disks, optical disks, digital versatile disks, Blu-ray disks, etc.), disk storage media or other magnetic storage devices, or any other medium that can be used to carry or store the desired executable code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this. For example, the memory 2202 is used to store a rule base or a first knowledge spectrum, etc. The memory 2202, for example, exists independently and is connected to the processor 2201 through a bus 2204. Or the memory 2202 and the processor 2201 are integrated together. The memory 2202 can store executable code, and when the executable code stored in the memory 2202 is executed by the processor 2201, the processor 2201 is used to execute part or all of the functions of the method provided in the embodiment of the present application. Please refer to the relevant description in the aforementioned embodiment for the implementation method of the processor 2201 to execute the process. The memory 2202 may also include software modules and data required for other running processes such as an operating system.

[0298] The communication interface 2203 uses a transceiver module such as, but not limited to, a transceiver to achieve communication with other devices or communication networks. For example, the communication interface 2203 can be any one or any combination of the following devices: a communication interface (such as an Ethernet interface), a wireless network card, and other devices with network access functions. The communication interface 2203 includes a receiving unit for receiving data / messages, and a sending unit for sending data / messages.

[0299] The bus 2204 is any type of communication bus for interconnecting the internal devices (e.g., memory 2202, processor 2201, communication interface 2203) of the computing device 2200. For example, a system bus. The embodiment of the present application takes the interconnection of the above-mentioned devices inside the computing device 2200 through the bus 2204 as an example. Optionally, the above-mentioned devices inside the computing device 2200 can also be connected to each other in communication with each other using other connection methods other than the bus 2204, for example, the above-mentioned devices inside the computing device 2200 are interconnected through an internal logical interface.

[0300] The input and output interface 2205 is used to implement human-computer interaction between the user and the computing device 2200. For example, text interaction or voice interaction between the user and the computing device 2200 is implemented. Among them, the input and output interface 2205 includes an input interface for implementing the user to input information to the computing device 2200, and an output interface for implementing the computing device 2200 to output information to the user. As an example, the input interface includes but is not limited to a touch screen, a keyboard, a mouse or a microphone, etc., and the output interface includes but is not limited to a display screen, a speaker, etc. Among them, the touch screen, keyboard or mouse is used to input text / image information, the microphone is used to input voice information, the display screen is used to output text / image information, and the speaker is used to output voice information.

[0301] In one example, the output interface in the input / output interface 2205 is used to output the processing result of the image to be processed to the user. In another example, the input interface in the input / output interface 2205 is used to receive feedback from the user based on the processing result of the image to be processed.

[0302] It should be noted that the above-mentioned multiple devices can be respectively arranged on independent chips, or at least partially or completely arranged on the same chip. Whether to independently arrange each device on different chips or to integrate and arrange it on one or more chips often depends on the needs of product design. The embodiments of the present application do not limit the specific implementation form of the above-mentioned devices. The descriptions of the processes corresponding to the above-mentioned figures have different focuses. For the parts not described in detail in a certain process, please refer to the relevant descriptions of other processes.

[0303] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. The computer program product providing the program development platform includes one or more computer instructions, and when these computer program instructions are loaded and executed on the computing device 2200, all or part of the functions of the image processing method or the method for constructing a rule base / first knowledge graph provided in the embodiments of the present application are implemented in whole or in part.

[0304] Furthermore, computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, computer instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium stores computer program instructions that provide a program development platform.

[0305] An embodiment of the present application also provides a computer-readable storage medium, which is a non-volatile computer-readable storage medium, and the computer-readable storage medium includes computer program instructions. When the computer program instructions are executed by a computing device, a computer system or a processor, the computing device, the computer system or the processor executes the image processing method and / or the method for constructing a rule base / first knowledge graph as provided in the embodiment of the present application.

[0306] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed by a computing device, a computer system or a processor, enables the computing device, the computer system or the processor to implement the image processing method and / or the method for constructing a rule base / first knowledge graph provided in the embodiment of the present application.

[0307] Among them, the computer system is a system with computing processing capabilities. The computer system generally includes a processor and a memory, and the processor is used to call and run instructions stored in the memory from the memory so that the computer system can implement the image processing method and / or the method for constructing a rule base / first knowledge graph described above. Optionally, the computer system may also include at least one of an input interface or an output interface. The processor, memory, input interface, and output interface of the computer system are connected through an internal connection path.

[0308] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0309] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0310] The embodiment of the present application also provides a chip, which includes a processor for running program instructions or codes, and the chip or a device including the chip can be used to execute the image processing method and / or the method for constructing a rule base / first knowledge graph provided in the embodiment of the present application. Exemplarily, the chip also includes: an input interface, an output interface, and a memory. Among them, the input interface, output interface, processor, and memory of the chip are connected through the internal connection path of the chip, the memory in the chip is used to store the program instructions or codes run by the processor, and the input interface and output interface of the chip are used for the connection and communication between the chip and other chips or devices.

[0311] In the embodiments of the present application, the terms "first", "second" and "third" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance. The term "at least one" means one or more, and the term "plurality" means at least two, unless otherwise expressly defined.

[0312] The term "and / or" in this application is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0313] It should be understood that the terms used in the description of the various examples herein are only for describing specific examples and are not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0314] It should be understood that determining B based on A does not mean determining B only based on A. B can also be determined based on A and / or other information.

[0315] It should be understood that the term “comprise” (also known as “includes”, “including”, “comprises” and / or “comprising”) when used in this specification specifies the presence of stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0316] It should also be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0317] The above description is only an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present application shall be included in the protection scope of the present application.

[0318] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image processing method, characterized in that: include: Get the image to be processed; Determine a target object in the image to be processed according to the image to be processed and a rule library, wherein the target object is an object on which a target operation is to be performed in the image to be processed; The rule base is extracted based on images in a data set, the data set includes multiple images, each of the multiple images includes an object marked with a target label, and the object marked with the target label is an object for performing the target operation; for a first rule in the rule base, the first rule is any rule in the rule base, the first rule is extracted based on a first image in the data set, and the first rule is used to indicate an object marked with the target label in the first image under the condition that the first image includes objects and relationships between objects; The target operation is performed on the target object in the image to be processed.

2. The method according to claim 1, characterized in that When the target tag is a tag representing privacy, the target operation is a desensitization operation.

3. The method according to claim 1, characterized in that When the target tag is a tag that the user particularly pays attention to, the target operation is a highlighting operation.

4. The method according to any one of claims 1 to 3, characterized in that The rules in the rule base are represented by directed paths in a first knowledge graph, which is constructed based on the multiple images. The directed path includes nodes and directed edges. The nodes of the directed path represent objects contained in the multiple images, and the directed edges of the directed path are used to describe the relationship between the objects contained in the multiple images. The last node in the direction indicated by the directed edge in the directed path is a label node, and the label node is used to record the object represented by at least one node on the directed path, and the objects represented by the at least one node are all objects marked with the target label in the multiple images.

5. The method according to claim 4, characterized in that The step of determining the target object in the image to be processed according to the image to be processed and the rule base includes: Constructing a second knowledge graph according to the image to be processed, wherein the nodes in the second knowledge graph represent objects contained in the image to be processed, and the directed edges in the second knowledge graph are used to describe the relationship between the objects contained in the image to be processed; Querying the first knowledge graph for a target path including nodes and directed edges in the second knowledge graph, wherein the target path is one of the multiple directed paths; The object recorded in the label node of the target path is determined as the target object.

6. The method according to any one of claims 1 to 5, characterized in that After determining the target object in the image to be processed according to the image to be processed and the rule base, the method further includes: In response to a first indication fed back by a user, the rule base is updated according to the image to be processed; wherein the first indication is used to instruct to perform the target operation on other objects in the image to be processed that are different from the target object.

7. The method according to claim 6, characterized in that When the first knowledge graph is a knowledge graph constructed according to the plurality of images, and the rules in the rule base are represented by directed paths in the first knowledge graph, updating the rule base according to the image to be processed includes: Delete the target path in the first knowledge graph; wherein the nodes on the target path used to represent objects and the directed edges used to describe the relationship between objects exist in the second knowledge graph, and the object recorded by the label node on the target path is the target object, and the second knowledge graph is a knowledge graph constructed according to the image to be processed; According to the image to be processed and the first indication, a directed path is added in the first knowledge graph, and the label nodes on the newly added directed path are used to record the other objects.

8. The method according to any one of claims 1 to 7, characterized in that The method is executed by a terminal device, the terminal device is deployed with the rule base, and the acquiring of the image to be processed includes: The image to be processed is obtained and stored locally on the terminal device or sent to the terminal by other devices.

9. The method according to any one of claims 1 to 7, characterized in that: The method is executed by an application server, the application server is deployed with the rule base, and the obtaining of the image to be processed includes: The image to be processed uploaded by the terminal device is received.

10. The method according to claim 4 or 5, characterized in that Before determining the target object in the image to be processed according to the image to be processed and the rule base, the method further includes: Constructing a plurality of knowledge graphs according to the plurality of images, wherein the number of the plurality of knowledge graphs is the same as the number of images in the data set; A label node is added to the tail end of the directed path containing preset nodes in the multiple knowledge graphs to obtain the first knowledge graph; the multiple knowledge graphs include multiple directed paths containing preset nodes, wherein the nodes on the directed paths represent objects contained in the multiple images, and the directed edges in the directed paths are used to describe the relationship between the objects contained in the multiple images, and the preset nodes contained in the directed paths are nodes located on the directed paths and used to represent objects in the multiple images marked with the target label; the tail end of the directed path is the end of the direction indicated by the directed edges of the directed path, and the label node added at the tail end of the directed path is used to record the objects represented by the preset nodes on the directed path.

11. The method according to claim 10, characterized in that After adding a label node to the tail end of the directed path containing the preset node in the plurality of knowledge graphs, the method further includes: By merging the nodes representing the same object and the same type of objects on the directed path to which the label node is added, the directed path to which the label node is added is obtained; wherein the same type of objects refers to objects of the same type; Record the merged nodes and directed paths to which the merged directed edges in the first knowledge graph belong, wherein the merged node is a node formed by merging multiple nodes belonging to different directed paths, and the merged directed edge is a directed edge formed by merging multiple directed edges belonging to different directed paths.

12. The method according to any one of claims 1 to 11, characterized in that The rule base includes at least one of the following first to sixth rules: The first rule: when the image includes an outdoor scene containing a person, a person wearing a mask, or a person including a face, the object marked with a privacy label in the image is a face; Second rule: If the image includes a bedroom / dormitory scene containing a person, and the person includes a face, the objects marked with privacy labels in the image are bedrooms / dormitories and faces; Rule 3: When the image includes an office scene containing a screen and the screen is bright, the object marked with a privacy label in the image is the screen; Fourth rule: If the image includes an outdoor / garage scene containing a car, and the car contains a license plate, the object marked with a privacy label in the image is the license plate; Rule 5: If the image is a chat screenshot and the chat screenshot contains a left avatar / left nickname, the object marked with a privacy label in the image is the left avatar / left nickname; Rule 6: When the image includes the card ID number / name / age / address, the object marked with a privacy label in the image is ID / name / age / address.

13. An image processing device, characterized in that: include: One or more units / modules, wherein the one or more units / modules are configured to execute the method according to any one of claims 1 to 12.

14. An image processing device, characterized in that: include: A memory, a communication interface and one or more processors, wherein the one or more processors receive or send data through the communication interface, and the one or more processors are configured to read program instructions stored in the memory to execute the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The method comprises computer program instructions. When the computer program instructions are executed by a computing device or a processor, the computing device or the processor performs the method according to any one of claims 1 to 12.

16. A computer program product comprising instructions, characterized in that When the instructions are executed by a computing device or a processor, the computing device or the processor is caused to perform the method according to any one of claims 1 to 12.

Images