Method, apparatus, medium and program product for processing image data

Abstract

The invention provides a method and device for processing image data, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that multi-level label information corresponding to a to-be-processed image is acquired, the multi-level label information comprises multiple labels used for describing target elements contained in the image, the multiple labels have a hierarchical relation, and a label tree structure is formed; the multi-level label information is converted into an image semantic graph, the image semantic graph comprises nodes and edges, and the nodes in the image semantic graph are in one-to-one correspondence with labels contained in the multi-level label information; matching the image semantic graph with a mode sub-graph in a decision rule base by using a graph structure matching algorithm to obtain a mode sub-graph matched with the image semantic graph, the mode sub-graph comprising a decision result; and generating a content identification result corresponding to the to-be-processed image according to the matched mode sub-graph. According to the image content analysis method and device, image content analysis fused with multi-granularity hierarchical semantic understanding is achieved through the graph structure matching algorithm, deep semantic understanding of complex scenes is achieved, the problem that semantic understanding is insufficient in the prior art is solved, the efficiency and accuracy of image content analysis are improved, and computing resources and labor cost are saved.

Description

Methods, apparatuses, media, and programs for processing image data Technical Field

[0001] This application relates to the field of computer technology, and more particularly to a method, apparatus, electronic device, computer-readable medium, and computer program product for processing image data. Background Technology

[0002] Image content recognition technology primarily relies on computer vision models to identify and classify image content. With the continuous advancement of deep learning technology, existing image content recognition solutions typically use models based on visual language models, such as CLIP, which can generate descriptive or labeled semantic information to provide a basis for risk prediction. Alternatively, they may rely on fixed label sets for identification and risk prediction.

[0003] However, existing solutions are still insufficient in characterizing the hierarchical relationship between tags and contextual associations, and models that rely on fixed tag sets may experience a decline in recognition ability when faced with novel or variant content. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, computer-readable medium, and computer program product for processing image data.

[0005] One aspect of this application provides a method for processing image data, wherein the method includes:

[0006] Obtain multi-level label information corresponding to the image to be processed. The multi-level label information includes multiple labels for describing the target elements contained in the image. The multiple labels have a hierarchical relationship and form a label tree structure.

[0007] The multi-level label information is converted into an image semantic graph, which contains nodes and edges. The nodes in the image semantic graph correspond one-to-one with the labels contained in the multi-level label information.

[0008] The image semantic graph is matched with the pattern subgraph in the decision rule base using a graph structure matching algorithm to obtain a pattern subgraph that matches the image semantic graph. The pattern subgraph contains the decision result.

[0009] Based on the matched pattern subgraph, the content recognition result corresponding to the image to be processed is generated.

[0010] In one aspect, this application provides an apparatus for processing image data, wherein the apparatus comprises:

[0011] The tag information acquisition module is used to acquire multi-level tag information corresponding to the image to be processed. The multi-level tag information includes multiple tags for describing the target elements contained in the image. The multiple tags have a hierarchical relationship and form a tag tree structure.

[0012] The semantic graph conversion module is used to convert the multi-level label information into an image semantic graph, which contains nodes and edges, and the nodes in the image semantic graph correspond one-to-one with the labels contained in the multi-level label information.

[0013] The graph structure matching module is used to match the image semantic graph with the pattern subgraph in the decision rule base using a graph structure matching algorithm to obtain a pattern subgraph that matches the image semantic graph, and the pattern subgraph contains the decision result;

[0014] The recognition result generation module is used to generate the content recognition result corresponding to the image to be processed based on the matched pattern subgraph.

[0015] In another aspect of this application, an electronic device is provided, the electronic device comprising:

[0016] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the methods of the embodiments of this application.

[0017] In another aspect, this application provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor to implement the methods of the embodiments of this application.

[0018] Another aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the methods of embodiments of this application.

[0019] The solution provided in this application adopts a graph structure matching algorithm to achieve image content analysis that integrates multi-granular hierarchical semantic understanding, realizing a deep semantic understanding of complex scenes, solving the problem of insufficient semantic understanding in the prior art, and improving the efficiency and accuracy of image content analysis. In the scenario of risk prediction of image content, this embodiment can accurately identify high-risk scenes and avoid misjudgment, reducing the possibility of false positives and false negatives. This application embodiment can display the successfully matched pattern subgraph and the corresponding nodes and edges in the image semantic graph, making the decision-making process of image content analysis transparent and traceable. By updating the identifiable labels and their descriptions, this application embodiment can adaptively expand its recognition capabilities, thereby continuously learning and evolving to use the constantly changing target elements that need to be identified, further improving the accuracy of image content analysis. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0022] Figure 1 shows a flowchart illustrating a method for processing image data according to an embodiment of this application;

[0023] Figure 2 illustrates a flowchart of an exemplary image content analysis according to an embodiment of this application;

[0024] Figure 3 shows a schematic diagram of a device for processing image data according to an embodiment of this application;

[0025] Figure 4 shows a schematic diagram of the structure of a device suitable for implementing the scheme in the embodiments of this application.

[0026] The same or similar reference numerals in the accompanying drawings represent the same or similar parts. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] In a typical configuration of this application, the terminal and the service network devices each include one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0029] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0030] Computer-readable media include permanent and non-permanent, removable and non-removable media, which can store information by any method or technology. Information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, read-only optical disc (CD-ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0031] Figure 1 shows a flowchart of a method for processing image data according to an embodiment of this application. The method includes at least steps S101, S102, S103, and S104.

[0032] In practical scenarios, the execution entity of this method can be a network device or an application running on a network device. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer clusters, and can be used to implement some processing functions when setting an alarm. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a loosely coupled set of computers.

[0033] Referring to Figure 1, in step S101, multi-level label information corresponding to the image to be processed is obtained. The multi-level label information includes multiple labels for describing the target elements contained in the image. The multiple labels have a hierarchical relationship and form a label tree structure.

[0034] The images to be processed include various types of image data, such as user-uploaded social media shared images, publicly visible advertising posters, news illustrations, e-commerce product images, or sample images used for model training or risk assessment. The images to be processed may contain people, objects, scenes, text, or combinations thereof.

[0035] It should be noted that the image data involved in the embodiments of this application were obtained with the consent or permission of the relevant parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0036] A label tree is a structured representation method used to describe in detail the labels contained in an image. Each node in a label tree represents a label, and the connections (edges) between nodes represent the relationships between these labels.

[0037] According to one embodiment, the label tree of this application is a multi-granularity hierarchical label tree. Here, "multi-granularity" refers to the ability to identify and distinguish different levels of detail in an image during image recognition operations. The method of this embodiment can identify features at various levels in an image, from very specific (fine-grained) objects or elements to more abstract (coarse-grained) categories.

[0038] Optionally, when performing risk prediction on an image, the label is used to describe risk elements contained in the image, such as people or objects that are not suitable to be present.

[0039] According to one embodiment, in step S101, the image to be processed is input into the label recognition model to perform label recognition processing, thereby obtaining the corresponding multi-level label information.

[0040] The label recognition model includes various models that can identify and analyze images and output the label information contained in the images. The model uses deep learning technology to identify various elements in the image, such as objects, scenes, and activities, and assigns one or more labels to each element.

[0041] Optionally, the label recognition model outputs multiple labels contained in the image, and the method constructs a hierarchical relationship based on the multiple labels output by the model to obtain multi-level label information corresponding to the image. The process of constructing the hierarchical relationship includes:

[0042] The relationships between the tags identified by the tag recognition model are analyzed to determine tags with hierarchical relationships. These hierarchical relationships include, but are not limited to, subordinate relationships, classification relationships, association relationships, and attribute relationships.

[0043] Next, for tags with hierarchical relationships, a tag hierarchy structure is constructed, which includes parent nodes and child nodes. Optionally, the parent node represents a broader category, and the child node represents a more specific instance or subcategory.

[0044] Optionally, the label hierarchy can be integrated to form a label tree. Specifically, the hierarchical relationships, such as dependency or classification relationships, of all label pairs identified through analysis are integrated to ensure no relationships are omitted or conflicting. Then, based on the integrated hierarchical relationships, a tree structure is constructed to form the label tree. For example, a directed acyclic graph (DAG) can be constructed, where each node represents a label and each directed edge represents a dependency relationship.

[0045] Optionally, the label recognition model can directly output a label tree for the image to be processed. For example, the RAM++ (Recognize Anything Model Plus Plus) model can be used as the label recognition model. The RAM++ model can output a hierarchical label tree that not only contains the basic elements of the image but also reflects the complex relationships between the elements.

[0046] It should be noted that those skilled in the art should be familiar with the fact that, in addition to the RAM++ model, the label recognition model may also include Semantic-SAM, PixelRefer, PAM (Perceive Anything Model), or UFO models, and those skilled in the art can select the appropriate label recognition model based on actual needs.

[0047] In step S102, the multi-level label information is converted into a corresponding image semantic graph. The image semantic graph contains nodes and edges, and each node in the image semantic graph corresponds one-to-one with a label contained in the multi-level label information.

[0048] The image semantic graph is a graph structure containing nodes and edges, obtained based on multi-level label information. Here, image semantics refers to the semantic relationships within the image content. This application embodiment constructs an image semantic graph to describe the target elements in the image to be processed and the relationships between them in a structured manner.

[0049] The image semantic graph of this application embodiment is denoted as G_img = (V_img, E_img), where the node V_img represents all the labels identified in the image. Each node may contain attribute label text (which may be represented as v.name) and recognition confidence (which may be represented as v.confidence).

[0050] Here, edge E_img represents the semantic relationship between nodes, including inheritance edges and co-occurrence edges. Inheritance edges are directed edges, representing an "is-a" relationship. For example, the node "Golden Retriever" has an inheritance edge pointing to the node "Dog". Co-occurrence edges are undirected edges, indicating that two tags that do not belong to the same direct inheritance chain appear simultaneously in the same visual scene. For example, a co-occurrence edge is established between the nodes "Swimsuit" and "Beach".

[0051] According to one embodiment, the process of converting multi-level label information into an image semantic graph includes steps S1021 to S1023.

[0052] In step S1021, an initial image semantic map is created.

[0053] Specifically, an empty image semantic graph G_img is created, and the node set V_img and edge set E_img are initialized to establish the basic structure for the transformation process.

[0054] In step S1022, each label in the multi-level label information is converted into a corresponding node in the image semantic graph.

[0055] Optionally, for the label tree, each label tree node is transformed into a corresponding node in the image semantic graph by traversing the label tree and an attribute is assigned, which may include label text or recognition confidence.

[0056] In step S1023, based on the various labels and the relationships between the labels contained in the multi-level label information, inherited edges and co-occurrence edges are constructed in the image semantic graph accordingly.

[0057] Specifically, inheritance edges are constructed in the image semantic graph according to the parent-child node relationship in the multi-level label information; co-occurrence edges are constructed in the image semantic graph for labels that appear simultaneously in the same visual scene.

[0058] Optionally, the weights of co-occurrence edges can be initialized to 1. Alternatively, the initial weights of co-occurrence edges can be calculated based on the spatial distance between the two labels in the image. For example, the initial weights can be calculated in an inverse proportional manner, such that the closer the distance, the higher the weight.

[0059] The following description continues with reference to Figure 1. In step S103, a graph structure matching algorithm is used to match the image semantic graph with the pattern subgraph in the decision rule base to obtain a pattern subgraph that matches the image semantic graph.

[0060] The graph structure matching algorithm is used to identify the correspondence between nodes and edges in the semantic graph of an image and pattern subgraphs in the rule base, thereby determining whether the image to be processed matches a specific rule in the decision rule base.

[0061] The decision rule base is a collection containing multiple specific patterns, each pattern being defined by a set of nodes and edges, representing specific decision features or behaviors.

[0062] Furthermore, the decision rule base in this application defines a pattern subgraph. The pattern subgraph contains the decision result and describes the semantic patterns that must be satisfied to trigger the decision (such as identifying inappropriate items or behaviors in an image). The pattern subgraph guides the system on how to make decisions based on labels and relationships in the image semantic graph through predefined rules and conditions. For example, the pattern subgraph can define that in certain scenarios, the simultaneous appearance of a specific item and a specific person (such as a minor) is not allowed; such co-occurrence will trigger a corresponding processing decision, such as marking the image as risky.

[0063] In one embodiment, the pattern subgraph (represented as G_pat) is a rule defined in the decision rule base, denoted as G_pat = (V_pat, E_pat, Action). Here, node V_pat represents the semantic concept required by the rule, and constraints can be defined, such as the name must be an exact match, the name belongs to a certain semantic set, or the name is a subclass of a certain parent class. Edge E_pat represents the relationship between nodes required by the rule, and can be an inheritance edge or a co-occurrence edge. Action is the operation performed when the image semantic graph G_img matches G_pat, such as marking it as a violation.

[0064] According to one embodiment, step S103, the matching process based on the graph structure matching algorithm, includes steps S1031 to S1033.

[0065] Optionally, before performing matching based on the obtained image semantic graph, all pattern subgraphs in the decision rule base are pre-compiled and indexed. Optionally, the index includes indexing by root node name or key node name.

[0066] In step S1031, the nodes of the obtained image semantic graph are filtered in the decision rule base to obtain multiple pattern subgraphs related to the nodes as candidate rule sets. Optionally, based on the nodes of the obtained image semantic graph, a pre-established index is used for retrieval to obtain multiple pattern subgraphs related to the nodes as candidate rule sets to achieve fast retrieval.

[0067] Optionally, for a given image semantic graph (G_img), all possible matching pattern subgraphs are selected as candidate rule sets from the decision rule base based on the node names (labels) it contains. This step quickly selects all possible matching pattern subgraphs as candidate rule sets. For example, if the image semantic graph G_img contains "car", then all rules containing "car" or related labels such as "vehicle" will be selected. The purpose is to narrow down the matching range, thereby focusing on processing the rules most likely to match the image semantic graph.

[0068] In step S1032, the pattern subgraphs in the candidate rule set are subjected to secondary screening according to the preset matching constraints, and one or more pattern subgraphs that meet the matching constraints are selected as the matching pattern subgraphs.

[0069] The matching constraints include node name matching, confidence threshold, edge type (inheritance / co-occurrence), etc.

[0070] According to one embodiment, in step S1032, a matching pattern subgraph is obtained by performing node mapping and edge constraint verification on each candidate pattern subgraph.

[0071] Specifically, for each candidate pattern subgraph G_pat, the following matching process is performed:

[0072] Node mapping: Find a mapping function M: V_pat -> V_img from nodes in G_pat to nodes in G_img, such that for each node u in G_pat, its corresponding node M(u) in G_img satisfies all the constraints of u (such as name matching, confidence threshold, etc.).

[0073] Edge constraint verification: After finding the node mapping M, verify whether each edge (u1, u2) in G_pat has a corresponding edge (M(u1), M(u2)) in G_img, and whether the edge types (inheritance / co-occurrence) are consistent. For inherited edges, the directed path of the inheritance relationship must be strictly matched; for co-occurrence edges, it is only necessary to confirm that there is a co-occurrence edge between M(u1) and M(u2).

[0074] Optionally, if multiple matching pattern subgraphs are obtained, one is selected as the matching pattern subgraph based on a predetermined strategy.

[0075] For example, in a scenario where risk is predicted for an image, the pattern subgraph with the highest risk level is selected as the matching pattern subgraph. If a G_img successfully matches multiple pattern subgraphs G_pat, the selection is based on the priority of the rules; for example, the priority of the "violation" rule is higher than that of the "requires review" rule.

[0076] In step S104, the content recognition result corresponding to the image to be processed is generated based on the matched pattern subgraph.

[0077] The content recognition result includes successfully matched pattern subgraphs and the corresponding decision rules for those subgraphs. Optionally, the result may also include identifiers of specific nodes and edges in the image semantic graph that are associated with the matched pattern subgraphs, thereby clearly showing which pattern subgraphs successfully match the image semantic graph and how these matches are associated with specific elements (nodes and edges) in the image.

[0078] Optionally, when performing risk prediction on the image to be processed, the content recognition result includes a judgment result on whether the image to be processed is risky and information indicating the severity of the risk, such as risk level.

[0079] According to one embodiment, steps S102 to S104 are performed using a trained risk prediction model. The image label tree is input into the risk prediction model, which converts the image label tree into an image semantic graph. The model then uses a graph structure matching algorithm to match the image semantic graph with the pattern subgraph in the risk rule base and outputs the corresponding risk prediction result.

[0080] The risk prediction model includes, but is not limited to, deep neural networks and graph neural networks (GNNs). Deep neural networks utilize convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to process image and sequence data, extract features, and perform risk prediction. Graph neural networks (GNNs) are used to process graph-structured data, capable of capturing the complex relationships between nodes and edges in the semantic graph of an image, and are suitable for pattern subgraph matching and risk prediction.

[0081] According to one embodiment, the method further includes steps S105 to S107.

[0082] In step S105, if the image to be processed contains new target elements that cannot be described by existing label information, a multimodal large model is used to perform image content analysis on the image to be processed to obtain one or more new labels related to the new target elements.

[0083] The multimodal large model is used to synthesize data from different sources and types to identify target elements contained in an image. Optionally, the multimodal large model is used to analyze the image content, contextual information, etc., of the image to be processed.

[0084] Optionally, the multimodal large model includes, but is not limited to, at least one of the following:

[0085] 1) Visual recognition model: used to process image data and identify objects, scenes, and activities in images;

[0086] 2) Natural Language Processing (NLP) Models: Used to analyze text content or related descriptive text in images to understand context and semantics;

[0087] 3) Behavioral analysis model: used to analyze behavioral patterns in images, such as human movements and object movements, in order to identify unusual or inappropriate behaviors;

[0088] 4) Deep learning models: Utilize deep neural networks to learn complex features and patterns from large amounts of data to improve the accuracy of content recognition.

[0089] The models described above can be used individually or integrated to form a powerful multimodal analysis system. In this way, the large multimodal model can analyze images from multiple angles and levels, providing a more comprehensive risk assessment and thus improving the quality and efficiency of image content analysis.

[0090] In step S106, a large language model is used to generate descriptive information for the new tag.

[0091] Specifically, the large language model generates descriptive text related to new labels by analyzing and understanding the image content and contextual information provided by the multimodal large model.

[0092] For example, inputting "This image contains mosaic-like stripes that could be used by malicious users to obscure inappropriate content. Please describe this stripe pattern using one or more words" into a multimodal large model, the multimodal model will output the required label, "Moiré stripes." Then, inputting the label "Moiré stripes" into a large language model (LLM) and requesting a detailed description of the label will result in the following description: "Moiré stripes are new, visually distinct interference ripple patterns produced when two (or more) sets of periodic patterns (such as lines, grids, or dot matrices) overlap at a certain angle and frequency."

[0093] In step S107, the existing tag information is updated based on the new tag and its description. Based on the updated tag information, the tag of the message can be identified and processed in subsequent image content recognition.

[0094] Optionally, the label recognition model can be updated based on new labels and their descriptive information, enabling the model to recognize new labels and descriptions. This process involves inputting new labels and their descriptions into the label recognition model, which then encodes and updates features to enhance its ability to recognize this type of label. Based on this approach, through continuous learning and updating, the label recognition model can adapt to constantly changing risk environments.

[0095] According to the method of this application embodiment, image content analysis with multi-granular hierarchical semantic understanding is achieved by employing a graph structure matching algorithm. This enables deep semantic understanding of complex scenes, solves the problem of insufficient semantic understanding in the prior art, and improves the efficiency and accuracy of image content analysis. In scenarios where image content is used for risk prediction, this embodiment can accurately identify high-risk scenarios and avoid misjudgments, reducing the possibility of false positives and false negatives. This application embodiment can display successfully matched pattern subgraphs and corresponding nodes and edges in the image semantic graph, making the decision-making process of image content analysis transparent and traceable. By updating identifiable labels and their descriptions, this application embodiment can adaptively expand its recognition capabilities, thereby continuously learning and evolving to use constantly changing target elements that need to be identified, further improving the accuracy of image content analysis.

[0096] The method of this application embodiment will be described below with reference to an example.

[0097] Figure 2 illustrates a flowchart of an exemplary image content analysis according to an embodiment of this application. This example determines whether the image under test contains inappropriate content by performing image content recognition on the image under test.

[0098] Referring to Figure 2, for the image to be tested, perform the following steps:

[0099] The image to be tested is input into the multi-granularity recognition model (RAM++ model), which is used to perform in-depth analysis of the image content and generate a multi-granularity image label tree.

[0100] Next, the generated image label tree is fed into the model and strategy for evaluation. Specifically, based on preset rules and strategies, it is determined whether the image to be tested contains content that is inappropriate for display. This process is implemented using a "scene semantic understanding engine," which employs a multi-granularity recognition model capable of outputting hierarchical label trees as its foundation, using a preset rule base and a corresponding graph structure matching algorithm. This engine converts the label tree output by the RAM++ model into an "image semantic graph" rich in inheritance and co-occurrence relationships. By matching the "image semantic graph" with the "pattern subgraphs" in the rule base, a leap from isolated label matching to complete scene semantic understanding is achieved.

[0101] Next, if no pattern subgraph is matched, meaning the image under test does not contain inappropriate content, a missing image check is performed to determine if there are any potential missing images. Missing images are obtained through user feedback, reports, and other means.

[0102] Next, new label recognition and model updates are performed. This step is implemented through an "open set recognition engine," which addresses the weakness of multi-granularity recognition models in recognizing target elements of entirely new categories outside the training data, specifically for content in unknown categories. When the content to be recognized cannot be described by labels recognizable by the multi-granularity recognition model, this engine is activated to perform the following operations: First, for missed images, a large multimodal model is used to summarize labels related to the reasons for violations. Then, a large language model is used to generate descriptions for these new labels, updating the multi-granularity recognition model to incorporate the newly recognized labels and descriptions, enabling it to recognize these new labels and thus improving its ability to identify and process violations in the future.

[0103] The "scene semantic understanding engine" and "open set recognition engine" mentioned above in this example work together to form an image content recognition and processing system that combines deep semantic understanding with broad generalization recognition capabilities. It can not only process known image tags, but also continuously update the recognizable image tags through dynamic learning, thereby improving the overall image content analysis capability.

[0104] Figure 3 shows a schematic diagram of a device for processing image data according to an embodiment of this application. The device includes a label information acquisition module 101, a semantic graph conversion module 102, a graph structure matching module 103, and a recognition result generation module 104.

[0105] The tag information acquisition module 101 acquires multi-level tag information corresponding to the image to be processed. The multi-level tag information includes multiple tags for describing the target elements contained in the image. The multiple tags have a hierarchical relationship and form a tag tree structure.

[0106] The images to be processed include various types of image data, such as user-uploaded social media shared images, publicly visible advertising posters, news illustrations, e-commerce product images, or sample images used for model training or risk assessment. The images to be processed may contain people, objects, scenes, text, or combinations thereof.

[0107] It should be noted that the image data involved in the embodiments of this application were obtained with the consent or permission of the relevant parties, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0108] A label tree is a structured representation method used to describe in detail the labels contained in an image. Each node in a label tree represents a label, and the connections (edges) between nodes represent the relationships between these labels.

[0109] According to one embodiment, the label tree of this application is a multi-granularity hierarchical label tree. Here, "multi-granularity" refers to the ability to identify and distinguish different levels of detail in an image during image recognition operations. The method of this embodiment can identify features at various levels in an image, from very specific (fine-grained) objects or elements to more abstract (coarse-grained) categories.

[0110] Optionally, when performing risk prediction on an image, the label is used to describe risk elements contained in the image, such as people or objects that are not suitable to be present.

[0111] According to one embodiment, the label information acquisition module 101 inputs the image to be processed into the label recognition model to perform label recognition processing and obtain the corresponding multi-level label information.

[0112] The label recognition model includes various models that can identify and analyze images and output the label information contained in the images. The model uses deep learning technology to identify various elements in the image, such as objects, scenes, and activities, and assigns one or more labels to each element.

[0113] Optionally, the label recognition model outputs multiple labels contained in the image, and constructs a hierarchical relationship based on the multiple labels output by the model to obtain multi-level label information corresponding to the image. The process of constructing the hierarchical relationship includes:

[0114] The relationships between the tags identified by the tag recognition model are analyzed to determine tags with hierarchical relationships. These hierarchical relationships include, but are not limited to, subordinate relationships, classification relationships, association relationships, and attribute relationships.

[0115] Next, for tags with hierarchical relationships, a tag hierarchy structure is constructed, which includes parent nodes and child nodes. Optionally, the parent node represents a broader category, and the child node represents a more specific instance or subcategory.

[0116] Optionally, the label hierarchy can be integrated to form a label tree. Specifically, the hierarchical relationships, such as dependency or classification relationships, of all label pairs identified through analysis are integrated to ensure no relationships are omitted or conflicting. Then, based on the integrated hierarchical relationships, a tree structure is constructed to form the label tree. For example, a directed acyclic graph (DAG) can be constructed, where each node represents a label and each directed edge represents a dependency relationship.

[0117] Optionally, the label recognition model can directly output a label tree for the image to be processed. For example, using the RAM++ (Recognize Anything Model Plus Plus) model as the label recognition model, the RAM++ model can output a hierarchical label tree that not only contains the basic elements of the image but also reflects the complex relationships between the elements.

[0118] It should be noted that those skilled in the art should be familiar with the fact that, in addition to the RAM++ model, the label recognition model may also include Semantic-SAM, PixelRefer, PAM (Perceive Anything Model), or UFO models, and those skilled in the art can select the appropriate label recognition model based on actual needs.

[0119] The semantic graph conversion module 102 converts the multi-level label information into a corresponding image semantic graph. The image semantic graph contains nodes and edges, and each node in the image semantic graph corresponds one-to-one with a label contained in the multi-level label information.

[0120] The image semantic graph is a graph structure containing nodes and edges, obtained based on multi-level label information. Here, image semantics refers to the semantic relationships within the image content. This application embodiment constructs an image semantic graph to describe the target elements in the image to be processed and their relationships in a structured manner. The representation method of the image semantic graph has been described above and will not be repeated here.

[0121] According to one embodiment, the process of converting multi-level label information into an image semantic graph includes:

[0122] First, create an initial image semantic graph.

[0123] Specifically, an empty image semantic graph G_img is created, and the node set V_img and edge set E_img are initialized to establish the basic structure for the transformation process.

[0124] Next, each label in the multi-level label information is converted into a corresponding node in the image semantic graph.

[0125] Optionally, for the label tree, each label tree node is transformed into a corresponding node in the image semantic graph by traversing the label tree and an attribute is assigned, which may include label text or recognition confidence.

[0126] Next, based on the various labels and the relationships between them contained in the multi-level label information, inheritance edges and co-occurrence edges are constructed accordingly in the image semantic graph.

[0127] Specifically, inheritance edges are constructed in the image semantic graph according to the parent-child node relationship in the multi-level label information; co-occurrence edges are constructed in the image semantic graph for labels that appear simultaneously in the same visual scene.

[0128] Optionally, the weights of co-occurrence edges can be initialized to 1. Alternatively, the initial weights of co-occurrence edges can be calculated based on the spatial distance between the two labels in the image. For example, the initial weights can be calculated in an inverse proportional manner, such that the closer the distance, the higher the weight.

[0129] The graph structure matching module 103 uses a graph structure matching algorithm to match the image semantic graph with the pattern subgraphs in the decision rule base to obtain a pattern subgraph that matches the image semantic graph.

[0130] The graph structure matching algorithm is used to identify the correspondence between nodes and edges in the semantic graph of an image and pattern subgraphs in the rule base, thereby determining whether the image to be processed matches a specific rule in the decision rule base.

[0131] The decision rule base is a collection containing multiple specific patterns, each pattern being defined by a set of nodes and edges, representing specific decision features or behaviors.

[0132] Furthermore, the decision rule base in this application defines a pattern subgraph. The pattern subgraph contains the decision result and describes the semantic pattern that must be satisfied to trigger the decision. The content and representation methods of the pattern subgraph have been described above and will not be repeated here.

[0133] According to one embodiment, the matching process based on the graph structure matching algorithm includes:

[0134] Optionally, before performing matching based on the obtained image semantic graph, all pattern subgraphs in the decision rule base are pre-compiled and indexed. Optionally, the index includes indexing by root node name or key node name.

[0135] The decision rule base filters nodes based on the obtained image semantic graph, obtaining multiple pattern subgraphs related to each node as candidate rule sets. Optionally, based on the nodes of the obtained image semantic graph, a pre-established index is used for retrieval to obtain multiple pattern subgraphs related to each node as candidate rule sets, thereby achieving fast retrieval.

[0136] Optionally, for a given image semantic graph (G_img), all possible matching pattern subgraphs are selected as candidate rule sets from the decision rule base based on the node names (labels) it contains. This step quickly selects all possible matching pattern subgraphs as candidate rule sets.

[0137] Next, the pattern subgraphs in the candidate rule set are further filtered according to the preset matching constraints, and one or more pattern subgraphs that meet the matching constraints are selected as the matching pattern subgraphs.

[0138] The matching constraints include node name matching, confidence threshold, edge type (inheritance / co-occurrence), etc.

[0139] According to one embodiment, a matching pattern subgraph is obtained by performing node mapping and edge constraint verification on each candidate pattern subgraph.

[0140] Specifically, for each candidate pattern subgraph G_pat, the following matching process is performed:

[0141] Node mapping: Find a mapping function M: V_pat -> V_img from nodes in G_pat to nodes in G_img, such that for each node u in G_pat, its corresponding node M(u) in G_img satisfies all the constraints of u (such as name matching, confidence threshold, etc.).

[0142] Edge constraint verification: After finding the node mapping M, verify whether each edge (u1, u2) in G_pat has a corresponding edge (M(u1), M(u2)) in G_img, and whether the edge types (inheritance / co-occurrence) are consistent. For inherited edges, the directed path of the inheritance relationship must be strictly matched; for co-occurrence edges, it is only necessary to confirm that there is a co-occurrence edge between M(u1) and M(u2).

[0143] Optionally, if multiple matching pattern subgraphs are obtained, one is selected as the matching pattern subgraph based on a predetermined strategy.

[0144] For example, in a scenario where risk is predicted for an image, the pattern subgraph with the highest risk level is selected as the matching pattern subgraph. If a G_img successfully matches multiple pattern subgraphs G_pat, the selection is based on the priority of the rules; for example, the priority of the "violation" rule is higher than that of the "requires review" rule.

[0145] The recognition result generation module 104 generates the content recognition result corresponding to the image to be processed based on the matched pattern subgraph.

[0146] The content recognition result includes successfully matched pattern subgraphs and the corresponding decision rules for those subgraphs. Optionally, the result may also include identifiers of specific nodes and edges in the image semantic graph that are associated with the matched pattern subgraphs, thereby clearly showing which pattern subgraphs successfully match the image semantic graph and how these matches are associated with specific elements (nodes and edges) in the image.

[0147] Optionally, when performing risk prediction on the image to be processed, the content recognition result includes a judgment result on whether the image to be processed is risky and information indicating the severity of the risk, such as risk level.

[0148] According to one embodiment, a trained risk prediction model is used to perform the operations of the semantic graph conversion module 102, the graph structure matching module 103, and the recognition result generation module 104. The image label tree is input into the risk prediction model, which converts the image label tree into an image semantic graph, and uses a graph structure matching algorithm to match the image semantic graph with the pattern subgraph in the risk rule base, and outputs the corresponding risk prediction result.

[0149] The risk prediction model includes, but is not limited to, deep neural networks and graph neural networks (GNNs). Deep neural networks utilize convolutional neural networks (CNNs) or recurrent neural networks (RNNs) to process image and sequence data, extract features, and perform risk prediction. Graph neural networks (GNNs) are used to process graph-structured data, capable of capturing the complex relationships between nodes and edges in the semantic graph of an image, and are suitable for pattern subgraph matching and risk prediction.

[0150] According to one embodiment, the device further includes a content analysis module, a tag information generation module, and a tag information update module.

[0151] If the image to be processed contains new target elements that cannot be described by existing label information, the content analysis module uses a multimodal large model to perform image content analysis on the image to be processed, and obtains one or more new labels related to the new target elements.

[0152] The multimodal large model is used to synthesize data from different sources and types to identify target elements contained in an image. Optionally, the multimodal large model is used to analyze the image content, contextual information, etc., of the image to be processed.

[0153] Optionally, the multimodal large model includes, but is not limited to, at least one of the following:

[0154] 1) Visual recognition model: used to process image data and identify objects, scenes, and activities in images;

[0155] 2) Natural Language Processing (NLP) Models: Used to analyze text content or related descriptive text in images to understand context and semantics;

[0156] 3) Behavioral analysis model: used to analyze behavioral patterns in images, such as human movements and object movements, in order to identify unusual or inappropriate behaviors;

[0157] 4) Deep learning models: Utilize deep neural networks to learn complex features and patterns from large amounts of data to improve the accuracy of content recognition.

[0158] The models described above can be used individually or integrated to form a powerful multimodal analysis system. In this way, the large multimodal model can analyze images from multiple angles and levels, providing a more comprehensive risk assessment and thus improving the quality and efficiency of image content analysis.

[0159] The tag information generation module uses a large language model to generate descriptive information for the new tag.

[0160] Specifically, the large language model generates descriptive text related to new labels by analyzing and understanding the image content and contextual information provided by the multimodal large model.

[0161] The label information update module updates existing label information based on new labels and their descriptions. Based on the updated label information, the labels for that information can be identified and processed in subsequent image content recognition.

[0162] Optionally, the label information update module updates the label recognition model based on new labels and their descriptive information, enabling the label recognition model to recognize new labels and descriptions. This process involves inputting new labels and their descriptions into the label recognition model, which, through encoding and updating features, can add the ability to recognize this type of label. Based on this approach, through continuous learning and updating, the label recognition model can adapt to the ever-changing risk environment.

[0163] The apparatus according to the embodiments of this application achieves image content analysis by employing a graph structure matching algorithm, which integrates multi-granular hierarchical semantic understanding. This enables deep semantic understanding of complex scenes, solves the problem of insufficient semantic understanding in the prior art, and improves the efficiency and accuracy of image content analysis. In scenarios where image content is used for risk prediction, this embodiment can accurately identify high-risk scenarios and avoid misjudgments, reducing the possibility of false positives and false negatives. This embodiment can display successfully matched pattern subgraphs and corresponding nodes and edges in the image semantic graph, making the decision-making process of image content analysis transparent and traceable. By updating identifiable labels and their descriptions, this embodiment can adaptively expand its recognition capabilities, thereby continuously learning and evolving to use constantly changing target elements that need to be identified, further improving the accuracy of image content analysis.

[0164] Based on the same inventive concept, this application also provides an electronic device. The method corresponding to the electronic device can be the method for processing image data in the foregoing embodiments, and its problem-solving principle is similar to that method. The electronic device provided in this application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the foregoing embodiments of this application.

[0165] The electronic device can be a user device, or a device formed by integrating user devices and network devices through a network, or it can be an application running on the aforementioned devices. The user device includes, but is not limited to, various terminal devices such as computers, mobile phones, tablets, smartwatches, and wristbands. The network device includes, but is not limited to, network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, and can be used to implement some processing functions when setting an alarm clock. Here, the cloud consists of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a virtual computer composed of a group of loosely coupled computer sets.

[0166] The figure illustrates the structure of an apparatus suitable for implementing the methods and / or technical solutions in the embodiments of this application. The apparatus 1200 includes a Central Processing Unit (CPU) 1201, which can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1202 or a program loaded from a storage portion 1208 into a Random Access Memory (RAM) 1203. The RAM 1203 also stores various programs and data required for system operation. The CPU 1201, ROM 1202, and RAM 1203 are interconnected via a bus 1204. An Input / Output (I / O) interface 1205 is also connected to the bus 1204.

[0167] The following components are connected to I / O interface 1205: an input section 1206 including a keyboard, mouse, touchscreen, microphone, infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), liquid crystal display (LCD), LED display, OLED display, etc., and speakers, etc.; a storage section 1208 including one or more computer-readable media such as hard disk, optical disk, magnetic disk, semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0168] In particular, the methods and / or embodiments in this application can be implemented as computer software programs. For example, the embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, it performs the functions defined in the methods of this application.

[0169] Another embodiment of this application provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of this application described above.

[0170] Specifically, this embodiment may employ any combination of one or more computer-readable media. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0171] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0172] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0173] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0174] The flowcharts or block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-specific system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0175] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0176] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or page components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between devices or units through some interfaces, and may be electrical, mechanical, or other forms.

[0177] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0178] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in a combination of hardware and software functional units.

[0179] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute some steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

[0181] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices described in an apparatus may also be implemented by a single unit or device through software or hardware. Terms such as "first," "second," etc., are used to indicate names and do not indicate any specific order.

Claims

1. A method for processing image data, wherein, The method includes: acquiring multi-level label information corresponding to the image to be processed, wherein the multi-level label information contains multiple labels for describing target elements contained in the image, the multiple labels having a hierarchical relationship and forming a label tree structure; converting the multi-level label information into an image semantic graph, wherein the image semantic graph contains nodes and edges, and the nodes in the image semantic graph correspond one-to-one with the labels contained in the multi-level label information; using a graph structure matching algorithm to match the image semantic graph with a pattern subgraph in a decision rule base to obtain a pattern subgraph that matches the image semantic graph, the pattern subgraph containing a decision result; and generating a content recognition result corresponding to the image to be processed based on the matched pattern subgraph.

2. The method according to claim 1, wherein, The method further includes: if the image to be processed contains new target elements that cannot be described by existing label information, performing image content analysis on the image to be processed using a multimodal large model to obtain one or more new labels related to the new target elements; generating descriptive information of the new labels using a large language model; and updating existing label information based on the new labels and their descriptive information.

3. The method according to claim 1, wherein, The process of obtaining multi-level label information corresponding to the image to be processed includes: inputting the image to be processed into a label recognition model to perform label recognition processing, and obtaining the corresponding multi-level label information.

4. The method according to claim 3, wherein, The label recognition model outputs multiple labels in the image. The method constructs a label hierarchy relationship based on the multiple labels output by the model to obtain multi-level label information corresponding to the image. The process of constructing the hierarchy relationship includes: analyzing the relationship between the labels identified by the label recognition model and determining the labels with a hierarchy relationship; for the labels with a hierarchy relationship, constructing a label hierarchy structure, which includes parent nodes and child nodes.

5. The method according to claim 1, wherein, Converting the multi-level label information into a corresponding image semantic graph includes: creating an initial image semantic graph; converting each label in the multi-level label information into a corresponding node in the image semantic graph; and constructing inheritance edges and co-occurrence edges in the image semantic graph according to the labels contained in the multi-level label information and the relationships between the labels.

6. The method according to claim 1, wherein, The process of matching the image semantic graph with the pattern subgraphs in the decision rule base using a graph structure matching algorithm includes: filtering the nodes of the obtained image semantic graph in the decision rule base to obtain multiple pattern subgraphs related to the nodes as a candidate rule set; and performing a secondary filtering of the pattern subgraphs in the candidate rule set according to preset matching constraints, and selecting one or more pattern subgraphs that meet the matching constraints as the matching pattern subgraphs.

7. The method according to claim 6, wherein, The precise selection of pattern subgraphs in the candidate rule set based on preset matching constraints includes: obtaining matching pattern subgraphs by performing node mapping and edge constraint verification on each candidate pattern subgraph.

8. The method according to claim 6 or 7, wherein, include: All pattern subgraphs in the decision rule base are pre-compiled and indexed; wherein, the step of filtering based on the nodes of the obtained image semantic graph in the decision rule base to obtain multiple pattern subgraphs related to the nodes as candidate rule sets includes: based on the nodes of the obtained image semantic graph, using the pre-established index to perform retrieval to obtain multiple pattern subgraphs related to the nodes as candidate rule sets.

9. An apparatus for processing image data, wherein, The device includes: a tag information acquisition module, used to acquire multi-level tag information corresponding to the image to be processed, wherein the multi-level tag information contains multiple tags for describing target elements contained in the image, and the multiple tags have a hierarchical relationship to form a tag tree structure; a semantic graph conversion module, used to convert the multi-level tag information into an image semantic graph, wherein the image semantic graph contains nodes and edges, and the nodes in the image semantic graph correspond one-to-one with the tags contained in the multi-level tag information; a graph structure matching module, used to use a graph structure matching algorithm to match the image semantic graph with a pattern subgraph in a decision rule base to obtain a pattern subgraph that matches the image semantic graph, wherein the pattern subgraph contains a decision result; and a recognition result generation module, used to generate a content recognition result corresponding to the image to be processed based on the matched pattern subgraph.

10. An electronic device, the electronic device comprising: At least one processor; And a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 8.

11. A computer-readable medium having stored thereon computer program instructions that can be executed by a processor to implement the method as described in any one of claims 1 to 8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images