Scene graph generation method and device based on graph semantic understanding, equipment, storage medium and program product

Abstract

The invention provides a scene graph generation method based on graph semantic understanding, which can be used in the financial science and technology field or other fields, and comprises the following steps: obtaining an initial environment image, and carrying out feature extraction and target detection on the initial environment image to obtain an initial category feature, a spatial feature and a visual feature of the initial environment image; performing knowledge embedding on the knowledge vector representation of the initial category feature and the visual feature to obtain a target category feature; based on the target category features, the spatial features and the visual features, obtaining context information and performing relation classification through a preset classification model to generate an initial scene graph; performing graph embedding on the initial scene graph, and performing similarity calculation to obtain a similar scene graph sorting list corresponding to the initial scene graph; and displaying the similar scene graph sorting list, and determining a target scene graph corresponding to the initial environment image. The invention further provides a scene graph generation device and equipment based on graph semantic understanding, a storage medium and a program product.

Description

Technical Field

[0001] The present disclosure relates to the field of financial technology, and more specifically to a scene graph generation method, apparatus, device, storage medium and program product based on graph semantic understanding. Background Art

[0002] Traditional credit card promotion and marketing mainly adopts a relatively broad coverage strategy, targeting a wide range of target customer groups rather than precise positioning. As a result, most customers are less active and it is difficult to accurately capture customer needs. In the absence of effective data and algorithm support, it is difficult to determine when and where to take measures for which specific customer group to effectively activate potential customers and bring about performance growth.

[0003] With the rapid development of computer vision and deep learning technologies, scene graph generation, as a way to convert image content into structured information, can effectively describe objects in an image and their relationships. By generating scene graphs, customer needs can be captured and the success rate of credit card promotion can be improved. However, scene graph generation methods in the prior art often lack an in-depth understanding of image semantic information, resulting in deficiencies in the accuracy and richness of the generated scene graphs. Summary of the invention

[0004] In view of the above problems, the present disclosure provides a scene graph generation method, apparatus, device, storage medium and program product based on graph semantic understanding.

[0005] According to a first aspect of the present disclosure, a scene graph generation method based on graph semantic understanding is provided, including: acquiring an initial environment image, and performing feature extraction and target detection on the initial environment image to obtain initial category features, spatial features and visual features of the initial environment image; performing knowledge embedding of the knowledge vector representation of the initial category features and the visual features to obtain target category features; based on the target category features, spatial features and visual features, obtaining context information and performing relationship classification through a preset classification model to generate an initial scene graph; performing graph embedding on the initial scene graph, and performing similarity calculations with multiple graph embedding vectors in a database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph; displaying the sorted list of similar scene graphs, and determining a target scene graph corresponding to the initial environment image.

[0006] According to an embodiment of the present disclosure, an initial environment image is acquired, and feature extraction and target detection are performed on the initial environment image to obtain initial category features, spatial features and visual features of the initial environment image, including: extracting visual features of the initial environment image based on a feature pyramid network of a deep residual network, wherein the visual features are multi-scale fused visual features; inputting the visual features into a region proposal network and performing candidate region pooling to generate candidate regions; uniformly representing the features of the candidate regions according to a regional feature aggregation method, and obtaining the target initial category features and spatial features according to full connection and softmax functions, respectively.

[0007] According to an embodiment of the present disclosure, obtaining a knowledge vector representation of an initial category feature includes: encoding the initial category feature based on a preset GloVe word vector model and extracting the corresponding GloVe word vector to obtain a knowledge vector representation of the initial category feature.

[0008] According to an embodiment of the present disclosure, the knowledge vector representation of the initial category feature and the visual feature are knowledge-embedded to obtain the target category feature, including: inputting the knowledge vector representation of the initial category feature and the visual feature into a bidirectional long short-term memory network for knowledge embedding to obtain the target category feature.

[0009] According to an embodiment of the present disclosure, context information is obtained based on target category features, spatial features and visual features, and relationship classification is performed through a preset classification model to generate an initial scene graph, including: determining the relationship between target category features, spatial features and visual features through a graph neural network model, and constructing context information; inputting the context information into a preset classification model to obtain relationship classification probability; and visualizing the relationship classification probability to obtain an initial scene graph.

[0010] According to an embodiment of the present disclosure, graph embedding is performed on the initial scene graph, including: representing the initial scene graph as a graph structure; and performing embedding learning on the initial scene graph represented by the graph structure based on a preset graph embedding model to obtain a graph embedding vector of the initial scene graph.

[0011] According to an embodiment of the present disclosure, the database is a preset database, and the database stores graph embedding vectors of multiple types of original scene graphs.

[0012] According to an embodiment of the present disclosure, the initial scene graph is graph-embedded, and similarity calculations are performed with multiple graph embedding vectors in a database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph, including: performing similarity calculations on the graph embedding vector of the initial scene graph and the graph embedding vectors of various types of original scene graphs to obtain similarity calculation results; and sorting the similar scene graphs corresponding to the initial scene graph according to the similarity calculation results to obtain a sorted list of similar scene graphs.

[0013] A second aspect of the present disclosure provides a scene graph generation device based on graph semantic understanding, including: a feature extraction module, used to obtain an initial environment image, and perform feature extraction and target detection on the initial environment image to obtain initial category features, spatial features and visual features of the initial environment image; a knowledge embedding module, used to perform knowledge embedding of the knowledge vector representation of the initial category features with the visual features to obtain target category features; an initial scene graph generation module, used to obtain context information based on the target category features, spatial features and visual features, and perform relationship classification through a preset classification model to generate an initial scene graph; a similar scene graph sorting list generation module, used to perform graph embedding on the initial scene graph, and perform similarity calculations with multiple graph embedding vectors in a database to obtain a sorting list of similar scene graphs corresponding to the initial scene graph; a target scene graph determination module, used to display the sorting list of similar scene graphs, and determine the target scene graph corresponding to the initial environment image.

[0014] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the above method.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium having a computer program or instructions stored thereon, which implements the steps of the above method when the above computer program or instructions are executed by a processor.

[0016] The fifth aspect of the present disclosure further provides a computer program product, including a computer program or instructions, which implement the steps of the above method when the above computer program or instructions are executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0018] Figure 1 An application scenario diagram of a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 2 A flowchart of a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown;

[0020] Figure 3 A flowchart of feature extraction and target detection according to an embodiment of the present disclosure is schematically shown;

[0021] Figure 4 Schematically shows a flow chart of generating an initial scene graph according to an embodiment of the present disclosure;

[0022] Figure 5 Schematically shows a flow chart of generating a sorted list of similar scene graphs according to an embodiment of the present disclosure;

[0023] Figure 6 A scene graph generation device based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown; and

[0024] Figure 7 A block diagram of an electronic device suitable for implementing a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the present disclosure. In the following detailed description, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0026] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise", "include", etc. used herein indicate the existence of features, steps, operations and / or components, but do not exclude the existence or addition of one or more other features, steps, operations or components.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] When using expressions such as "at least one of A, B, and C, etc.", they should generally be interpreted according to the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0029] It should be noted that the scene graph generation method and device based on graph semantic understanding disclosed in the present invention can be used in the field of financial technology, and can also be used in any field other than the field of financial technology. The application field of the scene graph generation method and device based on graph semantic understanding disclosed in the present invention is not limited.

[0030] In the technical solution of the present disclosure, the user information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of the relevant data comply with relevant laws, regulations and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0031] In the scenario of using personal information for automated decision-making, the methods, devices, and systems provided by the embodiments of the present disclosure provide users with corresponding operation portals for users to choose to agree or reject the automated decision-making results; if the user chooses to reject, the expert decision-making process will be entered. The expression "automated decision-making" here refers to the activity of automatically analyzing and evaluating a person's behavioral habits, interests and hobbies, or economic, health, credit status, etc. through computer programs, and making decisions. The expression "expert decision-making" here refers to the activity of making decisions by people who specialize in a certain field, have specialized experience, knowledge and skills, and have reached a certain level of professionalism.

[0032] An embodiment of the present disclosure provides a scene graph generation method based on graph semantic understanding, the method comprising: acquiring an initial environment image, and performing feature extraction and target detection on the initial environment image to obtain initial category features, spatial features and visual features of the initial environment image; performing knowledge embedding of a knowledge vector representation of the initial category features and visual features to obtain target category features; based on the target category features, spatial features and visual features, obtaining context information and performing relationship classification through a preset classification model to generate an initial scene graph; performing graph embedding on the initial scene graph, and performing similarity calculations with multiple graph embedding vectors in a database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph; displaying the sorted list of similar scene graphs, and determining a target scene graph corresponding to the initial environment image.

[0033] Through the embodiments of the present disclosure, the quality and accuracy of scene graph generation are effectively improved through multi-level feature extraction and knowledge embedding. By fusing the knowledge vector of the category feature with the visual feature to form the target category feature, the generated scene graph not only has visual information, but also incorporates semantic knowledge, thereby improving the semantic expression ability of the scene graph. In addition, by embedding the generated initial scene graph and calculating the similarity with the graph embedding vector in the database, not only efficient scene graph retrieval is achieved, but also a ranked list of relevant scene graphs can be provided to credit card business personnel, thereby improving the promotion efficiency and accuracy.

[0034] Figure 1An application scenario graph of a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown.

[0035] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or optical fiber cables, etc.

[0036] The user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0037] The first terminal device 101, the second terminal device 102, and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the scene graph generation method based on graph semantic understanding provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the scene graph generation device based on graph semantic understanding provided in the embodiment of the present disclosure can generally be set in the server 105. The scene graph generation method based on graph semantic understanding provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105. Correspondingly, the scene graph generation device based on graph semantic understanding provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the first terminal device 101, the second terminal device 102, the third terminal device 103 and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0041] The following will be based on Figure 1 The scene described by Figure 2~Figure 5 A scene graph generation method based on graph semantic understanding of the disclosed embodiment is described in detail.

[0042] Figure 2 A flowchart of a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown.

[0043] like Figure 2 As shown, in an embodiment of the present disclosure, a scene graph generation method based on graph semantic understanding specifically includes operations S210 to S250.

[0044] In operation S210, an initial environment image is acquired, and feature extraction and target detection are performed on the initial environment image to obtain initial category features, spatial features, and visual features of the initial environment image.

[0045] For example, business personnel go to a place where credit card promotion is needed, such as a shopping mall, and take relevant photos. The photos are the initial environment images, and the initial environment includes the shopping mall entrance area, the beverage store, the staircase corridor, etc. Based on the acquired initial environment image, feature extraction and target detection are performed to obtain the initial category features, spatial features and visual features of the initial environment image. For example, the initial category features mainly include objects of multiple categories (such as people, cars and seats, etc.), and the visual features include visual information such as people sitting on chairs and people holding mobile phones in their hands; spatial features include the position and size of objects in the image, etc.

[0046] Figure 3 The flowchart of feature extraction and target detection according to an embodiment of the present disclosure is schematically shown.

[0047] like Figure 3 As shown, in an embodiment of the present disclosure, an initial environment image is acquired, and feature extraction and target detection are performed on the initial environment image to obtain initial category features, spatial features, and visual features of the initial environment image, specifically including operations S310 to S330.

[0048] In operation S310 , visual features of an initial environment image are extracted based on a feature pyramid network of a deep residual network, wherein the visual features are multi-scale fused visual features.

[0049] Specifically, select a pre-trained deep residual network, such as ResNet-50, as the basic network. Input the initial environment image (such as a 640x480 RGB image) into the ResNet network, calculate the output features of each layer through forward propagation, and then use the feature pyramid network to fuse the multi-layer features of ResNet to form multi-scale visual features. In this process, feature maps of different scales are upsampled and spliced ​​to generate rich multi-scale feature representations. Visual features can be extracted from different layers of ResNet (such as conv2_x, conv3_x, conv4_x, conv5_x) and fused after adjusting the number of channels through 1x1 convolution.

[0050] In operation S320, the visual features are input into a region proposal network and candidate regions are pooled to generate candidate regions.

[0051] In operation S330, the features of the candidate regions are uniformly represented according to the regional feature aggregation method, and the target initial category features and spatial features are obtained according to the full connection and softmax functions respectively.

[0052] Exemplarily, the multi-scale visual features obtained in step S310 are input into a region proposal network (RPN). RPN uses a sliding window mechanism to generate multiple anchor boxes on the feature map, and predicts the object existence probability and the regression correction parameters of the box for each anchor box. Anchor boxes with a higher object existence probability are screened out according to the IoU threshold, and non-maximum suppression (NMS) is performed to finally generate candidate regions.

[0053] Furthermore, for each candidate region, the corresponding regional features are extracted from the feature map. For example, a RoIPooling operation is used to map candidate regions of different sizes to a feature map of a fixed size (e.g., 7x7).

[0054] Furthermore, the obtained candidate region features are input into the fully connected layer for processing. Specifically, the first fully connected layer is used to map the features to the target category feature space; the second fully connected layer is used to generate spatial feature representation.

[0055] Finally, the softmax function is used to transform the target category features into a probability distribution and determine the initial category of each candidate region.

[0056] Through the embodiments of the present disclosure, by combining a deep residual network with a feature pyramid network, multi-scale features of the initial environment image taken by business personnel are effectively extracted. This multi-scale feature fusion improves the robustness and accuracy of the target detection system in complex scenarios; accurate candidate regions are generated with the help of a region proposal network, which helps to narrow the search space for target detection and improve detection efficiency; and target classification is achieved through full connection and softmax function, which can further achieve more accurate target classification and positioning.

[0057] In operation S220, knowledge embedding is performed between the knowledge vector representation of the initial category feature and the visual feature to obtain the target category feature.

[0058] In an embodiment of the present disclosure, obtaining a knowledge vector representation of an initial category feature includes: encoding the initial category feature based on a preset GloVe word vector model and extracting a corresponding GloVe word vector to obtain a knowledge vector representation of the initial category feature.

[0059] Specifically, load the pre-trained GloVe word vector model. The GloVe model is usually trained with a text corpus and can be loaded through a deep learning framework.

[0060] Furthermore, each extracted initial category feature is encoded. Through encoding, the knowledge vector representation of each initial category feature is obtained. At this time, each category feature corresponds to a high-dimensional GloVe word vector as its knowledge vector.

[0061] In an embodiment of the present disclosure, the knowledge vector representation of the initial category feature and the visual feature are knowledge-embedded to obtain the target category feature, including: inputting the knowledge vector representation of the initial category feature and the visual feature into a bidirectional long short-term memory network for knowledge embedding to obtain the target category feature.

[0062] For example, visual features are combined with knowledge vectors and input into a bidirectional long short-term memory network, the model is initialized and forward propagation is performed to obtain target category features.

[0063] Through the embodiments of the present disclosure, by combining visual features and category knowledge vectors, two different types of information can be fully utilized to form a richer feature representation. This feature fusion can improve the model's ability to understand complex data, thereby improving classification performance. Through a bidirectional structure, the model can pay attention to the contextual information before and after the moment when processing the input at each moment, which helps to capture deeper feature relationships. Through effective knowledge embedding, feature representations that are more consistent with the target category can be generated, which helps to improve accuracy and robustness in subsequent classification or regression tasks.

[0064] In operation S230, context information is obtained based on target category features, spatial features, and visual features, and relationship classification is performed through a preset classification model to generate an initial scene graph.

[0065] Figure 4 The flowchart of generating an initial scene graph according to an embodiment of the present disclosure is schematically shown.

[0066] like Figure 4 As shown, in an embodiment of the present disclosure, context information is obtained based on target category features, spatial features and visual features, and relationship classification is performed through a preset classification model to generate an initial scene graph, which specifically includes operations S410 to S430.

[0067] In operation S410, the relationship between the target category features, the spatial features, and the visual features is determined through a graph neural network model, and context information is constructed.

[0068] For example, each object in the initial environment image is regarded as a node of the graph. The node attributes include target category features and visual features. The edges between nodes represent spatial relationships, such as the distance or overlap between two objects. Furthermore, a graph neural network model is constructed to update node features through a message passing mechanism and learn the relationship between nodes.

[0069] For example, nodes represent objects identified in the initial environment image, such as women, mobile phones, shopping bags, stools, corridors, etc., and edges represent the relationships between these objects.

[0070] Furthermore, the representation of the current node is updated with the information of neighboring nodes, thereby capturing the complex relationship between target category features, spatial features, and visual features. Finally, after training the graph neural network, the embedded representation of each node is obtained, and these node representations are combined to construct the overall context information.

[0071] In operation S420, the context information is input into a preset classification model to obtain a relationship classification probability.

[0072] In operation S430 , the relationship classification probabilities are visualized to obtain an initial scene graph.

[0073] Exemplarily, a fully connected neural network or other deep learning model can be used as a relationship classifier, context information is input, and the final result output by the model is the classification probability of the relationship between each pair of objects.

[0074] Furthermore, the relationship classification probability is converted into a visual graphical representation to form an initial scene graph. For example, a scene graph is drawn using a visualization library to display objects and their relationships.

[0075] Through the embodiments of the present disclosure, the graph neural network can effectively capture the complex relationships between different features, making the representation of target category features and spatial and visual features richer and more accurate, thereby improving the performance of the classification model; by integrating multiple features, contextual information is generated. This contextual information not only includes the representation of a single feature, but also includes the interactive relationship between features, which helps to understand the scene more comprehensively; by visualizing the relationship classification probability, the targets and their relationships in the initial scene graph can be intuitively displayed, which helps to build a more accurate and comprehensive scene graph.

[0076] In operation S240, the initial scene graph is graph-embedded, and similarity calculations are performed with a plurality of graph embedding vectors in the database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph.

[0077] In an embodiment of the present disclosure, graph embedding is performed on the initial scene graph, including: representing the initial scene graph as a graph structure; and performing embedding learning on the initial scene graph represented by the graph structure based on a preset graph embedding model to obtain a graph embedding vector of the initial scene graph.

[0078] Specifically, according to the initial scene graph obtained, the image contains multiple objects and the relationships between the objects (such as a woman, a mobile phone, a shopping bag, and a stool, etc.). The main objects in the image are extracted through the object detection algorithm, and a graph is constructed through the objects and relationships.

[0079] Furthermore, a suitable graph embedding model is pre-selected, and the graph structure is input into the graph embedding model for graph embedding learning. The model will generate an embedding vector through the information of adjacent nodes. The model training process includes: obtaining the local structural information of the node through random walk or neighbor sampling method; using deep learning methods (such as multi-layer perceptron) to convert the local node information into an embedding vector in a high-dimensional space.

[0080] Through the embodiments of the present disclosure, by calculating the similarity of embedded vectors, the similarity between scenes can be captured more accurately, thereby improving the retrieval accuracy of similar scenes.

[0081] Figure 5 The flowchart of generating a sorted list of similar scene graphs according to an embodiment of the present disclosure is schematically shown.

[0082] like Figure 5 As shown, in an embodiment of the present disclosure, the initial scene graph is graph-embedded, and similarity calculations are performed with multiple graph embedding vectors in the database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph, specifically including operations S510 to S520.

[0083] In operation S510, similarity calculations are performed on the graph embedding vector of the initial scene graph and the graph embedding vectors of various types of original scene graphs to obtain similarity calculation results.

[0084] In operation S520, similar scene graphs corresponding to the initial scene graph are sorted according to the similarity calculation result to obtain a sorted list of similar scene graphs.

[0085] Exemplarily, methods such as cosine similarity, Euclidean distance or Manhattan distance are used to calculate the similarity between the graph embedding vector of the initial scene graph and the graph embedding vectors of multiple types of original scene graphs stored in a preset database.

[0086] Furthermore, the similarity calculation result is associated with the corresponding original scene graph to form a tuple list containing scene graphs and their similarities, and the similar scene graph sorting list is sorted in descending order according to the similarity value so that the most similar scene graph is placed in front.

[0087] In operation S250 , a sorted list of similar scene graphs is displayed, and a target scene graph corresponding to the initial environment image is determined.

[0088] Exemplarily, it is determined that the scene graph that ranks first in the sorted list of similar scene graphs is the target scene graph corresponding to the initial environment image. For example, the acquired target scene graph should include a scene recommendation of a girl taking a break while shopping, which shows that the business personnel can promote credit cards based on the recommended target scene graph combined with the current scene.

[0089] Through the embodiments of the present disclosure, the sorted similar scene graph list allows users to quickly find the most relevant scenes, reduces the time for searching and filtering, and improves the response speed of the system.

[0090] Exemplarily, the above-mentioned scene graph generation method based on graph semantic understanding has learned the commonalities and differences between scenes with high promotion rates, and the scenes recommended by the system well contain previous promotion experience. According to the target scene graph obtained by this method, for example, on a weekend afternoon, a girl who is resting in the corridor while shopping and is more delicately dressed is obviously more inclined to apply for a credit card. Through this method, business personnel can quickly obtain all the previous credit card promotion scenes that have been inferred and summarized to obtain the target scene graph corresponding to the scene of the initial environment image at this moment, solve the inefficiency problem caused by blind sales promotion, and realize credit card promotion for targeted scenes and specific target customers.

[0091] Based on the above-mentioned scene graph generation method based on graph semantic understanding, the present disclosure also provides a scene graph generation device based on graph semantic understanding. Figure 6 The device is described in detail.

[0092] Figure 6 The structural block diagram of a scene graph generation device based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown.

[0093] like Figure 6 As shown, the scene graph generation device 600 based on graph semantic understanding of this embodiment includes a feature extraction module 610, a knowledge embedding module 620, an initial scene graph generation module 630, a similar scene graph sorting list generation module 640 and a target scene graph determination module 650.

[0094] The feature extraction module 610 is used to obtain an initial environment image, and perform feature extraction and target detection on the initial environment image to obtain initial category features, spatial features, and visual features of the initial environment image. In one embodiment, the feature extraction module 610 can be used to perform the operation S210 described above, which will not be repeated here.

[0095] The knowledge embedding module 620 is used to embed the knowledge vector representation of the initial category feature with the visual feature to obtain the target category feature. In one embodiment, the knowledge embedding module 620 can be used to perform the operation S220 described above, which will not be repeated here.

[0096] The initial scene graph generation module 630 is used to obtain context information based on target category features, spatial features and visual features and perform relationship classification through a preset classification model to generate an initial scene graph. In one embodiment, the initial scene graph generation module 630 can be used to perform the operation S230 described above, which will not be repeated here.

[0097] The similar scene graph sorting list generation module 640 is used to embed the initial scene graph, and perform similarity calculations with multiple graph embedding vectors in the database to obtain a sorting list of similar scene graphs corresponding to the initial scene graph. In one embodiment, the initial similar scene graph sorting list generation module 640 can be used to perform the operation S240 described above, which will not be repeated here.

[0098] The target scene graph determination module 650 is used to display a sorted list of similar scene graphs and determine a target scene graph corresponding to the initial environment image. In one embodiment, the target scene graph determination module 650 can be used to perform the operation S250 described above, which will not be described in detail here.

[0099] According to an embodiment of the present disclosure, any multiple modules of the feature extraction module 610, the knowledge embedding module 620, the initial scene graph generation module 630, the similar scene graph sorting list generation module 640 and the target scene graph determination module 650 can be combined in one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the feature extraction module 610, the knowledge embedding module 620, the initial scene graph generation module 630, the similar scene graph sorting list generation module 640 and the target scene graph determination module 650 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the feature extraction module 610, the knowledge embedding module 620, the initial scene graph generation module 630, the similar scene graph sorted list generation module 640 and the target scene graph determination module 650 can be at least partially implemented as a computer program module, which can perform corresponding functions when executed.

[0100] Figure 7 A block diagram of an electronic device suitable for implementing a scene graph generation method based on graph semantic understanding according to an embodiment of the present disclosure is schematically shown.

[0101] like Figure 7 As shown, the electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage part 708 to a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (for example, an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include an onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0102] In RAM 703, various programs and data required for the operation of electronic device 700 are stored. Processor 701, ROM 702 and RAM 703 are connected to each other via bus 704. Processor 701 performs various operations of the method flow according to the embodiment of the present disclosure by executing the program in ROM 702 and / or RAM 703. It should be noted that the program can also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 can also perform various operations of the method flow according to the embodiment of the present disclosure by executing the program stored in one or more memories.

[0103] According to an embodiment of the present disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to the bus 704. The electronic device 700 may further include one or more of the following components connected to the input / output (I / O) interface 705: an input portion 706 including a keyboard, a mouse, etc.; an output portion 707 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 708 including a hard disk, etc.; and a communication portion 709 including a network interface card such as a LAN card, a modem, etc. The communication portion 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 710 as needed, so that a computer program read therefrom is installed into the storage portion 708 as needed.

[0104] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed, the method according to the embodiment of the present disclosure is implemented.

[0105] According to an embodiment of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 702 and / or RAM 703 described above and / or one or more memories other than ROM 702 and RAM 703.

[0106] The embodiment of the present disclosure also includes a computer program product, which includes a computer program, and the computer program includes a program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the scene graph generation method based on graph semantic understanding provided by the embodiment of the present disclosure.

[0107] The above functions defined in the system / device of the embodiment of the present disclosure are performed when the computer program is executed by the processor 701. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0108] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 709, and / or installed from the removable medium 711. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0109] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, the above functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0110] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0111] The flow charts and block diagrams in the accompanying drawings illustrate the possible architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0112] It will be appreciated by those skilled in the art that the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways, even if such combinations or combinations are not explicitly described in the present disclosure. In particular, without departing from the spirit and teachings of the present disclosure, the features described in the various embodiments of the present disclosure may be combined and / or combined in a variety of ways. All of these combinations and / or combinations fall within the scope of the present disclosure.

[0113] The embodiments of the present disclosure are described above. However, these embodiments are only for illustrative purposes and are not intended to limit the scope of the present disclosure. Although the embodiments are described above, this does not mean that the measures in the various embodiments cannot be used in combination to advantage. Without departing from the scope of the present disclosure, those skilled in the art may make a variety of substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A scene graph generation method based on graph semantic understanding, characterized in that: The method comprises: Acquire an initial environment image, and perform feature extraction and target detection on the initial environment image to obtain initial category features, spatial features, and visual features of the initial environment image; Embedding the knowledge vector representation of the initial category feature with the visual feature to obtain the target category feature; Based on the target category features, spatial features and visual features, context information is obtained and relationship classification is performed through a preset classification model to generate an initial scene graph; Embedding the initial scene graph, and calculating similarity with multiple graph embedding vectors in the database, to obtain a sorted list of similar scene graphs corresponding to the initial scene graph; The sorted list of similar scene graphs is displayed, and a target scene graph corresponding to the initial environment image is determined.

2. The scene graph generation method based on graph semantic understanding according to claim 1, characterized in that: The obtaining of the initial environment image, and performing feature extraction and target detection on the initial environment image to obtain initial category features, spatial features, and visual features of the initial environment image include: Extracting visual features of the initial environment image based on a feature pyramid network of a deep residual network, wherein the visual features are multi-scale fused visual features; Inputting the visual features into a region proposal network and performing candidate region pooling to generate candidate regions; The features of the candidate regions are uniformly represented according to the regional feature aggregation method, and the target initial category features and spatial features are obtained respectively according to the full connection and softmax functions.

3. The scene graph generation method based on graph semantic understanding according to claim 1, characterized in that: Obtaining the knowledge vector representation of the initial category feature includes: Based on the preset GloVe word vector model, the initial category features are encoded and the corresponding GloVe word vectors are extracted to obtain the knowledge vector representation of the initial category features.

4. The scene graph generation method based on graph semantic understanding according to claim 1 or 3, characterized in that: The step of embedding the knowledge vector representation of the initial category feature with the visual feature to obtain the target category feature includes: The knowledge vector representation of the initial category feature and the visual feature are input into a bidirectional long short-term memory network for knowledge embedding to obtain the target category feature.

5. The scene graph generation method based on graph semantic understanding according to claim 1, characterized in that: The process of obtaining context information based on the target category features, spatial features and visual features and performing relationship classification through a preset classification model to generate an initial scene graph includes: Determine the relationship between target category features, spatial features, and visual features through the graph neural network model, and construct contextual information; Inputting the context information into a preset classification model to obtain a relationship classification probability; The relationship classification probability is visualized to obtain an initial scene graph.

6. The scene graph generation method based on graph semantic understanding according to claim 1, characterized in that: The embedding of the initial scene graph includes: Representing the initial scene graph as a graph structure; Based on a preset graph embedding model, embedding learning is performed on an initial scene graph represented by a graph structure to obtain a graph embedding vector of the initial scene graph.

7. The scene graph generation method based on graph semantic understanding according to claim 6, characterized in that: The database is a preset database, and the database stores graph embedding vectors of multiple original scene graphs of various types.

8. The scene graph generation method based on graph semantic understanding according to claim 1 or 7, characterized in that: The initial scene graph is embedded, and similarity is calculated with a plurality of graph embedding vectors in a database to obtain a sorted list of similar scene graphs corresponding to the initial scene graph, including: Performing similarity calculations on the graph embedding vector of the initial scene graph and the graph embedding vectors of each type of original scene graph to obtain similarity calculation results; According to the similarity calculation result, similar scene graphs corresponding to the initial scene graph are sorted to obtain a sorted list of similar scene graphs.

9. A scene graph generation device based on graph semantic understanding, characterized in that: The device comprises: A feature extraction module is used to obtain an initial environment image, and perform feature extraction and target detection on the initial environment image to obtain initial category features, spatial features and visual features of the initial environment image; A knowledge embedding module, used for embedding the knowledge vector representation of the initial category feature with the visual feature to obtain the target category feature; An initial scene graph generation module is used to obtain context information based on the target category features, spatial features and visual features and perform relationship classification through a preset classification model to generate an initial scene graph; A similar scene graph sorting list generating module is used to embed the initial scene graph, and respectively calculate similarity with a plurality of graph embedding vectors in the database to obtain a similar scene graph sorting list corresponding to the initial scene graph; The target scene graph determination module is used to display the sorted list of similar scene graphs and determine the target scene graph corresponding to the initial environment image.

10. An electronic device comprising: one or more processors; a memory for storing one or more computer programs, It is characterized in that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instruction stored thereon, characterized in that: When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images