Multimodal data processing method, system and virtualization system based on scene graph

Abstract

The present invention discloses a multimodal data processing method, system and virtualization system based on scene graph, which relates to the field of data processing. The method includes: obtaining multimodal data to be processed and corresponding application scenarios and application requirements; constructing the ontology required for the scene graph based on the application scenario; semantically parsing the multimodal data based on the ontology and the target model to obtain several scene graphs; the scene graph is used to characterize the metadata tags of the multimodal data; fusing several scene graphs according to the application requirements to obtain a fusion map, and storing the fusion map in a model warehouse. Automatically perform semantic parsing of multimodal data to obtain a scene graph, and automatically perform scene graph fusion according to business requirements, so that data storage costs are reduced, development costs are reduced, and the reusability of development results is improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a multimodal data processing method, system and virtualization system based on a scene graph. Background Art

[0002] The core goal of industrial digitalization is to achieve intelligent decision-making by developing data applications based on enterprise or industry data, respond to rapidly changing business needs, and form high-speed business decision-making capabilities.

[0003] Traditional data management methods or platforms, such as data middle platforms and data warehouses, are insufficient in supporting high-speed business decision-making. For example, centralized data storage leads to high transmission, storage, and operation and maintenance costs; data lacks unified standards, and data fusion involves a large amount of customization, which results in a lot of time-consuming and repetitive development of data applications. Summary of the invention

[0004] The main purpose of the present invention is to provide a multimodal data processing method, system and virtualization system based on scene graph, automatically perform multimodal data semantic analysis to obtain scene graph, and automatically perform scene graph fusion according to business needs, so as to reduce data storage costs, reduce development costs, and improve the reusability of development results.

[0005] In order to achieve the above objectives, the embodiments of the present application provide the following technical solutions:

[0006] According to a first aspect of an embodiment of the present application, a multimodal data processing method based on a scene graph is provided, the method comprising:

[0007] Obtain multimodal data to be processed and the corresponding application scenarios and application requirements;

[0008] Constructing the ontology required for the scene graph based on the application scenario;

[0009] Performing semantic parsing on the multimodal data based on the ontology and the target model to obtain a plurality of scene graphs; the scene graphs are used to represent metadata tags of the multimodal data;

[0010] According to the application requirements, several scene graphs are fused to obtain a fused graph, and the fused graph is stored in a model warehouse.

[0011] Optionally, semantic parsing of the multimodal data is performed based on the ontology and the target model to obtain several scene graphs, including:

[0012] Inputting the multimodal data into the target model, and determining a number of target relationships between the ontologies in the multimodal data; the multimodal data includes text data and image data;

[0013] The target model is input based on the plurality of target relationships to perform triple instance extraction to obtain a plurality of scene graphs.

[0014] Optionally, the determining a number of target relationships between the ontologies in the multimodal data includes:

[0015] If the multimodal data is text data, input the text data into the target model to obtain several target relations of the ontology;

[0016] If the multimodal data is image data, the image data is input into the target model for data conversion to obtain text data; the text data is input into the target model to obtain several target relationships of the ontology.

[0017] Optionally, fusing several scene graphs according to the application requirements to obtain a fused graph includes:

[0018] Input the application requirements and the ontology into the target model to determine whether scene fusion is required;

[0019] If scene fusion is not required, the target model directly outputs the ontology;

[0020] If scene fusion is required, the application requirements and the ontology are input into the target model to determine fusion rules; based on the fusion rules, the several scene graphs are fused to obtain a fusion graph.

[0021] Optionally, the ontology required for constructing the scene graph based on the application scenario includes:

[0022] Determining whether there is a related ontology in the model repository based on the multimodal data to be processed and the application scenario;

[0023] If it does not exist, construct the ontology required for the scene graph according to the application scenario; if it does exist, determine whether it is necessary to complete the ontology on the associated ontology;

[0024] If necessary, the ontology is completed on the associated ontology to obtain the ontology required by the scene graph.

[0025] According to a second aspect of an embodiment of the present application, a multimodal data processing system based on a scene graph is provided, the system comprising:

[0026] A data acquisition module is used to acquire the multimodal data to be processed and the corresponding application scenarios and application requirements;

[0027] An ontology determination module, used for constructing an ontology required for a scene graph based on the application scenario;

[0028] A scene graph determination module, configured to perform semantic analysis on the multimodal data based on the ontology and the target model to obtain a plurality of scene graphs; the scene graphs are used to represent metadata tags of the multimodal data;

[0029] The fusion module is used to fuse several scene graphs according to the application requirements to obtain a fusion graph, and store the fusion graph in the model warehouse.

[0030] According to a third aspect of an embodiment of the present application, a multimodal data virtualization system based on a scene graph is provided, the system comprising:

[0031] The ontology construction module is used to construct the ontology required by the scenario graph according to the business scenario needs; if the relevant ontology already exists, the ontology is completed based on the existing ontology;

[0032] The scene graph construction module is used to drive the data processing models for various modalities and fields in the model warehouse through the target model to realize the conversion of multimodal data into the form of scene graph;

[0033] The scene graph fusion module is used to intelligently determine whether and how to fuse scene graphs of multiple different business types of data based on business scenario needs through the target model;

[0034] Model warehouse, including target model module, general image parsing module and general document parsing module; used to manage various data processing models required for metadata tagging, and also used to parse user query requests;

[0035] The connector layer is used to connect to the physical data source through the data connector layer. The connector supports reading data from the data source into the memory and handing it over to the subsequent process for processing;

[0036] API interface layer, used to support several APIs for accessing resources;

[0037] The public basic layer includes data query module, query parsing module and data view module.

[0038] According to a fourth aspect of an embodiment of the present application, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0039] According to a fifth aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which computer-readable instructions are stored. The computer-readable instructions can be executed by a processor to implement the method described in the first aspect above.

[0040] In summary, the embodiments of the present application provide a multimodal data processing method, system and virtualization system based on a scene graph, by obtaining the multimodal data to be processed and the corresponding application scenarios and application requirements; constructing the ontology required for the scene graph based on the application scenario; semantically parsing the multimodal data based on the ontology and the target model to obtain several scene graphs; the scene graph is used to characterize the metadata tags of the multimodal data; according to the application requirements, several scene graphs are fused to obtain a fusion map, and the fusion map is stored in a model warehouse. Automatically perform semantic parsing of multimodal data to obtain a scene graph, and automatically perform scene graph fusion according to business requirements, so that data storage costs are reduced, development costs are reduced, and the reusability of development results is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the structures shown in these drawings without paying creative work.

[0042] The structures, proportions, sizes, etc. illustrated in this specification are only used to match the contents disclosed in the specification so as to facilitate understanding and reading by persons familiar with the technology. They are not used to limit the conditions under which the present invention can be implemented, and therefore have no substantial technical significance. Any structural modification, change in proportion or adjustment of size shall still fall within the scope of the technical contents disclosed in the present invention without affecting the effects and purposes that can be achieved by the present invention.

[0043] Figure 1 A schematic diagram of a multimodal data processing method based on a scene graph provided in an embodiment of the present application;

[0044] Figure 2 A schematic diagram of the system architecture provided for an embodiment of the present application;

[0045] Figure 3 A block diagram of a multimodal data processing system based on a scene graph provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown;

[0047] Figure 5 A schematic diagram of a computer-readable storage medium provided in an embodiment of the present application is shown.

[0048] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] It should be noted that all directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of the present invention are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0051] In addition, in the present invention, descriptions such as "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0052] In the present invention, unless otherwise clearly specified and limited, the terms "connection", "fixation", etc. should be understood in a broad sense. For example, "fixation" can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0053] In addition, the technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0054] Data virtualization can provide an abstract logical layer to virtually integrate multi-source heterogeneous data without the need to store physical data in a centralized manner, thereby greatly saving data storage and data development costs. At the same time, the logical layer can be combined with AI technology to form standardized data development and fusion generalization capabilities, allowing the application development process to be reused to the greatest extent, thereby improving data and application development efficiency.

[0055] In the digital transformation of industrial enterprises, unstructured data accounts for more than 80% of the total data, and is growing at a high rate of 65% per year, and is presented in multimodal forms such as documents (office, design, logs, etc.), videos (security, conference, etc.), images, etc. Compared with structured data, unstructured data is semantically rich and difficult to understand and integrate. The industry and academia have not yet formed a mature unstructured data virtualization system and corresponding supporting technologies and tools.

[0056] Scene graph is a graphical representation method of data and a special case of knowledge graph. A scene graph is used to describe a "scene" and can be applied to various data modalities. In computer vision, scene graphs describe graphic scenes in the form of nodes and edges. Each node represents an object in the scene, and the edges represent the relationship between objects. Through the scene graph, the computer can more accurately understand the image content, infer the relative position, occlusion relationship and semantic information between objects, thereby improving the effect of visual tasks. Scene graphs can also be applied to natural language understanding. By mapping text information to the nodes and edges of the scene graph, the scene graph provides a structured way for the model to understand the scene described by the text, so that the computer can better understand natural language and then infer the connotation of the scene. The embodiment of the present application provides a data virtualization system based on a scene graph for multimodal data.

[0057] The commonly used development ideas for multimodal unstructured data are as follows: First, business construction is carried out: in order to apply and build a business system, it may involve dozens or even more business requirements, such as statistical results from various dimensions, data dependency analysis, etc., and then data development work is carried out according to the following steps:

[0058] 1. Data access and storage: Aggregate and store data on a unified data platform to achieve unified scheduling and use of data; 2. Select a business requirement; 3. Data analysis: Based on the selected business requirement, conduct targeted data analysis technology research and development on the data required by the business to obtain the knowledge in the original document data. Enterprise documents may exist in a large number of different forms, including electronic documents and scanned documents, which contain a large number of mixed arrangements of text, pictures and tables; 4. Knowledge association and fusion: Based on the selected business requirement, a separate processing flow is constructed, and the analysis results of the previous step are further associated and fused in combination with the business goals to obtain the required final results; 5. If there are unfinished business requirements, return to step 2 until all business requirements are completed.

[0059] However, the existing document data-related application development process has the following problems:

[0060] 1. Data duplication storage cost problem: All used data needs to retain multiple copies during the processing process. Multimodal data itself occupies a large storage space, resulting in high storage costs;

[0061] 2. Repeated development cost problem: All business needs require independent data analysis and processing solutions and knowledge association and fusion processing solutions, resulting in high development / R&D costs;

[0062] 3. Complex data modality: The unstructured data processed by enterprises includes a large number of different modalities, such as video, audio, documents, and pictures. The development of semantic parsing technology for various modalities is very complex;

[0063] 4. Difficulty in knowledge fusion: There are also high technical difficulties in efficiently fusion and development of semi-structured knowledge obtained through semantic analysis.

[0064] The embodiment of the present application proposes a multimodal data virtualization method, which enables developers to reuse existing development results to the greatest extent, avoid a large amount of repeated storage and development costs, and in this process fully utilize the capabilities of large language models to reduce the difficulty of developing data processing algorithms. This reduces data storage costs, development costs, and improves the reusability of development results. Multimodal data semantic analysis is automatically performed to obtain scene graphs, and scene graphs are automatically fused according to business goals.

[0065] The following is an explanation of the concepts related to multimodal data involved in the embodiments of the present application:

[0066] Metadata: Data that describes data, used to support functions such as indicating data source, storage location, historical data, and data content description.

[0067] Scenario: A scenario represents a business direction or data source. For example, a technical invention patent is a scenario, and security video data captured by a camera is also a scenario. A scenario represents a data instance under a scenario, such as a technical patent, or a security picture captured by a camera at a certain moment.

[0068] Ontology: The schema of the graph structure used in one scenario, or the schema of the graph fusion structure used for the fusion of multiple scenarios.

[0069] Scene graph: A graphical data structure used to describe object instances in a scene and the relationships between them. The scene graph does not contain raw data, but metadata of the raw data.

[0070] Scene graph fusion: The scene graph is used to describe one piece of data. When describing multiple pieces of data, multiple scene graphs need to be fused. The fusion process depends on the specific business and can be either fusion within a single scene or fusion between multiple scenes.

[0071] Metadata graph (fusion graph): A metadata graph corresponds to a scene or a fusion of multiple scenes, and contains a fusion result of all data in one scene or multiple scenes. The fusion method depends on the specific business. The fusion graph does not contain the original data, but the metadata of the original data.

[0072] Query (retrieval): The process by which a specific application obtains a subgraph related to the query target from the fused graph through some expression (natural language, graph query statement, etc.).

[0073] Large language models (LLMs) refer to natural language processing models with a large number of learning parameters. Representative examples of such models include GPT-3, llama, etc. The main feature of large language models is their large parameter size, which enables them to learn and capture complex, abstract patterns and relationships in language. Such models usually learn rich language representations through a large-scale pre-training phase, and then fine-tune them on specific tasks to adapt them to specific application areas. Large language models are widely used in the field of natural language processing. They can achieve tasks such as language generation, text understanding, question answering, and dialogue systems, and have achieved significant performance improvements on these tasks. These models can produce natural and accurate texts through strong context understanding and language generation capabilities, providing powerful language processing capabilities for various application scenarios.

[0074] Figure 1 A multimodal data processing method based on a scene graph provided in an embodiment of the present application is shown, and the method includes:

[0075] Step 101: Obtain multimodal data to be processed and corresponding application scenarios and application requirements;

[0076] Step 102: constructing an ontology required for the scene graph based on the application scenario;

[0077] Step 103: semantically parse the multimodal data based on the ontology and the target model to obtain a number of scene graphs; the scene graphs are used to represent metadata tags of the multimodal data;

[0078] Step 104: Fuse several scene graphs according to the application requirements to obtain a fused graph, and store the fused graph in a model warehouse.

[0079] In a possible implementation, in step 102, the ontology required for constructing the scene graph based on the application scenario includes:

[0080] Based on the multimodal data to be processed and the application scenario, determine whether there is an associated ontology in the model repository; if not, construct the ontology required for the scene graph according to the application scenario; if so, determine whether it is necessary to complete the ontology on the associated ontology; if necessary, complete the ontology on the associated ontology to obtain the ontology required for the scene graph.

[0081] In a possible implementation, in step 103, semantic parsing is performed on the multimodal data based on the ontology and the target model to obtain several scene graphs, including:

[0082] The multimodal data is input into the target model to determine a number of target relations between the ontologies in the multimodal data; the multimodal data includes text data and image data; based on the number of target relations, the target model is input to perform triple instance extraction to obtain a number of scene graphs, which are used to characterize metadata tags of the multimodal data.

[0083] In a possible implementation, the determining a number of target relationships between the ontologies in the multimodal data includes:

[0084] If the multimodal data is text data, the text data is input into the target model to obtain several target relations of the ontology; if the multimodal data is image data, the image data is input into the target model for data conversion to obtain text data; the text data is input into the target model to obtain several target relations of the ontology.

[0085] In a possible implementation, in step 104, fusing a plurality of scene graphs according to the application requirements to obtain a fused graph includes:

[0086] The application requirements and the ontology are input into the target model to determine whether scene fusion is required; if scene fusion is not required, the target model directly outputs the ontology; if scene fusion is required, the application requirements and the ontology are input into the target model to determine the fusion rules; based on the fusion rules, the several scene graphs are fused to obtain a fusion graph.

[0087] In a possible implementation, generating a fusion rule includes:

[0088] The application requirements and the ontology are input into the target model to determine fusion rules.

[0089] In a possible implementation, after the application requirements and the ontology are input into the target model to determine whether scene fusion is required, the method further includes: if scene fusion is not required, the target model directly outputs the ontology.

[0090] Figure 2 The data virtualization system architecture provided by the embodiment of the present application is shown. It includes a data application development layer, a multimodal data virtualization system and a physical data source. The multimodal data virtualization system includes an API interface layer, a public basic layer, an ontology construction module, a scene graph construction module, a scene graph fusion module, a model warehouse and a connector layer. The public basic layer includes a data query module, a query parsing module, and a data view module. The model warehouse includes an LLM module, a general image parsing module, and a general document parsing module.

[0091] The API interface layer uses API automatic generation and management capabilities, and uses an open method to support multiple APIs for accessing resources. That is, the same software resources can be accessed through different interfaces, supporting multiple protocol APIs such as Restful, Webservice, MQTT, Socket, etc.

[0092] When the data query module is used for actual query, the system provides filling of data entities and obtains the data required by the user by passing it into the Neo4j database based on Neo4j search statements.

[0093] The query parsing module is used for users to implement queries by inputting natural language, and the data retrieval requirements described in the scene graph fusion are processed as described above.

[0094] The data view module is used to convert the scene graph into an abstract view. The view represents a certain perspective of the scene graph. Taking the contract scenario as an example, the scene graph can be represented as a table centered on the contract number through the view. The fields in the table are the key attributes of the contract, such as Party A, Party B, and subject matter. A row of the table is a scene graph corresponding to a contract. The view only includes the table header and does not contain actual data. Data is filled in when the user queries to achieve on-demand data retrieval.

[0095] The connector layer is a unified access specification for software resources. Connect to the physical data source through the data connector layer. The connector supports reading data from the data source into memory and handing it over to subsequent processes for processing. Connectors are customized according to the type of data source. Common data sources for multimodal data storage include non-relational databases (such as MongoDB), local file systems (such as NTFS and Ext4), distributed file systems (such as NTFS), and other storage systems. The connection to non-relational databases is achieved through the database's own connector, and the connection to local file systems, distributed file systems, and other storage systems is achieved by developing and deploying a set of independent connection tools on the storage system. For new data source types, support is provided to expand the required software resources by adding connectors without the need for data migration, which facilitates users to quickly access various commonly used heterogeneous software resources according to established specifications.

[0096] All multimodal unstructured data are tagged with metadata. The embodiment of the present application uses a scene graph to define metadata tags, that is, each piece of data has one or more scene graphs (each scene graph is associated with a set of ontologies) to describe the key information in the data. The scene graph is stored in the Neo4j database in the form of a graph.

[0097] The model warehouse is used to manage various models required for metadata tagging, and is also used to parse user query requests and other goals. It includes parsing algorithms under large language models (LLM), documents, images, videos and other modalities. The parsing scenarios include various typical user usage scenarios. Document data includes document layout analysis algorithms, OCR algorithms, etc. Image algorithms include 1000 classification algorithms trained on ImageNet, target detection algorithms trained on COCO datasets, license plate detection algorithms, face detection algorithms, face recognition algorithms, etc. Video algorithms include general target tracking algorithms, vehicle tracking algorithms, video target detection algorithms, etc. Each algorithm is manually labeled to annotate the model's functions, model inputs and model output formats.

[0098] The ontology construction module is used to support users to build a set of ontologies required for scene graphs according to the needs of business scenarios. Based on the multimodal data to be processed and the application scenario, it is determined whether there is a related ontology in the model warehouse; if not, the ontology required for the scene graph is constructed according to the application scenario; if it exists, it is determined whether it is necessary to complete the ontology on the related ontology; if necessary, the ontology is completed on the related ontology to obtain the ontology required for the scene graph. If a related ontology already exists, the ontology is completed based on the existing ontology. For example, in the application development for contract documents, the target of data retrieval includes Party A of the contract, so an ontology of the type {contract, Party A, company} is added to the ontology.

[0099] The scene graph construction module is used to convert multimodal data into scene graphs by driving the data processing models for various modalities and fields in the model warehouse through LLM. The input of LLM includes the ontology corresponding to the scene graph to be parsed and the instructions required for each model. This process includes two steps:

[0100] First: Relationship identification: Use LLM to identify whether a specific relationship exists in the data. Determine whether each target relationship appears in the data;

[0101] Enter the prompt of LLM to guide the model to generate data processing flow. The processing flow may include document layout analysis, table parsing, OCR (optical character recognition), document layout restoration and other steps, depending on the data type. Use the processed data to enter LLM again to identify the relationships defined in the existing ontology.

[0102] Second: Triple extraction: For the identified relations, LLM is used to extract instantiated triples.

[0103] Enter the prompt of LLM to instruct the model to extract entity pairs of specific relations from the text. The output should be in table form, listing all entity pairs found, and answering "none" if none are found. Then store the scene graph in a graph database (such as Neo4j).

[0104] The following is an explanation of the scene graph construction module provided by the embodiment of the present application in conjunction with a specific application scenario. The process of using a large language model (LLM) to construct a scene graph involves two different scenarios: contract data and security image data.

[0105] Scenario 1: The data to be parsed is contract data, and the ontology includes {contract, party A, company}. The process is as follows:

[0106] Use LLM to generate data processing flow for contract text. Identify ontological relationships in the text. Extract instantiated triples in the text, such as (contract, party A, company).

[0107] The first stage: relationship identification, obtaining the target relationship existing in the data.

[0108] If the multimodal data is text data, the text data is input into the target model to obtain several target relations of the ontology; the text data is input into the target model to obtain several target relations of the ontology.

[0109] The following text is used as the LLM prompt to drive the process: "Your task is to determine whether certain relationships exist in the {text category} data. The relationships that need to be determined include {the list of relationships in the ontology}.

[0110] To achieve this goal, we first need to generate a data processing pipeline to obtain understandable data. We can choose from the following models and string them together as tasks to obtain text input that can be processed:

[0111] The model description is as follows: {description of all available models}, the data source format to be processed is {format of source data}, and the output format should be {model processing flow}". The above process obtains the model processing flow, such as the "document layout analysis-table parsing-OCR-document layout restoration" flow. This flow is used to process the data in the connector and convert the data into text data.

[0112] Then input the following text as prompt into LLM: "Your task is to determine whether certain relationships exist in text data. The relationships that need to be determined include {the list of relationships in the ontology}. The text to be determined is {the text obtained in the previous step}. List the existing relationships in a list."

[0113] Phase 2: Triple extraction to obtain all scenarios present in the data Figure 3 Tuple instance. A triple instance is extracted based on a target model inputted into a target relationship to obtain a plurality of scene graphs, which are used to represent metadata tags of multimodal data.

[0114] For each relationship obtained in the previous step (taking the relationship {contract, party A, company} as an example), enter the LLM with the following text as the prompt: "Your task is to extract a specific relationship from the following text. The two entity types are {(contract, company)} and the relationship between them is {party A}. Please find these two entities. If there are multiple groups, list them all in groups. If they do not exist, answer: none. Respond in a table format with two columns and the table header is {(subject, object)}".

[0115] Scenario 2: The data to be parsed is security image data, and the entity includes {personnel, action, object}. The process is as follows:

[0116] Use LLM to generate a data processing flow for security images. Identify the ontology relationships in the images. Extract instantiated triples in the images, such as (person, action, object), and attach related images to corresponding nodes. If the multimodal data is image data, input the image data into the target model for data conversion to obtain text data. Input the text data into the target model to obtain several target relationships of the ontology.

[0117] The first stage: relationship identification, obtaining the target relationship existing in the data.

[0118] The following text is used as the prompt of LLM to drive the process: "Your task is to determine whether there are certain relationships in the {security image category} data. The relationships that need to be determined include {the relationship list in the ontology}.

[0119] To achieve this goal, we first generate a data processing pipeline to obtain understandable data, select from the following models and string them together as tasks to obtain processable text input and give corresponding relational discriminations from them:

[0120] The model description is as follows: {description of all available models}, the data source format to be processed is {format of source data}, and your output format should be {model processing flow}". The above process obtains the model processing flow, such as "dense crowd detection-face recognition-personnel behavior detection-data assembly". Use this flow to process the data in the connector and convert the data into text data.

[0121] Then input the following text as prompt into LLM: "Your task is to determine whether certain relationships exist in text data. The relationships that need to be determined include {the list of relationships in the ontology}. The text to be determined is {the text obtained in the previous step}. List the existing relationships in a list."

[0122] The second stage: triple extraction to obtain all the scenarios in the data Figure 3 Tuple instance: Based on a number of target relations, the target model is input to perform triple instance extraction to obtain a number of scene graphs, which are used to represent metadata tags of multimodal data.

[0123] For each relationship obtained in the previous step (taking {person, action, object} as an example), enter the following text as the prompt into the LLM: "Your task is to extract a specific relationship from the following text. The two entity types are {(person, object)} and the relationship between them is {action}. Please find these two entities. If there are multiple groups, list them all by group. If they do not exist, answer: None. Reply in a table with two columns and the header is {(subject, object)}". The above process obtains all the scenarios existing in the data. Figure 3 Tuple instance. The scene graph contains image data, such as attaching a person's photo to a person node.

[0124] A scene graph is used to describe one piece of data. When describing multiple pieces of data, multiple scene graphs need to be fused together. The fusion process depends on the specific business and can be either fusion within a single scene or fusion between multiple scenes.

[0125] The scene graph fusion module is driven by business needs (business retrieval needs) and uses LLM to intelligently determine whether scene graphs of multiple different business type data need to be fused and how to fuse them. Based on the multimodal data to be processed and the application scenario, it is determined whether there is an associated ontology in the model warehouse; if not, the ontology required for the scene graph is constructed according to the application scenario; if it exists, it is determined whether it is necessary to complete the ontology on the associated ontology; if necessary, the ontology is completed on the associated ontology to obtain the ontology required for the scene graph. This process includes 3 steps: fusion judgment, judging whether fusion is needed based on the needs; fusion rule generation, automatically generating fusion reasoning rules; fusion, reasoning using reasoning rules.

[0126] Scenario: The data retrieval requirement is "What patents were published by the company's customers last year?" The existing businesses include several types, including contracts, patents, and others. The process is as follows:

[0127] The first stage: fusion judgment, judging whether fusion is needed based on the requirements, and obtaining a list of ontologies to be fused.

[0128] Enter the following text as the prompt into LLM: "Your task is to determine whether different ontologies need to be integrated to support the following data retrieval requirement {retrieval requirement}. All ontologies to be determined include {all ontologies}. If integration is not required, answer the ontology that can directly obtain the result. Otherwise, answer the list of ontologies to be integrated in a list format." The above process obtains a list of ontologies to be integrated, such as [contract, patent].

[0129] The second stage: Generate fusion rules and automatically generate fusion reasoning rules.

[0130] Enter the following text as a prompt into LLM: "Your task is to give an inference rule for a relationship. The relationship to be inferred is obtained from the following data retrieval requirements {data retrieval requirements}. The ontology to be fused is {ontology list}. Give the name of the fused relationship and its inference method." The name of the fused relationship obtained above is "patents published by customers", and the inference rule is "∃X. Application (X, ∃W. Party B (W, ∃T. Party A (name, T))", where X represents the patent entity, W represents the company entity, T represents the contract entity, name represents the known company name, and application, party B, and party A represent the relationship type.

[0131] The third stage: fusion, reasoning using inference rules.

[0132] Using the inference rules generated above, a reasoning tool is constructed to implement rule reasoning between all contract scenario graphs and patent scenario graphs, and obtain a new relationship named "patents published by customers".

[0133] In summary, the embodiments of the present application provide a multimodal data processing method based on a scene graph, by obtaining the multimodal data to be processed and the corresponding application scenarios and application requirements; constructing the ontology required for the scene graph based on the application scenario; semantically parsing the multimodal data based on the ontology and the target model to obtain several scene graphs; the scene graph is used to characterize the metadata tags of the multimodal data; and fusing several scene graphs according to the application requirements to obtain a fusion graph, and storing the fusion graph in a model warehouse. Automatically perform semantic parsing of multimodal data to obtain a scene graph, and automatically perform scene graph fusion according to business requirements, so that data storage costs are reduced, development costs are reduced, and the reusability of development results is improved.

[0134] Based on the same technical concept, the embodiment of the present application also provides a multimodal data processing system based on a scene graph, such as Figure 3 As shown, the system comprises:

[0135] The data acquisition module 301 is used to acquire the multimodal data to be processed and the corresponding application scenarios and application requirements;

[0136] An ontology determination module 302 is used to construct an ontology required for a scene graph based on the application scenario;

[0137] A scene graph determination module 303 is used to perform semantic analysis on the multimodal data based on the ontology and the target model to obtain a plurality of scene graphs; the scene graphs are used to represent metadata tags of the multimodal data;

[0138] The fusion module 304 is used to fuse several scene graphs according to the application requirements to obtain a fused graph, and store the fused graph in the model warehouse.

[0139] Based on the same technical concept, the embodiment of the present application also provides a multimodal data virtualization system based on a scene graph, the system comprising:

[0140] The ontology construction module is used to construct the ontology required by the scenario graph according to the business scenario needs; if the relevant ontology already exists, the ontology is completed based on the existing ontology;

[0141] The scene graph construction module is used to drive the data processing models for various modalities and fields in the model warehouse through the target model to realize the conversion of multimodal data into the form of scene graph;

[0142] The scene graph fusion module is used to intelligently determine whether and how to fuse scene graphs of multiple different business types of data based on business scenario needs through the target model;

[0143] Model warehouse, including target model module, general image parsing module and general document parsing module; used to manage various data processing models required for metadata tagging, and also used to parse user query requests;

[0144] The connector layer is used to connect to the physical data source through the data connector layer. The connector supports reading data from the data source into the memory and handing it over to the subsequent process for processing;

[0145] API interface layer, used to support several APIs for accessing resources;

[0146] The public basic layer includes data query module, query parsing module and data view module.

[0147] The present application also provides an electronic device corresponding to the method provided in the above embodiment. Figure 4 , which shows a schematic diagram of an electronic device provided by some embodiments of the present application. The electronic device 20 may include: a processor 200, a memory 201, a bus 202 and a communication interface 203, wherein the processor 200, the communication interface 203 and the memory 201 are connected via the bus 202; the memory 201 stores a computer program that can be run on the processor 200, and the processor 200 executes the method provided by any of the aforementioned embodiments of the present application when running the computer program.

[0148] The memory 201 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one physical port 203 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0149] The bus 202 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 201 is used to store programs, and the processor 200 executes the programs after receiving execution instructions. The method disclosed in any implementation of the aforementioned embodiment of the present application may be applied to the processor 200, or implemented by the processor 200.

[0150] The processor 200 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 200. The above processor 200 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 201, and the processor 200 reads the information in the memory 201 and completes the steps of the above method in combination with its hardware.

[0151] The electronic device provided in the embodiment of the present application and the method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, operated or implemented by them.

[0152] The present application also provides a computer-readable storage medium corresponding to the method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is run by a processor, the method provided in any of the aforementioned embodiments is executed.

[0153] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0154] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the method adopted, run or implemented by the application program stored therein.

[0155] It should be noted that:

[0156] The algorithm and display provided herein are not inherently related to any particular computer, virtual device or other equipment. Various general purpose devices can also be used together with the teachings based on this. According to the above description, it is obvious to construct the structure required for this type of device. In addition, the application is not directed to any specific programming language. It should be understood that the content of the application described herein can be realized using various programming languages, and the description of the specific language above is to disclose the best mode of implementation of the application.

[0157] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0158] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following intention: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0159] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition they may be divided into a plurality of submodules or subunits or subcomponents. Except that at least some of such features and / or processes or units are mutually exclusive, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this manner may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0160] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0161] The various component embodiments of the present application can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or digital signal processor (DSP) can be used in practice to implement some or all functions of some or all components in the creation device of the virtual machine according to the embodiment of the present application. The present application can also be implemented as a device or device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present application can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0162] It should be noted that the above embodiments illustrate the present application rather than limit the present application, and that those skilled in the art may design alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets should not be constructed as a limitation to the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of multiple such elements. The present application may be implemented by means of hardware including several different elements and by means of a suitably programmed computer. In a unit claim that lists several devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc. does not indicate any order. These words may be interpreted as names.

[0163] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

[0164] The above description is only a preferred embodiment of the present invention, and does not limit the patent scope of the present invention. All equivalent structural changes made by using the contents of the present invention specification and drawings under the concept of the present invention, or directly / indirectly applied in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A multimodal data processing method based on a scene graph, characterized in that: The method comprises: Obtain multimodal data to be processed and the corresponding application scenarios and application requirements; Based on the multimodal data to be processed and the application scenario, it is determined whether there is an associated ontology in the model warehouse; if not, the ontology required for the scene graph is constructed according to the application scenario; if it exists, it is determined whether it is necessary to complete the ontology on the associated ontology; if necessary, the ontology is completed on the associated ontology to obtain the ontology required for the scene graph; If the multimodal data is text data, the text data is input into the target model to obtain several target relations of the ontology; or, if the multimodal data is image data, the image data is input into the target model for data conversion to obtain text data; the text data is input into the target model to obtain several target relations of the ontology; Inputting the target model based on the target relationships to extract triple instances, and obtaining a plurality of scene graphs; the scene graphs are used to represent metadata tags of the multimodal data; The application requirements and the ontology are input into the target model to determine whether scene fusion is required; if scene fusion is not required, the target model directly outputs the ontology; if scene fusion is required, the application requirements and the ontology are input into the target model to determine fusion rules; based on the fusion rules, the several scene graphs are fused to obtain a fusion graph, and the fusion graph is stored in the model warehouse.

2. A multimodal data processing system based on a scene graph, characterized in that: The system comprises: A data acquisition module is used to acquire the multimodal data to be processed and the corresponding application scenarios and application requirements; An ontology determination module is used to determine whether there is an associated ontology in the model warehouse based on the multimodal data to be processed and the application scenario; if not, construct the ontology required for the scene graph according to the application scenario; if it exists, determine whether it is necessary to complete the ontology on the associated ontology; if necessary, complete the ontology on the associated ontology to obtain the ontology required for the scene graph; A scene graph determination module is used for, if the multimodal data is text data, inputting the text data into a target model to obtain a number of target relations of the ontology; or, if the multimodal data is image data, inputting the image data into the target model for data conversion to obtain text data; inputting the text data into the target model to obtain a number of target relations of the ontology; extracting triple instances based on the several target relations into the target model to obtain a number of scene graphs; the scene graphs are used to characterize metadata tags of the multimodal data; A fusion module is used to input the application requirements and the ontology into the target model to determine whether scene fusion is required; if scene fusion is not required, the target model directly outputs the ontology; if scene fusion is required, the application requirements and the ontology are input into the target model to determine the fusion rules; based on the fusion rules, the several scene graphs are fused to obtain a fusion map, and the fusion map is stored in the model warehouse.

3. A multimodal data virtualization system based on scene graph, characterized in that: The system comprises: An ontology construction module is used to determine whether there is an associated ontology in the model warehouse based on the multimodal data to be processed and the application scenario; if not, to construct the ontology required for the scene graph according to the application scenario; if it exists, to determine whether it is necessary to complete the ontology on the associated ontology; if necessary, to complete the ontology on the associated ontology to obtain the ontology required for the scene graph; A scene graph construction module is used to drive the data processing models for various modalities and fields in the model warehouse through the target model to realize the conversion of multimodal data into the form of a scene graph; if the multimodal data is text data, the text data is input into the target model to obtain several target relations of the ontology; or, if the multimodal data is image data, the image data is input into the target model for data conversion to obtain text data; the text data is input into the target model to obtain several target relations of the ontology; based on the several target relations, the target model is input to perform triple instance extraction to obtain several scene graphs; the scene graph is used to characterize the metadata tags of the multimodal data; The scene graph fusion module is used to intelligently determine whether and how to fuse scene graphs of multiple different business types of data based on business scenario needs through the target model; Model warehouse, including target model module, general image parsing module and general document parsing module; used to manage various data processing models required for metadata tagging, and also used to parse user query requests; The connector layer is used to connect to the physical data source through the data connector layer. The connector supports reading data from the data source into the memory and handing it over to the subsequent process for processing; API interface layer, used to support several APIs for accessing resources; The public basic layer includes data query module, query parsing module and data view module.

4. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to claim 1 when executing the computer program.

5. A computer-readable storage medium, characterized in that: Computer readable instructions are stored thereon, and the computer readable instructions can be executed by a processor to implement the method as claimed in claim 1.

Images