Target detection method and device based on multimodal fusion and dynamic knowledge graph

Through the multimodal fusion and dynamic knowledge graph methods, the problem that traditional object detection methods can only detect predefined categories is solved, and efficient detection of unknown categories of targets is achieved, which significantly improves detection performance and generalization capabilities.

CN118861324BActive Publication Date: 2024-12-03JIANGSU ZHENGFANG TRANSPORTATION TECH CO LTD
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411347670.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-26
Publication Date
2024-12-03
Estimated Expiration
2044-09-26

AI Technical Summary

Technical Problem

Traditional object detection methods can only detect targets in predefined categories, and cannot effectively detect unknown categories or emerging targets, which have limitations.

Method used

Using a method based on multimodal fusion and dynamic knowledge graph, we can obtain image, text and speech data for preprocessing and feature extraction, build knowledge graphs and dynamic updates, combine multi-layer perceptron to predict target categories, and dynamically expand the vocabulary to adapt to the new category.

Benefits of technology

Efficient detection of known and unknown categories targets is achieved, and the performance and generalization ability of open vocabulary target detection is significantly improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118861324B_ABST
    Figure CN118861324B_ABST
Patent Text Reader

Abstract

The present invention discloses a target detection method and device based on multimodal fusion and dynamic knowledge graph in the field of computer vision. The method comprises: obtaining image data to be identified and multimodal data related to the image data to be identified; preprocessing the multimodal data, extracting features to generate multimodal features; constructing a knowledge graph, fusing the multimodal features with the information of the knowledge graph, and generating a unified vector representation; performing target detection on the image data to be identified through a region proposal network to generate target candidate regions; performing bounding box regression to locate the target position for each target candidate region, and predicting the target category using a multilayer perceptron in combination with the unified vector representation; wherein, according to the unified vector representation, inferring potential categories related to the current prediction of the multilayer perceptron, adding the potential categories to a dynamic vocabulary, and expanding the prediction range of the multilayer perceptron; the present invention can realize efficient detection of targets of known categories and unknown categories.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a target detection method and device based on multimodal fusion and dynamic knowledge graph. Background Art

[0002] Object detection is an important task in the field of computer vision, which aims to identify and locate objects of a specific category from images or videos. Traditional object detection methods are usually based on the closed set assumption, that is, they can only detect objects of predefined categories. However, in practical applications, it is often necessary to detect unknown categories or newly emerging objects, and traditional object detection methods cannot be used normally. Summary of the invention

[0003] The purpose of the present invention is to overcome the deficiencies in the prior art and to provide a target detection method and device based on multimodal fusion and dynamic knowledge graphs, so as to solve the problem that traditional target detection methods can only detect targets of predefined categories and have certain limitations in use.

[0004] To achieve the above object, the present invention is implemented by adopting the following technical solutions:

[0005] In a first aspect, the present invention provides a target detection method based on multimodal fusion and dynamic knowledge graph, comprising:

[0006] Acquire image data to be identified and multimodal data related to the image data to be identified;

[0007] Preprocessing the multimodal data, and extracting features from the preprocessed multimodal data to generate multimodal features;

[0008] Constructing a knowledge graph, fusing the multimodal features with the information of the knowledge graph, and generating a unified vector representation;

[0009] Performing target detection on the image data to be identified by using a region proposal network to generate a target candidate region;

[0010] For each of the target candidate regions, bounding box regression is performed to locate the target position, and a multi-layer perceptron is used to predict the target category in combination with a unified vector representation;

[0011] The method of combining a unified vector representation with a multi-layer perceptron to predict the target category includes:

[0012] Designing a dynamic vocabulary mechanism for the multilayer perceptron, and inferring a potential category related to the current prediction of the multilayer perceptron based on a unified vector representation, wherein the potential category is a target category currently unknown to the multilayer perceptron;

[0013] The potential categories are added to a dynamic vocabulary to expand the prediction range of the multi-layer perceptron; and the target category of each target candidate area is predicted by the multi-layer perceptron.

[0014] Optionally, the multimodal data includes image data, text data and voice data;

[0015] The preprocessing of the multimodal data includes:

[0016] The preprocessing of the image data includes scaling, cropping and normalization;

[0017] The preprocessing of the text data includes word segmentation, stop word removal and lemma restoration;

[0018] The preprocessing of the voice data includes converting the voice data into voice text data and preprocessing the voice text data.

[0019] Optionally, the extracting features from the preprocessed multimodal data to generate multimodal features includes:

[0020] For the preprocessed image data, the image visual features are extracted through the pre-trained image network model;

[0021] For the preprocessed text data and speech data, the text semantic features are extracted through the pre-trained language network model;

[0022] For speech data, speech semantic features are extracted through a pre-trained speech network model.

[0023] Optionally, the constructing the knowledge graph includes:

[0024] Construct an initial knowledge graph based on a predefined ontology and a large-scale knowledge base; the ontology includes entities, attributes, and relationships, and the entities are target categories;

[0025] Design a dynamic update mechanism for the knowledge graph, which includes adding new entities, updating relationships, and resolving conflicts;

[0026] Obtain target detection results or external feedback information, and continuously update the initial knowledge graph based on the dynamic update mechanism of the knowledge graph.

[0027] Optionally, fusing the multimodal features with the information of the knowledge graph to generate a unified vector representation includes:

[0028] Design a feature alignment mechanism to map multimodal features to the semantic space of the knowledge graph;

[0029] The attention mechanism is used to calculate the correlation weights between modal features after the mapping operation;

[0030] Based on the correlation weights, a cross-modal Transformer encoder is used to perform deep feature fusion to generate a unified vector representation.

[0031] Optionally, for the potential category, the multilayer perceptron can be quickly adapted to its new category through a few-shot learning method or a zero-shot learning method.

[0032] In a second aspect, the present invention provides a target detection device based on multimodal fusion and dynamic knowledge graph, comprising:

[0033] A data acquisition module, configured to acquire image data to be identified and multimodal data related to the image data to be identified;

[0034] A preprocessing module is configured to preprocess the multimodal data and extract features from the preprocessed multimodal data to generate multimodal features;

[0035] A feature fusion module is configured to construct a knowledge graph, fuse the multimodal features with the information of the knowledge graph, and generate a unified vector representation;

[0036] A region candidate module is configured to perform target detection on the image data to be identified through a region proposal network to generate a target candidate region;

[0037] The category recognition module is configured to perform bounding box regression to locate the target position for each of the target candidate areas, and use a multi-layer perceptron to predict the target category in combination with a unified vector representation.

[0038] In a third aspect, the present invention provides an electronic device, including a processor and a storage medium;

[0039] The storage medium is used to store instructions;

[0040] The processor is used to operate according to the instructions to execute the steps according to the above method.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the above method when executed by a processor.

[0042] In a fifth aspect, the present invention provides a computer program product, comprising a computer program / instruction, which implements the steps of the above method when executed by a processor.

[0043] Compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention provides a method and device for object detection based on multimodal fusion and dynamic knowledge graph, which realizes efficient detection of objects of known and unknown categories by integrating multimodal information such as vision, text and speech, and combining with dynamically updated knowledge graph. The method can adaptively learn new categories and dynamically expand the vocabulary during the detection process, significantly improving the performance and generalization ability of open vocabulary object detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flowchart of a target detection method based on multimodal fusion and dynamic knowledge graph provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0046] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0047] Embodiment 1:

[0048] like Figure 1 As shown, an embodiment of the present invention provides a target detection method based on multimodal fusion and dynamic knowledge graph, comprising the following steps:

[0049] Step S1: Acquire image data to be identified and multimodal data related to the image data to be identified.

[0050] A common usage scenario is that when a user inputs image data to be recognized, he or she also performs operations such as speaking or keyboard input. Therefore, commonly used multimodal data are image data, text data, and voice data.

[0051] Step S2: preprocess the multimodal data, and extract features from the preprocessed multimodal data to generate multimodal features. Specifically in this embodiment, the multimodal data includes image data, text data, and voice data.

[0052] The preprocessing of image data includes scaling, cropping and normalization. For example, the image is scaled to a fixed size of 224*224 pixels, and data enhancement is performed, including random cropping, horizontal flipping, color jittering, etc. The pixel values ​​are normalized to a range between [-1, 1].

[0053] The preprocessing of text data includes word segmentation, stop word removal and lemma restoration; for example, using the NLTK toolkit for word segmentation, removing stop words and punctuation, and using WordNet for lemma restoration.

[0054] The preprocessing of speech data includes converting speech data into speech text data and preprocessing speech text data; for example, using Google Speech-to-Text API to convert speech into speech text data.

[0055] Specifically, extracting features from the preprocessed multimodal data to generate multimodal features includes:

[0056] For the preprocessed image data, the image visual features are extracted through the pre-trained image network model; for example, the pre-trained ViT-B / 16 model is used as the backbone to extract the visual features of the image, and the extracted feature dimension is 768.

[0057] For the preprocessed text data and speech data, the text semantic features are extracted through the pre-trained language network model; for example, the pre-trained BERT-base-uncased model is used to extract the semantic features of the text, and the output corresponding to the [CLS] tag is taken as the text feature with a dimension of 768.

[0058] For speech data, the speech semantic features are extracted through a pre-trained speech network model (such as Wav2Vec 2.0); for example, the pre-trained Wav2Vec 2.0 model is used to extract the acoustic features of the speech, and the extracted features are average pooled to obtain a feature vector of fixed dimension (768).

[0059] Step S3: construct a knowledge graph, fuse the multimodal features with the information of the knowledge graph, and generate a unified vector representation;

[0060] Building a knowledge graph includes:

[0061] Build an initial knowledge graph based on predefined ontologies and large-scale knowledge bases; ontologies include entities, attributes, and relationships, and entities are target categories; for example, build an initial knowledge graph based on ConceptNet and Wikidata;

[0062] Design a dynamic update mechanism for the knowledge graph, which includes adding new entities, updating relationships, and resolving conflicts. For example, use Neo4j graph database to store and manage the knowledge graph. Use a confidence mechanism to resolve potential knowledge conflicts. Regularly compress and optimize the knowledge graph, and delete low-confidence and low-frequency nodes and edges.

[0063] Obtain target detection results or external feedback information, and continuously update the initial knowledge graph based on the dynamic update mechanism of the knowledge graph.

[0064] Knowledge query can also realize efficient retrieval interface based on Cypher query language, and support complex semantic query, such as path query, similarity query, etc.

[0065] Specifically, the multimodal features are integrated with the information of the knowledge graph to generate a unified vector representation including:

[0066] Design a feature alignment mechanism to map multimodal features to the semantic space of the knowledge graph;

[0067] The attention mechanism is used to calculate the correlation weights between modal features after the mapping operation;

[0068] Based on the correlation weight, a cross-modal Transformer encoder is used to perform deep feature fusion and generate a unified vector representation.

[0069] A 6-layer Transformer encoder structure is used to achieve deep feature fusion. Each layer contains a multi-head self-attention sublayer and a feedforward neural network sublayer. LayerNorm and residual connections are used to ensure training stability.

[0070] Step S4: Perform target detection on the image data to be identified through a region proposal network to generate target candidate regions.

[0071] For example, the improved RPN network is used to generate high-quality candidate regions, and the multi-scale feature pyramid is used to improve the detection capability of objects of different sizes.

[0072] Step S5: For each target candidate area, perform bounding box regression to locate the target position, and use a multi-layer perceptron to predict the target category in combination with a unified vector representation;

[0073] Among them, the target category prediction using a multi-layer perceptron combined with a unified vector representation includes:

[0074] Design a dynamic vocabulary mechanism for the multi-layer perceptron, and infer the latent categories related to the current prediction of the multi-layer perceptron based on the unified vector representation. The latent categories are the target categories that are currently unknown to the multi-layer perceptron.

[0075] The latent categories are added to the dynamic vocabulary to expand the prediction range of the multi-layer perceptron; the target category of each target candidate area is predicted by the multi-layer perceptron.

[0076] For latent categories, the multilayer perceptron can be quickly adapted to its new categories through few-shot learning or zero-shot learning.

[0077] Embodiment 2:

[0078] Based on the target detection method provided in Example 1, the embodiment of the present invention provides a target detection device based on multimodal fusion and dynamic knowledge graph, including:

[0079] A data acquisition module, configured to acquire image data to be identified and multimodal data related to the image data to be identified;

[0080] A preprocessing module is configured to preprocess the multimodal data and extract features from the preprocessed multimodal data to generate multimodal features;

[0081] The feature fusion module is configured to construct a knowledge graph, fuse the multimodal features with the information of the knowledge graph, and generate a unified vector representation;

[0082] A region proposal module is configured to generate target candidate regions by performing target detection on the image data to be identified through a region proposal network;

[0083] The category recognition module is configured to perform bounding box regression to locate the target position for each target candidate area, and use a multi-layer perceptron to predict the target category in combination with a unified vector representation.

[0084] Embodiment three:

[0085] Based on the target detection method provided in the first embodiment, the embodiment of the present invention provides an electronic device, including a processor and a storage medium;

[0086] The storage medium is used to store instructions;

[0087] The processor is used to operate according to the instructions to execute the steps according to the above method.

[0088] Embodiment 4:

[0089] Based on the target detection method provided in the first embodiment, the embodiment of the present invention provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps of the above method are implemented.

[0090] Embodiment five:

[0091] Based on the target detection method provided in the first embodiment, a computer program product according to the embodiment of the present invention includes a computer program / instruction, and the steps of the above method are implemented when the computer program / instruction is executed by a processor.

[0092] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0096] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A target detection method based on multimodal fusion and dynamic knowledge graph, characterized in that: include: Acquire image data to be identified and multimodal data related to the image data to be identified; Preprocessing the multimodal data, and extracting features from the preprocessed multimodal data to generate multimodal features; Constructing a knowledge graph, fusing the multimodal features with the information of the knowledge graph, and generating a unified vector representation; Performing target detection on the image data to be identified by using a region proposal network to generate a target candidate region; For each of the target candidate regions, bounding box regression is performed to locate the target position, and a multi-layer perceptron is used to predict the target category in combination with a unified vector representation; The method of combining a unified vector representation with a multi-layer perceptron to predict the target category includes: Designing a dynamic vocabulary mechanism for the multilayer perceptron, and inferring a potential category related to the current prediction of the multilayer perceptron based on a unified vector representation, wherein the potential category is a target category currently unknown to the multilayer perceptron; The potential categories are added to a dynamic vocabulary to expand the prediction range of the multi-layer perceptron; and the target category of each target candidate area is predicted by the multi-layer perceptron.

2. The target detection method based on multimodal fusion and dynamic knowledge graph according to claim 1, characterized in that: The multimodal data includes image data, text data and voice data; The preprocessing of the multimodal data includes: The preprocessing of the image data includes scaling, cropping and normalization; The preprocessing of the text data includes word segmentation, stop word removal and lemma restoration; The preprocessing of the voice data includes converting the voice data into voice text data and preprocessing the voice text data.

3. The target detection method based on multimodal fusion and dynamic knowledge graph according to claim 1, characterized in that: The step of extracting features from the preprocessed multimodal data to generate multimodal features comprises: For the preprocessed image data, the image visual features are extracted through the pre-trained image network model; For the preprocessed text data and speech data, the text semantic features are extracted through the pre-trained language network model; For speech data, speech semantic features are extracted through a pre-trained speech network model.

4. The target detection method based on multimodal fusion and dynamic knowledge graph according to claim 1, characterized in that: The construction of the knowledge graph includes: Construct an initial knowledge graph based on a predefined ontology and a large-scale knowledge base; the ontology includes entities, attributes, and relationships, and the entities are target categories; Design a dynamic update mechanism for the knowledge graph, which includes adding new entities, updating relationships, and resolving conflicts; Obtain target detection results or external feedback information, and continuously update the initial knowledge graph based on the dynamic update mechanism of the knowledge graph.

5. The target detection method based on multimodal fusion and dynamic knowledge graph according to claim 1, characterized in that: The multimodal features are integrated with the information of the knowledge graph to generate a unified vector representation. include: Design a feature alignment mechanism to map multimodal features to the semantic space of the knowledge graph; The attention mechanism is used to calculate the correlation weights between modal features after the mapping operation; Based on the correlation weights, a cross-modal Transformer encoder is used to perform deep feature fusion to generate a unified vector representation.

6. The target detection method based on multimodal fusion and dynamic knowledge graph according to claim 1, characterized in that: For the potential category, the multilayer perceptron is made to quickly adapt to its new category through a few-shot learning method or a zero-shot learning method.

7. An object detection device based on multimodal fusion and dynamic knowledge graph, characterized in that: include: A data acquisition module, configured to acquire image data to be identified and multimodal data related to the image data to be identified; A preprocessing module is configured to preprocess the multimodal data and extract features from the preprocessed multimodal data to generate multimodal features; A feature fusion module is configured to construct a knowledge graph, fuse the multimodal features with the information of the knowledge graph, and generate a unified vector representation; A region candidate module is configured to perform target detection on the image data to be identified through a region proposal network to generate a target candidate region; A category recognition module is configured to perform bounding box regression to locate the target position for each target candidate area, and use a multi-layer perceptron to predict the target category in combination with a unified vector representation; The method of combining a unified vector representation with a multi-layer perceptron to predict the target category includes: Designing a dynamic vocabulary mechanism for the multilayer perceptron, and inferring a potential category related to the current prediction of the multilayer perceptron based on a unified vector representation, wherein the potential category is a target category currently unknown to the multilayer perceptron; The potential categories are added to a dynamic vocabulary to expand the prediction range of the multi-layer perceptron; and the target category of each target candidate area is predicted by the multi-layer perceptron.

8. An electronic device, characterized in that: including processor and storage medium; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Citation Information

Patent Citations

  • Image retrieval method based on visual description statements

    CN112256904A

  • Target detection method and device, electronic equipment and storage medium

    CN113887423A