Image-text question answering method and device based on target detection and rule enhancement and electronic equipment

Abstract

The invention relates to the technical field of data processing, and discloses a method and device for enhancing image-text questioning and answering based on target detection and rules and electronic equipment, and the method comprises the steps: obtaining and storing a plurality of basic images of a target object; inputting the basic image into a pre-constructed target detection model, and extracting first feature data of the basic image; inputting the first feature data into a pre-constructed rule enhancement model, and extracting second feature data of the basic image; converting the second feature data into a text description statement; generating a plurality of image-text description pairs according to the plurality of basic images and the plurality of text description statements; and in response to the received question information for the target object, converting the question information into a target semantic vector, retrieving a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generating reply information for the question information according to the target image-text description pair. According to the method, deep understanding and semantic-level question answering of the image content can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a target detection and rule enhanced image-text question answering method, a target detection and rule enhanced image-text question answering device, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the rapid development of artificial intelligence, computer vision and LLM (Large Language Model), image understanding and intelligent question answering technology has gradually attracted widespread attention in the industrial field. In the actual scenes of power inspection, traffic security, industrial quality inspection, etc., the current image-based target detection algorithm can identify various targets such as equipment defects, structural abnormalities, etc. However, most of the current systems can only output structured detection results, lack of deep understanding of image semantic content, and are difficult to meet the complex application requirements of users such as "image question answering" or "knowledge assisted diagnosis". In recent years, the proposal of RAG (Retrieval-Augmented Generation) technology provides a new idea to solve this problem. By introducing external knowledge into the LLM inference process, RAG technology can combine image context and domain knowledge in question answering to achieve more targeted and accurate responses. In power image recognition, the detection results and image information are fused to construct a multi-modal description, and then the defect rule knowledge is used for enhanced question answering, which not only improves the accuracy of question answering, but also conforms to the actual use habits of industry users.

[0003] However, in the related art, the construction of image-text description, defect knowledge association, question answering logic organization, etc. is still relatively preliminary, and there is a lack of unified and effective engineering solutions. In particular, in the target detection task, how to convert the structured inference results into information that can be used for knowledge retrieval and natural language interaction still presents challenges. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, the first object of the present application is to propose a target detection and rule enhanced image-text question answering method, which can realize deep understanding of image content and semantic level question answering, automatically analyze the structural information and semantic relationship contained in the image, convert the image content into natural language expression, and drive the question answering process through multi-modal vectorization technology, and is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis, which have high requirements for image understanding and question answering accuracy.

[0005] The second object of the present application is to propose a target detection and rule enhanced image-text question answering device.

[0006] The third object of the present application is to propose an electronic device.

[0007] A fourth object of the present application is to propose a computer-readable storage medium.

[0008] To achieve the above object, the first aspect of the present application proposes a method for question and answer based on target detection and rule enhancement, comprising: acquiring and storing a plurality of basic images of a target object; inputting the basic images into a pre-constructed target detection model to extract first feature data of the basic images; wherein the first feature data represents explicit features directly expressed by the basic images; inputting the first feature data into a pre-constructed rule enhancement model to extract second feature data of the basic images; wherein the second feature data represents implicit features not directly expressed by the basic images and relationships between different explicit features; converting the second feature data into text description sentences; generating a plurality of image-text description pairs according to the plurality of basic images and the plurality of text description sentences; wherein an image-text description pair includes at least one basic image and at least one text description sentence, and the basic image and the text description sentence correspond one by one; in response to receiving question information for the target object, converting the question information into a target semantic vector, retrieving a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generating reply information for the question information according to the target image-text description pair; wherein the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold.

[0009] In addition, the method for question and answer based on target detection and rule enhancement according to the above embodiments of the present application can have the following additional technical features: According to some embodiments of the present application, acquiring and storing a plurality of basic images of a target object comprises: image acquisition of the target object by a front-end acquisition device to obtain a plurality of basic images of the target object and metadata information of each basic image; hierarchical storage of the basic images according to the metadata information.

[0010] According to some embodiments of the present application, inputting the basic images into a pre-constructed target detection model to extract first feature data of the basic images comprises: detecting target detection objects in the basic images; extracting bounding box coordinates, class names and confidence of the target detection objects; filling the bounding box coordinates, class names and confidence into a pre-constructed structured template to obtain the first feature data.

[0011] According to some embodiments of the present application, the method further comprises: constructing a typical detection object combination for the target detection object, determining a structure mode corresponding to the typical detection object combination, and constructing a structure template library according to the typical detection object combination and the structure mode; wherein the typical detection object combination is used to describe the combination relationship of different target detection objects, and the structure mode is used to describe the structure corresponding to the combination relationship of different target detection objects; defining a spatial constraint condition for the target detection object and a logical constraint condition when the target detection object is combined, and constructing a condition constraint set according to the spatial constraint condition and the logical constraint condition; defining a priority constraint condition when a conflict occurs in the process of determining the structure mode corresponding to the typical detection object combination; and constructing a rule enhancement model according to the structure template library, the condition constraint set and the priority constraint condition.

[0012] According to some embodiments of the present application, the first feature data is input into the pre-constructed rule enhancement model, and the second feature data of the basic image is extracted, comprising: determining the combination relationship between different target detection objects and the structure corresponding to the combination relationship under the constraint of the condition constraint set and the priority constraint condition, to obtain the second feature data.

[0013] According to some embodiments of the present application, the method further comprises: determining whether there is an object associated with the target detection object and having an association degree greater than a preset association threshold and not being detected based on the spatial relationship of the target detection object and the scene information of the target object; and defining the object as the target detection object in response to the existence of the object associated with the target detection object and having an association degree greater than the preset association threshold and not being detected.

[0014] According to some embodiments of the present application, the reply information for the question information is generated according to the target text description, comprising: constructing a structured prompt word according to the target text description and the question information, inputting the prompt word into a pre-set large language model, and obtaining the reply information and the reference evidence information associated with the reply information.

[0015] According to the target detection and rule enhancement image-text question and answer method based on the embodiment of the application, a plurality of basic images of a target object are acquired and stored; the basic images are input into a pre-constructed target detection model to extract first feature data of the basic images; wherein the first feature data represents explicit features directly expressed by the basic images; the first feature data is input into a pre-constructed rule enhancement model to extract second feature data of the basic images; wherein the second feature data represents implicit features and relationships between different explicit features that are not directly expressed by the basic images; the second feature data is converted into text description sentences; a plurality of image-text description pairs are generated according to the plurality of basic images and the plurality of text description sentences; wherein an image-text description pair includes at least one basic image and at least one text description sentence, and the basic image and the text description sentence correspond to each other; in response to receiving question information for the target object, the question information is converted into a target semantic vector, a target image-text description pair is retrieved from the plurality of image-text description pairs based on the target semantic vector, and reply information for the question information is generated according to the target image-text description pair; wherein the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold. Thus, the method can realize deep understanding of image content and semantic-level question and answer, automatically analyze structural information and semantic relationships contained in the image, convert image content into natural language expression, and drive the question and answer process through multi-modal vectorization technology, and is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis that have high requirements for image understanding and question and answer accuracy.

[0016] A second object of the application is to provide a target detection and rule enhancement image-text question and answer device, which can realize deep understanding of image content and semantic-level question and answer, automatically analyze structural information and semantic relationships contained in the image, convert image content into natural language expression, and drive the question and answer process through multi-modal vectorization technology, and is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis that have high requirements for image understanding and question and answer accuracy.

[0017] To achieve the above object, the second aspect of the present application provides a device for question and answer based on target detection and rule enhancement, comprising: an image acquisition and uploading module configured to acquire and store a plurality of basic images of a target object; an image detection and structured output module configured to input the basic images into a pre-constructed target detection model to extract first feature data of the basic images; wherein the first feature data represents explicit features directly expressed by the basic images; a rule-based target relationship analysis and completion module configured to input the first feature data into a pre-constructed rule enhancement model to extract second feature data of the basic images; wherein the second feature data represents implicit features not directly expressed by the basic images and relationships between different explicit features; a natural language description generation module configured to convert the second feature data into text description sentences; a graphic-text information vectorization storage and index construction module configured to generate a plurality of graphic-text description pairs according to the plurality of basic images and the plurality of text description sentences; wherein each graphic-text description pair comprises at least one basic image and at least one text description sentence, and the basic image and the text description sentence correspond to each other; a question-driven vector retrieval and question and answer generation module configured to, in response to receiving question information about the target object, convert the question information into a target semantic vector, retrieve a target graphic-text description pair from the plurality of graphic-text description pairs based on the target semantic vector, and generate reply information for the question information according to the target graphic-text description pair; wherein the vector similarity between the target graphic-text description pair and the target semantic vector is greater than a preset similarity threshold.

[0018] According to the target detection and rule enhancement image-text question and answer device, the image content can be understood in depth and at a semantic level, the structural information and the semantic relationship contained in the image can be automatically analyzed, the image content can be converted into natural language expression, the question and answer process can be driven through the multi-modal vectorization technology, and the device is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis, which have high requirements on image understanding and question and answer accuracy.

[0019] To achieve the above object, the third aspect of the present application provides an electronic device, comprising: a processor and a memory, the memory stores programs or instructions executable on the processor, and the programs or instructions are executed by the processor to implement the steps of the above-mentioned target detection and rule enhancement image-text question and answer method.

[0020] According to the electronic device, by executing the above-mentioned target detection and rule enhancement image-text question and answer method, the image content can be understood in depth and at a semantic level, the structural information and the semantic relationship contained in the image can be automatically analyzed, the image content can be converted into natural language expression, the question and answer process can be driven through the multi-modal vectorization technology, and the device is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis, which have high requirements on image understanding and question and answer accuracy.

[0021] To achieve the above object, the fourth aspect of the present application provides a computer readable storage medium, and the readable storage medium stores programs or instructions, and the programs or instructions are executed by a processor to realize the steps of the target detection and rule enhanced graphic-text question answering method.

[0022] The computer readable storage medium according to the embodiment of the present application can realize deep understanding of image content and semantic level question answering by executing the target detection and rule enhanced graphic-text question answering method, automatically analyzes structural information and semantic relationship contained in the image, converts the image content into natural language expression, and drives the question answering process through the multi-modal vectorization technology, and is suitable for power inspection, equipment identification, structural relationship analysis and other scenes with high requirements for image understanding and question answering accuracy.

[0023] Additional aspects and advantages of the present application will be described in the following description, some of which will become apparent to those skilled in the art from the following description, or will be learned from practice of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 Flow chart of the target detection and rule enhanced graphic-text question answering method according to some embodiments of the present application; Figure 2 Flow chart of the target detection and rule enhanced graphic-text question answering method according to some other embodiments of the present application; Figure 3 Schematic diagram of the target detection and rule enhanced graphic-text question answering device according to some embodiments of the present application; Figure 4 Block schematic diagram of the electronic device according to some embodiments of the present application. DETAILED DESCRIPTION

[0025] To make the object, technical scheme and advantages of the present application clearer, further detailed description will be made in combination with specific embodiments and with reference to the drawings.

[0026] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the present application shall be understood as their common meanings to those having ordinary skills in the art to which the present application pertains. The terms "first", "second" and similar terms used in the embodiments of the present application do not denote any order, quantity or importance, but are only used to distinguish different components. The terms "comprising" or "including" and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connected" or "connected" and similar terms are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to represent relative positional relationships, which may change accordingly when the absolute position of the described object changes.

[0027] As described in the background section, with the rapid development of artificial intelligence, computer vision and LLM, image understanding and intelligent question answering technology has gradually attracted widespread attention in the industrial field. In practical scenarios such as power inspection, traffic security, industrial quality inspection, etc., current image-based target detection algorithms can already identify various targets such as equipment defects, structural abnormalities, etc. However, most current systems can only output structured detection results (such as detection boxes and categories), lack deep understanding of image semantic content, and are difficult to meet the complex application needs of users "image question answering" or "knowledge-assisted diagnosis". In recent years, the introduction of RAG technology provides a new way to solve this problem. By introducing external knowledge into the LLM inference process, RAG technology can combine image context and domain knowledge in question answering to achieve more targeted and accurate responses. In power image recognition, the fusion of detection results and image information, the construction of multi-modal description, and the enhancement of question answering with defect rule knowledge can not only improve the accuracy of question answering, but also better meet the actual use habits of industry users.

[0028] In addition, the development of multi-modal large models (such as Qwen-VL, ChatGLM-Vision, etc.) makes it possible to jointly model text and images. Compared to NLP models that only process pure text input, multi-modal models with image understanding capabilities can perform content completion, diagnosis suggestions, and other tasks based on image content, significantly improving the intelligence level of the system.

[0029] In related technologies, the construction of image-text description, defect knowledge association, and question answering logic organization is still relatively primitive, and there is a lack of unified and effective engineering solutions. In particular, in the target detection task, how to convert structured inference results into information that can be used for knowledge retrieval and natural language interaction still presents challenges. Therefore, there is an urgent need for an integrated system that combines target detection, multi-modal representation, knowledge enhancement, and question answering generation to support more intelligent, reliable, and usable image-text question answering functions.

[0030] The applicant found in the process of implementing the present application that, in the related art, the fusion of image understanding and language generation has become an important direction of multi-modal intelligent question and answer systems. In the task of combining target detection and question and answer, the general technical path is to first identify and classify the region of interest in the image through a target detection model, generate a structured detection result or a brief textual description, and then combine it with the user's natural language question to complete the response generation to the question. The implementation of such systems generally includes three types: one is to rely on the detection result to construct a fixed format template answer, which is suitable for closed question and answer; two is to encode the image or its detection text representation through a vectorization model to construct a retrieval augmented generation (RAG) system, retrieve knowledge through semantic similarity with the user's question, and then generate an answer by a language model; three is to directly use a multi-modal large language model with image-text alignment capability to output a natural language answer based on the input image and question. Although these methods have promoted the fusion of image understanding and language question and answer to some extent, there are still technical bottlenecks in many aspects.

[0031] Firstly, in terms of modality fusion, most systems use a serial processing flow, and there is a lack of end-to-end optimization between image information and language models, resulting in insufficient utilization of image detection results in language generation, such as generalization of generated sentences and lack of targeted description. Secondly, current models generally have difficulty in introducing industry knowledge and cannot reason or propose processing suggestions based on professional rules, resulting in poor explainability and usability of the answers. In addition, vector retrieval systems often only consider single modality information of images or texts when constructing indexes, ignoring joint semantic expression of images and texts, which reduces the semantic matching degree of retrieval results with user questions and ultimately reduces the quality of question and answer. Furthermore, multi-modal large models often show understanding bias or ambiguous answers when dealing with professional domain problems due to lack of training corpus support, making it difficult to meet the needs of defect type, hazard analysis and rectification suggestions in industrial scenarios.

[0032] In summary, the current technology is still difficult to effectively unify semantic understanding of image detection results, knowledge enhancement and natural language interaction, especially in professional fields, facing complex defect scenarios, unstructured questions and the need for standardized processing suggestions, the current methods cannot provide accurate, reliable and explainable question and answer results. Therefore, there is an urgent need for an intelligent system that has image-text collaborative expression capability, supports vector semantic retrieval and knowledge rule enhancement, and can combine language models for question and answer generation to meet the actual application needs of high reliability, strong interactivity and multi-modal response.

[0033] Embodiments of the present application provide a target detection and rule enhanced image-text question answering method, a target detection and rule enhanced image-text question answering device, an electronic device and a computer readable storage medium.

[0034] Reference Figure 1 A flowchart of the target detection and rule enhanced image-text question answering method according to some embodiments of the present application is shown in FIG. 1.

[0035] As Figure 1 shown, the target detection and rule enhanced image-text question answering method according to some embodiments of the present application can include the following steps: S101, obtaining and storing a plurality of basic images of a target object.

[0036] Specifically, the target object (such as a power plant, a photovoltaic plant, etc.) can be imaged by a mobile terminal APP, a drone, a high-definition monitoring camera, etc., to obtain all the basic images of the target object. The all basic images of the target object are uploaded to a backend server through an image uploading interface to provide a data basis for subsequent image archiving management and accurate semantic positioning. Then, the basic images are stored in a hierarchical manner. It can be understood that the uploaded basic images will be automatically archived in an intelligent file management system to ensure the orderly management and quick retrieval of data.

[0037] S102, inputting the basic images into a pre-constructed target detection model to extract first feature data of the basic images; wherein the first feature data represents the explicit features directly expressed by the basic images.

[0038] Specifically, after obtaining the plurality of basic images of the target object, the basic images are inputted into the pre-constructed target detection model to extract the first feature data of the basic images from the pre-constructed target detection model. The first feature data represents the explicit features (such as bounding box coordinates, class names, etc.) directly expressed by the basic images. The pre-constructed target detection model can be a lightweight deep learning target detection model. The pre-constructed target detection model includes advanced architectures such as YOLOv5 / YOLOv8, RT-DETR, Swin Transformer, etc., and is implemented by ONNX labeling format or TensorRT deep optimization to realize model inference acceleration. The output result of the target detection model is standardized JSON format structured data. The structured output provides a basic semantic unit for subsequent rule enhancement processing and text generation.

[0039] Among them, YOLOv5 / YOLOv8 (target detection algorithm) directly predicts the target category and position end to end (fast speed), YOLOv5 uses pre-defined anchors, YOLOv8 cancels the anchor, and simplifies the design. RT-DETR (Real-Time Detection Transformer) uses the Hungarian algorithm to directly predict target-query matching, without NMS (Non-Maximum Suppression) post-processing, avoiding the redundant frame suppression problem of traditional detectors. Swin Transformer (Shifted Window Transformer) can divide the image into blocks, calculate self-attention in the local window, reduce the computational complexity, and interact across windows to improve global modeling capability. ONNX (Open Neural Network Exchange) is used to represent deep learning models, which allows models trained in different frameworks (such as PyTorch, TensorFlow, MXNet, etc.) to be converted and shared with each other, solving the compatibility problem between frameworks. TensorRT is a high-performance deep learning inference engine developed by NVIDIA, designed for GPU (Graphics Processing Unit) acceleration. It significantly improves inference speed through techniques such as optimizing model structure, quantization, and kernel automatic tuning.

[0040] S103, input the first feature data to the pre-constructed rule enhancement model to extract the second feature data of the basic image; wherein the second feature data represents the hidden features and the relationship between different explicit features that are not directly expressed by the basic image.

[0041] Specifically, after obtaining the first feature data, the first feature data is input into the pre-constructed rule enhancement model, for example, the detected target list (such as ["pole tower", "wire", "insulator"]) and the attributes of each target (such as bounding box coordinates, category, confidence, etc.) are input into the pre-constructed rule enhancement model, and the second feature data (i.e. enhanced structured features) of the basic image is extracted, for example, the second feature data can include the combination relationship between targets (such as "jueyuanzi+daoxian→suspension relationship") and the priority label of the relationship (such as "the relationship triggered by the safety rule is prior to the geometric rule"). A comprehensive structured rule knowledge base can be constructed, combining the spatial position relationship of the target in the detection result, the category combination mode and other multi-dimensional features, intelligently inferring the structural relationship not explicitly expressed or completing the missing information, and improving the understanding ability of the complex structural relationship in the image.

[0042] S104, convert the second feature data into a text description sentence.

[0043] Specifically, after obtaining the second feature data (i.e., the enhanced structured semantic information), a rule-driven natural language generation engine is used to convert the second feature data into a fluent text description sentence. The generation engine uses a template filling and sentence intelligent transformation strategy to ensure that the generated content has both structural integrity and natural language expression. For example, “The figure contains a tower and two conductors, the conductors are drawn from both sides of the tower, and the whole constitutes a typical distribution line structure.” “The conductor is detected to be connected to the tower, and the tower is located in the upper left corner of the image.” The detection results can be automatically converted into natural language descriptions, supporting bilingual generation in Chinese and English, and the language output mode can be flexibly switched according to the actual deployment scenario. To further improve the quality of the text, post-processing functions such as deduplication processing, entity naming standardization, and phrase splicing optimization are integrated.

[0044] S105, generating a plurality of image-text description pairs according to the plurality of base images and the plurality of text description sentences; wherein the image-text description pair includes at least one base image and at least one text description sentence, and the base image and the text description sentence correspond one by one.

[0045] Specifically, after obtaining the plurality of base images and the plurality of text description sentences, a plurality of labeled image-text description pairs are constructed according to the plurality of base images and the plurality of text description sentences to serve as the core semantic index basis for the downstream question and answer engine. A multi-modal encoding model (such as CLIP (Contrastive Language-Image Pretraining) or Fine-tuned CLIP (Fine-tuned Contrastive Language-Image Pretraining Model)) is used to perform high-dimensional vectorization embedding on the image-text description pair, intelligently fuse the image encoding vector and the text description encoding vector, and generate a unified high-quality semantic vector representation.

[0046] The vector storage uses high-performance vector databases (such as FAISS and Milvus) and supports the construction of inverted indexes and HNSW (Hierarchical Navigable Small World) graph structure indexes, significantly improving the retrieval speed and accuracy in large-scale image libraries. Each image record also contains rich meta-information (such as task ID (Identity Document), timestamp, detection quality score, etc.), facilitating subsequent accurate screening and quality evaluation. The image and its reasoning result can be used to generate a fusion semantic text embedding as an index item, ensuring that the retrieved content is consistent with the actual context of visual detection, improving semantic matching effect, supporting asynchronous updating and batch refreshing mechanism of vector index, and being suitable for large-scale image corpus scenarios with continuous growth.

[0047] wherein, FAISS and Milvus are both open source tools for efficient vector similarity search and vector database, widely used in recommendation systems, image retrieval, natural language processing and other scenarios that need to process high-dimensional vector data. FAISS is a high-performance vector similarity search library suitable for large-scale vector retrieval; Milvus is a distributed vector database that supports storing, retrieving and managing vector data.

[0048] S106, in response to receiving the question information for the target object, converting the question information into a target semantic vector, retrieving a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generating reply information for the question information according to the target image-text description pair; wherein the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold.

[0049] Specifically, the user inputs the question information for the target object (such as "Which pole does this wire connect to?" "Is there an abnormal device in the figure?") in the system interface. When the question information for the target object is received, the text semantic encoding model (for example, BERT, CoSENT, etc.) is first called to convert the question information into a target semantic vector. Subsequently, the system retrieves the image-text pair closest to the question semantics from the plurality of image-text description pairs based on the target semantic vector similarity in the pre-constructed image-text multi-modal vector library, and then selects the top K (default K=3) image-text pairs with the highest semantic relevance from the image-text pair closest to the question semantics. The image-text pair is taken as the target image-text description pair and is input as context. The reply information for the question information is generated according to the target image-text description pair to generate an accurate and natural answer text.

[0050] Among them, BERT (Bidirectional Encoder Representations from Transformers) is to analyze the context before and after the text through the Transformer encoder, solve the one-way limitation of traditional models (such as Word2Vec). CoSENT (Cosine Sentence Embedding Network) is to directly optimize the semantic similarity of sentence embedding through the cosine similarity loss function.

[0051] In some embodiments of the present application, a plurality of basic images of a target object are acquired and stored, comprising: acquiring images of the target object through a front-end acquisition device to obtain a plurality of basic images of the target object and metadata information of each basic image; and storing the basic images according to the metadata information.

[0052] Specifically, the front-end acquisition device (such as a mobile terminal APP, a drone, a high-definition monitoring camera, etc.) is used to acquire images of the target object (such as a power plant, a photovoltaic plant, etc.), and all basic images of the target object and metadata information of each basic image are acquired, wherein the metadata information can include structured information such as device GPS (Global Positioning System) position coordinates (such as longitude, latitude, and altitude), shooting angle parameters (such as pitch angle, azimuth angle, and roll angle), task number, device model, and timestamp (accurate to milliseconds, ensuring time sequence consistency). The all basic images of the target object and the metadata information of each basic image are uploaded to the back-end server through an image upload interface to provide a data basis for subsequent image archiving management and accurate semantic positioning, wherein the image upload interface supports carrying rich metadata information. The basic images are stored according to the metadata information, which can be understood as the uploaded basic images being automatically archived in an intelligent file management system using a three-level hierarchical structure of "project-task-device (such as project A task 001 device 123)", so as to ensure the orderly management and rapid retrieval of data. After the archiving of the basic images is completed, the intelligent file management system will automatically trigger the subsequent intelligent recognition processing flow.

[0053] In some embodiments of the present application, the basic image is input into a pre-constructed target detection model to extract first feature data of the basic image, comprising: detecting a target detection object in the basic image; extracting the bounding box coordinates, class name, and confidence of the target detection object; filling the bounding box coordinates, class name, and confidence into a pre-constructed structured template to obtain the first feature data.

[0054] Specifically, after obtaining all the basic images of the target object, the image recognition module calls the target detection model to automatically analyze and process the contents of all the basic images, detects a plurality of target detection objects from all the basic images, and then extracts the bounding box coordinates (bbox), category name (category) and confidence (score) from each target detection object, fills the bounding box coordinates, category name and confidence into a pre-constructed structured template, for example, the pre-constructed structured template can be: [{“bbox”:[],“category”:“ ”,“score”:}, {“bbox”:[],“category”:“ ”,“score”:}]; obtain the first feature data, that is: [{“bbox”:[100,200,50,80],“category”:“ganta”,“score”:0.94}, {“bbox”:[150,220,30,30],“category”:“daoxian”,“score”:0.87}].

[0055] Exemplarily, when the confidence is less than a certain value, the confidence can be deleted to improve the accuracy of the first feature data.

[0056] In some embodiments of the present application, the above method further comprises: constructing a typical detection object combination for the target detection object, determining a structure mode corresponding to the typical detection object combination, and constructing a structure template library according to the typical detection object combination and the structure mode; wherein the typical detection object combination is used to describe the combination relationship of different target detection objects, and the structure mode is used to describe the structure corresponding to the combination relationship of different target detection objects; defining a spatial constraint condition for the target detection object and a logical constraint condition when the target detection object is combined, and constructing a condition constraint set according to the spatial constraint condition and the logical constraint condition; defining a priority constraint condition when a conflict occurs in the process of determining the structure mode corresponding to the typical detection object combination; and constructing a rule enhancement model according to the structure template library, the condition constraint set and the priority constraint condition.

[0057] Exemplarily, taking the power scene as an example, a typical detection object combination for a target detection object is constructed, for example, ganta+daoxian, jueyuanzi+daoxian, a structure mode corresponding to the typical detection object combination is determined, for example, a tower-conductor composite structure, a suspension relationship, a structure template library is constructed according to the typical detection object combination and the structure mode, for example, ganta+daoxian→tower-conductor composite structure, jueyuanzi+daoxian→suspension relationship, the construction of the structure template library can convert industry knowledge into computable rule templates to guide the system to recognize common combination relationships between targets. Spatial constraint conditions for the target detection object are defined, wherein the spatial constraint conditions include geometric relationship constraints and direction constraints, for example, the geometric relationship constraint can be that the intersection union ratio of the conductor bounding box and the tower bounding box is greater than 0.1, and the direction constraint can be that the inclination angle of the conductor deviates from the main direction of the tower by less than 30°, and logical constraint conditions when the target detection object is combined are defined, wherein the logical constraint conditions include existence constraints and uniqueness constraints, for example, the existence constraint can be that if daoxian is detected, at least one ganta exists, and the uniqueness constraint can be that a single daoxian is usually connected to only one main ganta, a condition constraint set is constructed according to the spatial constraint conditions and the logical constraint conditions, for example, there must be at least one ganta target, and the position thereof should be higher than the center of daoxian or there is a certain intersection, the construction of the condition constraint set can filter invalid reasoning and ensure that the relationship conforms to the rules of the physical world (such as “daoxian must be located above or near the ganta”). Priority constraint conditions when conflicts occur in the process of determining the structure mode corresponding to the typical detection object combination are defined, for example, a high priority can be set for safety-related rules (such as “high-voltage equipment must be grounded”), a low priority can be set for statistical rules (such as “conductors are usually arranged in a straight line”), a priority score (such as 1-10) is assigned to each rule, and the result with the highest score is selected during reasoning, and when multiple rules are triggered at the same time, the optimal reasoning result is selected according to the priority (such as “safety specifications take precedence over geometric constraints”). After obtaining the structure template library, the condition constraint set and the priority constraint condition, a rule enhancement model is constructed according to the structure template library, the condition constraint set and the priority constraint condition.

[0058] In some embodiments of the present application, the first feature data is input into the pre-constructed rule enhancement model, and the second feature data of the basic image is extracted, including: determining the combination relationship between different target detection objects and the structure corresponding to the combination relationship under the constraint of the condition constraint set and the priority constraint condition, to obtain the second feature data.

[0059] Exemplarily, when the system detects a "ganta" pole tower target and multiple "daoxian" conductor targets in an image at the same time, the system can intelligently determine that they constitute a "tower-line combination" structure, verify the spatial constraint condition and the logical constraint condition, further infer the implicit relationship that "the conductors are connected to the pole tower", and generate an enhanced structured semantic result, that is: { "objects": [...], "relations": [ {"from": "daoxian1", "to": "ganta1", "relation": "attached"}, {"from": "daoxain2", "to": "ganta1", "relation": "attached"} ] }.

[0060] In some embodiments of the present application, the above method further comprises: determining whether there is an object that has an association degree greater than a preset association threshold with the target detection object and has not been detected based on the spatial relationship of the target detection object and the scene information of the target object; and defining the object as the target detection object in response to the existence of the object that has an association degree greater than the preset association threshold with the target detection object and has not been detected.

[0061] Specifically, it is determined whether there is an object that has an association degree greater than a preset association threshold with the target detection object and has not been detected based on the spatial relationship (such as distance, direction, and topology) of the target detection object and the scene information (such as the domain knowledge of the "power tower-line system") of the target object. When there is an object that has an association degree greater than the preset association threshold with the target detection object and has not been detected, it can be considered that there is a missed target. For example, for the case of missing target due to occlusion or detection algorithm limitation, such as "cement pole" not being detected but having an obvious combination relationship with "ganta", the object is defined as the target detection object, and the target detection object is added to the detection result, and the source is marked as "obtained by rule inference". Through spatial relationship analysis and scene semantic fusion, the dynamic completion of the missed target is realized, and the completeness and domain adaptability of target detection are significantly improved.

[0062] In some embodiments of the present application, the reply information for the question information is generated according to the target text description, comprising: constructing a structured prompt word according to the target text description and the question information, inputting the prompt word into a preset large language model to obtain the reply information and the reference evidence information associated with the reply information.

[0063] Specifically, after obtaining the target image-text description pair and the question information, a structured prompt is constructed based on the target image-text description pair (natural language description of the image-text pair) and the question information (user question). This structured prompt is then input into a pre-defined large language model (such as ChatGLM, Qwen, GPT, etc.) to generate accurate and natural response text. The generated content not only includes the response information but can also include related citation evidence, i.e., image-related citation evidence (such as structural descriptions, positional annotations in the image, etc.), thereby significantly enhancing the interpretability and credibility of the question-and-answer results. The final output format is as follows: { "Result": "The conductor is connected to the tower on the left side of the diagram." “Evidence”:[“Graph description: ...”], "Confidence": 0.91 } Among them, ChatGLM (Chat Generative Language Model) is optimized for Chinese scenarios and performs excellently in Chinese understanding and generation tasks; it supports local deployment (open source version) and is suitable for academic and research purposes. Qwen (Qianwen Language Model) has strong multilingual support (mainly Chinese and English) and outstanding long text understanding capabilities; the open source version can be used for free commercially (subject to licensing agreements); it is deeply integrated with the Alibaba Cloud ecosystem. GPT (Generative Pre-trained Transformer) has extremely strong English capabilities, with Chinese optimization gradually improving (but slightly weaker than native models); it is closed source and can only be used through API (Application Programming Interface) or products (such as ChatGPT Plus); it has a rich plugin ecosystem and supports complex tasks (such as data analysis and plotting).

[0064] As a specific example, such as Figure 2 As shown, the image-text question answering method based on object detection and rule enhancement of the present invention may include the following steps: S201, Obtain user question information.

[0065] S202, Obtain the image and text description pair (basic image + multiple text description statements).

[0066] S203: Retrieve the image-text description pair that is semantically closest to the question from multiple image-text description pairs.

[0067] S204, select the top K (default K=3) semantic relevance of the image-text pair from the image-text pair closest to the question semantics, and take the image-text pair as the target image-text description pair.

[0068] S205, construct a structured prompt word prompt according to the target image-text description pair and the question information.

[0069] S206, input the structured prompt word into a preset large language model, and the preset large language model generates reply information and reference evidence information associated with the reply information.

[0070] S207, output the reply information and the reference evidence information associated with the reply information.

[0071] S208, visualize the reference of the reply information and the reference evidence information associated with the reply information.

[0072] Thus, the present application improves the understanding of image semantics and the accuracy of question and answer by building a multi-modal information joint library; introduces a defect knowledge base and a rule engine to enhance the generation of professional question and answer content for the detection result, realizes knowledge controllable question and answer generation; by unified management and structured representation of image, detection frame and natural language description, the final question and answer have reasoning chain and visual basis, and the professional reliability of the result is enhanced.

[0073] To sum up, according to the method for question and answer of image and text based on target detection and rule enhancement, the plurality of basic images of the target object are acquired and stored; the basic images are input into the pre-constructed target detection model to extract first feature data of the basic images; the first feature data represents the explicit features directly expressed by the basic images; the first feature data is input into the pre-constructed rule enhancement model to extract second feature data of the basic images; the second feature data represents the implicit features and the relationship between different explicit features which are not directly expressed by the basic images; the second feature data is converted into text description sentences; a plurality of image-text description pairs are generated according to the plurality of basic images and the plurality of text description sentences; the image-text description pair includes at least one basic image and at least one text description sentence, and the basic image corresponds to the text description sentence; in response to receiving the question information for the target object, the question information is converted into a target semantic vector, the target image-text description pair is retrieved from the plurality of image-text description pairs based on the target semantic vector, and the reply information for the question information is generated according to the target image-text description pair; the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold. Thus, the method can realize deep understanding of image content and semantic-level question and answer, automatically analyze the structural information and semantic relationship contained in the image, convert the image content into natural language expression, and drive the question and answer process through the multi-modal vectorization technology, and is suitable for scenes such as power inspection, equipment identification, and structural relationship analysis which have high requirements for image understanding and question and answer accuracy.

[0074] It should be noted that the method of the embodiments of the present application can be executed by a single device, such as a computer or a server. The method of the embodiments can also be applied in a distributed scenario, and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiments of the present application, and the multiple devices can interact with each other to complete the above method.

[0075] It should be noted that some embodiments of the present application have been described above. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in a different order than the order described above and still achieve desirable results. In addition, the processes depicted in the figures do not necessarily require the particular order shown, or sequential order, to achieve the desired results. In certain implementations, multitasking and parallel processing can be advantageous.

[0076] Corresponding to the above embodiments, the present application also proposes a device for question and answer of image and text based on target detection and rule enhancement.

[0077] As Figure 3As shown, the target detection and rule enhancement-based image-text question answering device according to the embodiments of the present application comprises an image acquisition and uploading module 310, an image detection and structured output module 320, a rule-based target relationship analysis and completion module 330, an image-text natural language description generation module 340, an image-text information vectorization storage and index construction module 350, and a question-driven vector retrieval and question answering generation module 360.

[0078] The image acquisition and uploading module 310 is configured to acquire and store a plurality of basic images of a target object. The image detection and structured output module 320 is configured to input the basic images into a pre-constructed target detection model to extract first feature data of the basic images. The first feature data represents explicit features directly expressed by the basic images. The rule-based target relationship analysis and completion module 330 is configured to input the first feature data into a pre-constructed rule enhancement model to extract second feature data of the basic images. The second feature data represents implicit features and relationships between different explicit features that are not directly expressed by the basic images. The image-text natural language description generation module 340 is configured to convert the second feature data into text description sentences. The image-text information vectorization storage and index construction module 350 is configured to generate a plurality of image-text description pairs according to the plurality of basic images and the plurality of text description sentences. Each image-text description pair comprises at least one basic image and at least one text description sentence, and the basic image and the text description sentence correspond to each other. The question-driven vector retrieval and question answering generation module 360 is configured to, in response to receiving question information about the target object, convert the question information into a target semantic vector, retrieve a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generate reply information about the question information according to the target image-text description pair. The vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold.

[0079] In some embodiments of the present application, the image acquisition and uploading module 310 acquires and stores a plurality of basic images of a target object, specifically for: acquiring images of the target object through a front-end acquisition device to obtain a plurality of basic images of the target object and metadata information of each basic image; and storing the basic images according to the metadata information.

[0080] In some embodiments of the present application, the image detection and structured output module 320 inputs the basic images into a pre-constructed target detection model to extract first feature data of the basic images, specifically for: detecting a target detection object in the basic images; extracting a bounding box coordinate, a class name, and a confidence of the target detection object; and filling the bounding box coordinate, the class name, and the confidence into a pre-constructed structured template to obtain the first feature data.

[0081] In some embodiments of the present application, the rule-based target relationship analysis and completion module 330 is further configured to construct a typical detection object combination for the target detection object, determine a structure mode corresponding to the typical detection object combination, and construct a structure template library according to the typical detection object combination and the structure mode; wherein the typical detection object combination is used to describe the combination relationship of different target detection objects, and the structure mode is used to describe the structure corresponding to the combination relationship of different target detection objects; define a spatial constraint condition for the target detection object and a logical constraint condition when the target detection object is combined, and construct a condition constraint set according to the spatial constraint condition and the logical constraint condition; define a priority constraint condition when a conflict occurs in the process of determining the structure mode corresponding to the typical detection object combination; and construct a rule enhancement model according to the structure template library, the condition constraint set, and the priority constraint condition.

[0082] In some embodiments of the present application, the rule-based target relationship analysis and completion module 330 inputs the first feature data into the pre-constructed rule enhancement model and extracts the second feature data of the basic image, specifically for: determining the combination relationship between different target detection objects and the structure corresponding to the combination relationship under the constraint of the condition constraint set and the priority constraint condition, to obtain the second feature data.

[0083] In some embodiments of the present application, the rule-based target relationship analysis and completion module 330 is further configured to determine whether there is an object that has an association degree greater than a preset association threshold with the target detection object and has not been detected based on the spatial relationship of the target detection object and the scene information of the target object; and in response to the existence of the object that has an association degree greater than the preset association threshold with the target detection object and has not been detected, define the object as the target detection object.

[0084] In some embodiments of the present application, the question-driven vector retrieval and question and answer generation module 360 generates reply information for the question information according to the target image-text description, specifically for: constructing a structured prompt word according to the target image-text description and the question information, inputting the prompt word into a pre-set large language model, and obtaining the reply information and reference evidence information associated with the reply information.

[0085] It should be noted that the details of the target detection and rule enhancement image-text question and answer device in the embodiments of the present application are not disclosed, please refer to the details disclosed in the target detection and rule enhancement image-text question and answer method in the embodiments of the present application, and specific details will not be repeated.

[0086] In summary, the target detection and rule enhancement-based image-text question answering device according to the embodiment of the present application comprises: an image acquisition and uploading module configured to acquire and store a plurality of basic images of a target object; an image detection and structured output module configured to input the basic images into a pre-constructed target detection model to extract first feature data of the basic images; wherein the first feature data represents explicit features directly expressed by the basic images; a rule-based target relationship analysis and completion module configured to input the first feature data into a pre-constructed rule enhancement model to extract second feature data of the basic images; wherein the second feature data represents implicit features not directly expressed by the basic images and relationships between different explicit features; an image-text natural language description generation module configured to convert the second feature data into text description sentences; an image-text information vectorization storage and index construction module configured to generate a plurality of image-text description pairs according to the plurality of basic images and the plurality of text description sentences; wherein an image-text description pair comprises at least one basic image and at least one text description sentence, and the basic image and the text description sentence correspond to each other; a question-driven vector retrieval and question answering generation module configured to, in response to receiving questioning information about the target object, convert the questioning information into a target semantic vector, retrieve a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generate reply information for the questioning information according to the target image-text description pair; wherein the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold. Thus, the device can realize deep understanding of image content and semantic-level question answering, automatically analyze structural information and semantic relationships contained in the image, convert the image content into natural language expression, and drive the question answering process through multi-modal vectorization technology, and is suitable for scenarios such as power inspection, device identification, and structural relationship analysis, which have high requirements for image understanding and question answering accuracy.

[0087] For the sake of convenience, the above system is described in various modules according to functions. Of course, when implementing the present application, the functions of the modules can be implemented in one or more software and / or hardware.

[0088] The system of the above embodiment is used to implement the corresponding method in any of the above embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be described here.

[0089] Corresponding to the above embodiment, the present application also proposes an electronic device.

[0090] Reference Figure 4Fig. 1 is a block diagram of an electronic device according to some embodiments of the present application. As shown in Fig. 1, the electronic device can include a processor 410, a memory 420, an input / output interface 430, a communication interface 440, and a bus 450. The processor 410, the memory 420, the input / output interface 430, and the communication interface 440 are connected to each other via the bus 450.

[0091] The processor 410 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0092] The memory 420 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, or the like. The memory 420 can store an operating system and other application programs. When the technical solutions provided by the embodiments of the present application are implemented by software or firmware, the related program codes are stored in the memory 420 and are invoked and executed by the processor 410.

[0093] The input / output interface 430 is configured to connect to an input / output module to implement information input and output. The input / output module can be configured as a component in the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, and the like, and the output device can include a display, a speaker, a vibrator, an indicator, and the like.

[0094] The communication interface 440 is configured to connect to a communication module (not shown in the figure) to implement communication interaction between the device and other devices. The communication module can implement communication by a wired manner (for example, a USB, a network cable, or the like) or by a wireless manner (for example, a mobile network, WIFI, Bluetooth, or the like).

[0095] The bus 450 includes a path to transmit information between various components (for example, the processor 410, the memory 420, the input / output interface 430, and the communication interface 440) of the device.

[0096] It should be noted that although the above device only shows the processor 410, the memory 420, the input / output interface 430, the communication interface 440 and the bus 450, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain components necessary to implement the embodiments of the present application, and does not necessarily contain all the components shown in the figure.

[0097] The electronic device of the above embodiment is used to implement the corresponding method in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0098] Based on the same inventive concept, the present application also provides a computer readable storage medium, which stores computer instructions for making a computer execute the method of any of the above embodiments.

[0099] The above computer readable storage medium can be any available medium or data storage device accessible by a computer, including but not limited to magnetic storage (such as floppy disk, hard disk, magnetic tape, magneto-optical disk (MO) and the like), optical storage (such as CD, DVD, BD, HVD and the like), and semiconductor storage (such as ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid state disk (SSD)) and the like.

[0100] The computer instructions stored in the storage medium of the above embodiment are used to make the computer execute the method of any of the above exemplary method embodiments, and have the beneficial effects of the corresponding method embodiment, which will not be repeated here.

[0101] In addition, although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the shown operations must be performed to achieve the desired result. On the contrary, the steps depicted in the flowchart can change the order of execution. Additionally or alternatively, some steps can be omitted, a plurality of steps can be combined into one step, and / or one step can be divided into a plurality of steps.

[0102] It should be understood that aspects of the application can be implemented in hardware, software, firmware or combinations thereof, as desired. In the embodiments described above, various steps or methods can be implemented, in software or firmware which are stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following technologies, known in the art, or combinations thereof, can be employed: discrete logic circuitry having logic gates for implementing logic functions upon data signals, application specific integrated circuits having appropriate combinational logic gates, programmable gate arrays (PGA), field programmable gate arrays (FPGA), and so forth.

[0103] It should be noted that, unless otherwise defined, technical terms or scientific terms used in the embodiments of the application shall have the common meaning understood by one of ordinary skill in the art to which the application pertains. The terms "first", "second", and similar terms used in the embodiments of the application do not denote any order, quantity, or importance, but are used to distinguish different components. The terms "include", "contain", and similar terms mean that the elements or objects before the terms encompass the elements or objects listed after the terms and their equivalents, and do not exclude other elements or objects. The terms "connect" or "connected" and similar terms do not mean only physical or mechanical connections, but can include electrical connections, whether direct or indirect. The terms "upper", "lower", "left", "right", and the like are used only to indicate relative positional relationships, and when the absolute positions of the described objects are changed, the relative positional relationships can also be changed accordingly.

[0104] While the principles and spirit of the application have been described with reference to several specific embodiments, it is to be understood that the application is not limited to the specific embodiments disclosed, and that the division of aspects is not meant to imply that features from these aspects cannot be combined to benefit, but is merely for ease of presentation. The application is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims. The scope of the appended claims is the broadest interpretation under the law, to include all such modifications and equivalent structures and functions.

Claims

1. A graph-text question answering method based on object detection and rule enhancement, characterized in that, include: Acquire and store multiple base images of the target object; The base image is input into a pre-built target detection model to extract the first feature data of the base image; wherein, the first feature data represents the explicit features directly expressed by the base image; The first feature data is input into a pre-built rule enhancement model to extract the second feature data of the base image; wherein, the second feature data characterizes the latent features not directly expressed by the base image and the relationship between different explicit features; The second feature data is converted into a text description statement; Multiple image-text description pairs are generated based on multiple base images and multiple text description statements; wherein, each image-text description pair includes at least one base image and at least one text description statement, and the base image and the text description statement correspond one-to-one; In response to receiving a question about the target object, the question is converted into a target semantic vector. Based on the target semantic vector, a target image-text description pair is retrieved from the plurality of image-text description pairs. A response is generated based on the target image-text description pair. The vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold.

2. The image-text question answering method based on object detection and rule enhancement according to claim 1, characterized in that, The acquisition and storage of multiple base images of the target object includes: The target object is captured by a front-end acquisition device to obtain multiple basic images of the target object and metadata information of each basic image; The base image is stored hierarchically based on the metadata information.

3. The image-text question answering method based on object detection and rule enhancement according to claim 2, characterized in that, The step of inputting the base image into a pre-built target detection model and extracting the first feature data of the base image includes: The target object in the base image was detected; Extract the bounding box coordinates, category name, and confidence score of the target object; The bounding box coordinates, the category name, and the confidence score are filled into a pre-built structured template to obtain the first feature data.

4. The image-text question answering method based on object detection and rule enhancement according to claim 3, characterized in that, The method further includes: A typical combination of detection objects for the target detection object is constructed, a structural pattern corresponding to the typical combination of detection objects is determined, and a structural template library is constructed based on the typical combination of detection objects and the structural pattern; wherein, the typical combination of detection objects is used to describe the combination relationship of different target detection objects, and the structural pattern is used to describe the structure corresponding to the combination relationship of different target detection objects; Define spatial constraints for the target detection object and logical constraints when the target detection objects are combined, and construct a set of condition constraints based on the spatial constraints and the logical constraints. Define priority constraints when conflicts arise during the process of determining the structural patterns corresponding to the typical combination of detection objects; The rule enhancement model is constructed based on the structure template library, the set of condition constraints, and the priority constraints.

5. The image-text question answering method based on object detection and rule enhancement according to claim 4, characterized in that, The step of inputting the first feature data into a pre-built rule enhancement model to extract the second feature data of the base image includes: Under the constraints of the set of conditions and the priority constraints, the combination relationships between different target detection objects and the structure corresponding to the combination relationships are determined to obtain the second feature data.

6. The image-text question answering method based on object detection and rule enhancement according to claim 5, characterized in that, The method further includes: Based on the spatial relationship of the target detection object and the scene information of the target object, determine whether there are objects that have a correlation degree with the target detection object greater than a preset correlation threshold but have not been detected; In response to the existence of an object whose correlation with the target detection object is greater than a preset correlation threshold and which has not been detected, the object is defined as the target detection object.

7. The image-text question answering method based on object detection and rule enhancement according to claim 6, characterized in that, The step of generating response information for the question based on the target image and text description includes: Based on the target image and text description and the question information, a structured prompt word is constructed. The prompt word is then input into a preset large language model to obtain the response information and the citation evidence information associated with the response information.

8. A text-based question-answering device based on object detection and rule enhancement, characterized in that, include: The image acquisition and uploading module is configured to acquire and store multiple basic images of the target object; The image detection and structured output module is configured to input the base image into a pre-built target detection model and extract first feature data of the base image; wherein the first feature data represents the explicit features directly expressed by the base image; The rule-based target relationship parsing and completion module is configured to input the first feature data into a pre-built rule enhancement model to extract the second feature data of the base image; wherein the second feature data represents the latent features not directly expressed by the base image and the relationship between different explicit features; The image-text natural language description generation module is configured to convert the second feature data into text description statements; The image and text information vectorization storage and indexing construction module is configured to generate multiple image and text description pairs based on multiple base images and multiple text description statements; wherein, each image and text description pair includes at least one base image and at least one text description statement, and the base image and the text description statement correspond one-to-one. The question-driven vector retrieval and question-answer generation module is configured to, in response to receiving a question for the target object, convert the question into a target semantic vector, retrieve a target image-text description pair from the plurality of image-text description pairs based on the target semantic vector, and generate a response to the question based on the target image-text description pair; wherein the vector similarity between the target image-text description pair and the target semantic vector is greater than a preset similarity threshold.

9. An electronic device, characterized in that, include: A processor and a memory, the memory storing a program or instructions executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the object detection and rule-enhanced text-based question answering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the object detection and rule-enhanced text-to-image question answering method as described in any one of claims 1 to 7.

Images