Method and device for complex scene recognition based on multi-modal agent, and storage medium

Abstract

The application discloses a complex scene recognition method and device based on a multi-modal intelligent agent and a storage medium, and relates to the technical field of artificial intelligence. The method comprises the following steps: determining a task type based on an image and a natural language request input by a user; in the case that the task type is a target event, determining a target category set by using a multi-modal large model; mapping the target category set to a query category set, inputting the image and the query category set into an open set target detector, and obtaining target detection evidence, wherein the open set target detector adopts a dynamic query mechanism based on a prompt word feature library to predict the target category of the image; inputting the image, the target detection evidence and the natural language request into the multi-modal large model, and obtaining a judgment conclusion and supporting evidence of the target event. The application solves the technical problem in the prior art that the multi-modal large model lacks quantifiable target level evidence support in complex scene recognition, which makes it difficult to realize complex scene recognition.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to a method, apparatus, and storage medium for complex scene recognition based on multimodal intelligent agents. Background Technology

[0002] With the widespread application of high-point video surveillance and drone image surveillance technologies, visual surveillance systems have evolved from the stage of simple target detection to the stage of complex scene understanding and event reasoning. In actual operations, such as forest fire identification, monitoring of illegal construction on land, and detection of abnormalities in power facilities, it is often not enough to simply detect a single target. Instead, it requires high semantic tasks such as determining "what state the scene is in," "whether a certain type of complex event has occurred," or "whether there are any safety hazards."

[0003] However, traditional object detection models can output bounding boxes, categories, and quantities, but they can only perform low-semantic object recognition and cannot directly draw conclusions about complex events at the scene level. When faced with complex events, traditional solutions usually require manual decomposition of the complex events into a large number of low-level rules, which is not only labor-intensive but also difficult to cover ever-changing business scenarios. Although multimodal large models have strong image understanding and semantic reasoning capabilities and are suitable for comprehensive judgment of complex scene states, they lack stability in object-level bounding box selection, quantity statistics, and structured evidence. They lack accuracy when directly outputting detection boxes and structured object evidence and are unable to provide quantifiable object-level evidence support.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides a method, apparatus, and storage medium for complex scene recognition based on multimodal intelligent agents, in order to at least solve the technical problem in the prior art that the lack of quantifiable target-level evidence to support the recognition of complex scenes in multimodal large models makes it difficult to achieve complex scene recognition.

[0006] According to one aspect of the embodiments of this application, a method for complex scene recognition based on a multimodal intelligent agent is provided, comprising: determining a task type based on an image and a natural language request input by a user; when the task type is a target event, determining a target category set through a multimodal large model, wherein the target event represents a semantic task composed of multiple target objects, spatial relationships, scene context states, or temporal logic combinations, and the target category set represents the specific object category to be identified in the target event; mapping the target category set to a query category set, and inputting the image and the query category set into an open-set target detector to obtain target detection evidence, wherein the open-set target detector uses a dynamic query mechanism based on a prompt word feature library to predict the target category of the image; and inputting the image, target detection evidence, and natural language request into the multimodal large model to obtain the judgment conclusion and supporting evidence of the target event.

[0007] Optionally, when the task type is a target event, the target category set is determined by a multimodal large model, including: using the multimodal large model to perform semantic analysis on the natural language request and extract key entities and state features related to the target event; generating corresponding category description text based on the key entities and state features and combined with a preset business semantic mapping table; and performing deduplication and standardization processing on the category description text to form a target category set.

[0008] Optionally, the target category set is mapped to the query category set, and the image and the query category set are input into an open-set target detector to obtain target detection evidence. This includes: inputting the image and the query category set into the open-set target detector, obtaining initial query features corresponding to the query category set through the open-set target detector, wherein the initial query features are obtained by encoding any query category text in the query category set through a text encoder; generating target query features based on a cue word feature library by performing attention enhancement and weighted summation on the initial query features, wherein the cue word feature library is trained based on the target dataset; obtaining multi-scale visual features of the image, and generating a set of detection boxes based on the multi-scale visual features, wherein each detection box in the detection box set is associated with a normalized visual feature, coordinates, and confidence score; determining the similarity between the target query features and the normalized visual features of each detection box in the detection box set, and determining the predicted category of the detection box based on the similarity; and obtaining target detection evidence based on the predicted category, coordinates, and confidence score of each detection box.

[0009] Optionally, before generating target query features by attention enhancement and weighted fusion of the initial query features based on the prompt word feature library, the method further includes: obtaining an unlabeled set of business images; using a first model to pre-label each image in the business image set with candidate targets to obtain a set of candidate targets, wherein the set of candidate targets includes multiple candidate targets, and each candidate target includes a candidate bounding box and a pre-labeled category; constructing a complete evidence map for each candidate target in the set of candidate targets, wherein the complete evidence map includes a left-side localization map and a right-side local evidence map, the left-side localization map is used to show the spatial position and contextual relationship of the candidate target in the corresponding image, and the right-side local evidence map is stitched together from the local cropped images of the candidate targets after area adaptive expansion; inputting the complete evidence map into a second model for correction to obtain a correction result, wherein the second model is used to perform target-by-target category discrimination for each candidate target, and the correction result includes the final category, retention status and judgment reason; constructing a labeled dataset based on all candidate targets with a retention status and the final category corresponding to the candidate targets, and merging the labeled dataset with a general open dataset to form a target dataset; and constructing a prompt word feature library based on the target dataset.

[0010] Optionally, before constructing the complete evidence map, the method further includes: for each candidate target in the candidate target set, determining the proportion of the area of ​​the candidate box in the candidate target to the total area of ​​the corresponding image; determining a target expansion coefficient based on the proportion, wherein the target expansion coefficient is a first expansion coefficient or a second expansion coefficient, and the second expansion coefficient is less than the first expansion coefficient; if the proportion is less than a preset first threshold, expanding the candidate box using the first expansion coefficient to obtain a first local region; if the proportion is greater than or equal to the preset first threshold, expanding the candidate box using the second expansion coefficient to obtain a second local region; and stitching together the first local regions or second local regions corresponding to all candidate targets according to a preset stitching method to form the local evidence map on the right.

[0011] Optionally, the complete evidence map is input into the second model for correction to obtain the correction result, including: determining whether there is a real target in the candidate box of the candidate target; if there is a real target, the retention status is marked as retained; if there is no real target, the retention status is marked as discarded; when the retention status is retained, it is determined whether the real category of the target in the candidate box is consistent with the pre-labeled category of the candidate target; if the real category is inconsistent with the pre-labeled category, the pre-labeled category is corrected to the real category, and the real category is used as the final category of the candidate target, generating a judgment reason indicating category correction; if the real category is consistent with the pre-labeled category, the pre-labeled category is used as the final category of the candidate target, and a judgment reason indicating category consistency is generated; integrating the retention status, final category and judgment reason of the candidate target to generate the correction result.

[0012] Optionally, based on the target dataset, a prompt word feature library is constructed, including: extracting category names from the target dataset to construct a set of descriptive texts; encoding the set of descriptive texts using a text encoder to obtain a set of semantic features; selecting K semantic features from the set of semantic features as semantic anchors, where the semantic anchors are used to represent the most dispersed semantic features in the set of semantic features, and K is a positive integer greater than or equal to 1; and performing multi-head attention aggregation based on the semantic anchors and the set of semantic features to obtain the prompt word feature library.

[0013] Optionally, after obtaining the prompt word feature library, the method further includes: for positive sample candidate boxes in the current batch, calculating the attention weight of the j-th semantic feature in the prompt word feature library for the positive sample candidate boxes in the current batch, wherein the positive sample candidate box represents a candidate box with a real category label and satisfying a preset confidence threshold, and j is a positive integer greater than or equal to 1; calculating the update amount of the j-th semantic feature in the prompt word feature library based on the attention weight, the fusion mapping result of the normalized visual features of the positive sample candidate box and the corresponding real category query features; and updating the j-th semantic feature in the prompt word feature library based on the update amount.

[0014] Optionally, based on the prompt word feature library, attention enhancement and weighted summation are performed on the initial query features to generate target query features, including: obtaining preset fusion weight coefficients; using the prompt word feature library to enhance the attention of the initial query features to obtain enhanced query features; weighting and summing the initial query features and enhanced query features according to the fusion weight coefficients, and normalizing the summation result to obtain the target query features.

[0015] Optionally, the training steps of the open-set object detector further include: updating model parameters based on the overall loss, wherein the overall loss is a weighted sum of bounding box regression loss, object confidence loss, classification loss based on query feature similarity, and cue word feature library constraint term loss. The bounding box regression loss is used to optimize the deviation between the predicted coordinates and the true labeled coordinates of positive sample candidate boxes, the object confidence loss is used to optimize the judgment of object existence, the classification loss based on query feature similarity is used to optimize the matching of query features and visual features, and the cue word feature library constraint term is used to constrain the update direction of the cue word feature library.

[0016] According to another aspect of the embodiments of this application, a complex scene recognition device based on a multimodal intelligent agent is also provided, comprising: a first determining unit, configured to determine a task type based on an image and a natural language request input by a user; a second determining unit, configured to determine a target category set through a multimodal large model when the task type is a target event, wherein the target event represents a semantic task composed of multiple target objects, spatial relationships, scene context states, or temporal logic combinations, and the target category set represents the specific object category to be identified in the target event; a first detection unit, configured to map the target category set to a query category set, and input the image and the query category set into an open-set target detector to obtain target detection evidence, wherein the open-set target detector uses a dynamic query mechanism based on a prompt word feature library to predict the target category of the image; and a second detection unit, configured to input the image, target detection evidence, and natural language request into the multimodal large model to obtain a judgment conclusion and supporting evidence for the target event.

[0017] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-described complex scene recognition method based on multimodal intelligent agents.

[0018] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described complex scene recognition method based on multimodal intelligent agents.

[0019] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described complex scene recognition method based on a multimodal intelligent agent.

[0020] In this embodiment, a multimodal intelligent agent collaborative architecture is adopted. By using a multimodal large model as the core of semantic understanding and task scheduling, and an open-set target detector based on a prompt word feature library as a low-level detection and evidence-gathering tool, a division of labor and cooperation mechanism for semantic judgment and evidence support is realized. This achieves the goal of organically combining high-level semantic reasoning in complex scenarios with precise quantitative detection at the target level. As a result, it can automatically complete the identification of complex events without manual rule decomposition, and the results have both semantic understanding capabilities and quantifiable and traceable detection evidence. This solves the technical problem in the prior art where multimodal large models lack quantifiable target-level evidence support in complex scene identification, making it difficult to achieve complex scene identification. Attached Figure Description

[0021] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0022] Figure 1 This is a flowchart of an optional complex scene recognition method based on a multimodal intelligent agent according to an embodiment of this application;

[0023] Figure 2 This is an architecture diagram of an optional complex scene recognition system based on a multimodal intelligent agent according to an embodiment of this application;

[0024] Figure 3 This is a schematic diagram of an optional complex scene recognition device based on a multimodal intelligent agent according to an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] According to an embodiment of this application, a method embodiment for complex scene recognition based on a multimodal intelligent agent is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0028] According to the embodiments of this application, a complex scene recognition system based on multimodal intelligent agents (hereinafter referred to as the system) can be used as the execution subject of the complex scene recognition method based on multimodal intelligent agents in the embodiments of this application. The complex scene recognition system based on multimodal intelligent agents can be a software system or an embedded system combining software and hardware. Of course, the execution subject of the method in the embodiments of this application can also be other forms of execution subject, such as devices or equipment. Those skilled in the art should know that this application does not particularly limit the specific form of the execution subject.

[0029] Figure 1 This is a flowchart of an optional complex scene recognition method based on multimodal intelligent agents according to an embodiment of this application, such as... Figure 1 As shown, the method includes the following steps:

[0030] Step S101: Determine the task type based on the user-input image and natural language request.

[0031] Optionally, the image input by the user refers to a raw visual image from a high-point video surveillance camera or a drone inspection device.

[0032] Optionally, a natural language request refers to a query entered by a user through a text interface, which may include vague descriptions or specific instructions.

[0033] Optionally, the task type is the system's classification result of the request intent, and in this embodiment, it includes at least three categories: ordinary image question answering tasks, simple object detection tasks, and complex event analysis tasks. Among them, complex event analysis tasks refer to those tasks that cannot be directly determined by a single object category, but require comprehensive reasoning by combining multiple objects, spatial location relationships, scene context states, or temporal logic, such as determining "whether there is illegal construction at the construction site" or "whether there are foreign objects entangled in the power lines".

[0034] Optionally, after receiving the image and natural language request submitted by the user, the system first sends the input data to the multimodal big data model. The multimodal big data model utilizes its built-in intent recognition capabilities to analyze the semantic features of the natural language request and the overall content of the image. The system has a pre-set set of intent classification prompts to guide the multimodal big data model in determining the task type of the request. For example, if the request is "How many workers are in the image?", the model identifies it as a simple object detection task; if the request is "Determine if there is a fire hazard in this area", the model, combining the distribution of smoke and flames in the image with the environmental background, identifies it as a complex event analysis task. After determining the task type, the system divides the processing flow according to the type.

[0035] Step S102: When the task type is a target event, the target category set is determined by a multimodal large model. The target event represents a semantic task based on a combination of multiple target objects, spatial relationships, scene context states, or temporal logic. The target category set represents the specific object categories that need to be identified in the target event.

[0036] Optionally, the target event, or complex event, is characterized by semantic complexity and depends on multiple objects and their relationships. For example, the event of illegal construction is not a single target, but rather consists of multiple objects such as excavators, people not wearing safety helmets, and exposed soil, as well as their spatial relationships.

[0037] Optionally, the target category set refers to a specific list of target categories generated after multimodal large model parsing. Each category in this set corresponds to a specific object or object attribute that needs to be searched in the image.

[0038] Optionally, when the task type is determined to be a target event, the multimodal big data model first performs deep semantic analysis on the natural language request to extract key entities and state features. Then, the system uses a pre-defined business semantic mapping table or the big data model's common-sense reasoning capabilities to transform the abstract event description into specific detection targets. Finally, the system performs deduplication and standardization on the detected targets, ultimately forming a set of target categories. Furthermore, the system can also perform semantic expansion based on fuzzy queries. For example, if a user inputs "car," the system can automatically expand it to include more specific categories such as "truck," "car," and "motorcycle" to cover more potential targets.

[0039] Step S103: Map the target category set to the query category set, and input the image and the query category set into the open set target detector to obtain target detection evidence. The open set target detector uses a dynamic query mechanism based on the prompt word feature library to predict the target category of the image.

[0040] Optionally, the query category set is a list of feature vectors formed by encoding the target category set for recognition by the open set target detector.

[0041] Alternatively, the open-set object detector is a detection model based on a dynamic query mechanism using a cue word feature library. Its classification space is not fixed but dynamically generated according to the input query category. The open-set object detector is not limited to the closed categories seen during training and can detect targets described in any text.

[0042] Optionally, object detection evidence refers to the structured results output by the open-set object detector, including the bounding box coordinates, predicted category, confidence score, and corresponding normalized visual features.

[0043] Optionally, the system first converts the text descriptions in the target category set into initial query features using a text encoder. Then, it enhances the initial query features with attention using a pre-trained cue word feature library. This library stores semantic features learned from historical data; through a multi-head attention mechanism, it fuses general text semantics with visual semantics specific to the business scenario to generate enhanced target query features. Subsequently, the image is input into the backbone network of the open-set object detector to extract multi-scale visual features and generate candidate detection boxes. For each candidate detection box, the similarity between its normalized visual features and the target query features is calculated. Based on the query category with the highest similarity, the predicted category of the detection box is determined, and the detection result, including coordinates, category, and confidence score, is output—the object detection evidence.

[0044] Step S104: Input the image, object detection evidence, and natural language request into the multimodal large model to obtain the judgment conclusion and supporting evidence of the target event.

[0045] Optionally, the judgment conclusion refers to the final system output, which answers the user's question.

[0046] Optionally, the supporting evidence refers to the reasoning process and evidence presentation of the multimodal large model, including which detection boxes are referenced, the confidence level of the referenced detection boxes, and how they together constitute the logical chain of the event.

[0047] Optionally, the multimodal large model receives images, object detection evidence, and natural language requests from users. The multimodal large model first combines the global semantic context of the image to understand the specific meaning of the detection evidence within the scene. For example, detecting "excavator" and "exposed soil" does not directly equate to a violation; the model needs to make a comprehensive judgment based on contextual information such as time, location, and personnel status. Subsequently, the multimodal large model uses its thought chain reasoning ability to match the detection evidence with the event definition. If the evidence is sufficient, the model generates a "yes" judgment and lists the specific detection evidence supporting this conclusion. If the evidence is insufficient or contradictory, the model generates a "no" or "uncertain" conclusion and explains the reasons.

[0048] Through the above steps, an organic integration of advanced semantic understanding and low-level visual evidence can be achieved in complex scenarios. Specifically, the system first uses a multimodal large model to perform intent recognition and semantic decomposition on natural language requests, transforming abstract and complex events into a concrete and quantifiable set of target detection categories. This solves the problems of traditional rule engines, which require manual decomposition of complex logic and have poor generalization ability. Subsequently, using an open-set target detector based on a prompt word feature library, the system performs precise target selection, counting, and attribute recognition on images according to dynamically generated query categories. This obtains structured detection evidence with clear coordinates, confidence levels, and categories, compensating for the lack of stability of the multimodal large model in terms of fine-grained target perception and quantifiable evidence. Finally, the multimodal large model combines the global image context with the quantifiable evidence provided by the detector to perform comprehensive reasoning, generating a judgment conclusion that is both logically rigorous and based on factual evidence.

[0049] In one optional embodiment, when the task type is a target event, the target category set is determined by a multimodal large model, including: using the multimodal large model to perform semantic analysis on the natural language request and extract key entities and state features related to the target event; generating corresponding category description text based on the key entities and state features and combined with a preset business semantic mapping table; and performing deduplication and standardization processing on the category description text to form a target category set.

[0050] Optionally, a key entity refers to the core object or subject that constitutes the target event. For example, in an illegal construction incident, key entities may include excavators, construction workers, etc.

[0051] Optionally, state characteristics refer to the attributes, actions, or relationships exhibited by key entities in the current scenario, such as not wearing a safety helmet, being in a moving state, or being located in a restricted area.

[0052] Optionally, the multimodal large model, as an intelligent agent with natural language understanding capabilities, receives natural language requests and, through its internal semantic encoding and attention mechanisms, parses the grammatical structure and semantic logic in the text, removes irrelevant modifiers, locates and extracts key entities and state features, thereby transforming unstructured and fuzzy natural language queries into structured semantic elements, laying the foundation for the subsequent generation of specific detection targets.

[0053] Optionally, the business semantic mapping table refers to a pre-built dictionary or rule base used to map abstract semantic elements to specific detection category standards, which records the correspondence between general semantics and standard detection categories. For example, the semantic element "heavy engineering vehicle" is mapped to the detection category "excavator" or "crane", and "no head protection" is mapped to the detection category "not wearing a safety helmet".

[0054] Optionally, the multimodal large model takes the extracted key entities and state features as input, queries the business semantic mapping table, and converts the abstract semantic elements into standard category description text that meets the input requirements of the open set target detector. This achieves a standardized conversion from business semantics to visual detection terminology, which helps the generated detection instructions to be accurately identified and executed by subsequent detection models. At the same time, the introduction of the mapping table helps to ensure the consistency and standardization of detection categories.

[0055] Optionally, standardization refers to unifying the format, length, and terminology of category descriptions, such as unifying "safety helmet" and "safety helmet" into "safety helmet".

[0056] Optionally, after standardization, the system organizes all unique, standardized category description texts into an ordered list, i.e., the target category set. Each element in the set represents a specific object category that needs to be detected in the image.

[0057] Through the steps described above, this embodiment leverages the semantic understanding capabilities of a multimodal large-scale model to transform user natural language requests into structured semantic elements. These elements are then mapped to standardized detection categories using a business semantic mapping table, ultimately forming a deduplicated and standardized set of target categories. This process automatically converts fuzzy natural language into precise detection instructions, reducing user reliance on technical terminology and improving the system's usability and adaptability in complex business scenarios. Furthermore, the standardized processing flow helps subsequent open-set object detectors execute detection tasks efficiently and accurately, providing a high-quality list of detection targets for complex scene recognition.

[0058] In one optional embodiment, the target category set is mapped to a query category set, and the image and the query category set are input into an open-set target detector to obtain target detection evidence. This includes: inputting the image and the query category set into the open-set target detector; obtaining initial query features corresponding to the query category set through the open-set target detector, wherein the initial query features are obtained by encoding any query category text in the query category set using a text encoder; generating target query features based on a cue word feature library by performing attention enhancement and weighted summation on the initial query features, wherein the cue word feature library is trained based on the target dataset; acquiring multi-scale visual features of the image and generating a set of detection boxes based on the multi-scale visual features, wherein each detection box in the detection box set is associated with a normalized visual feature, coordinates, and confidence score; determining the similarity between the target query features and the normalized visual features of each detection box in the detection box set, and determining the predicted category of the detection box based on the similarity; and obtaining target detection evidence based on the predicted category, coordinates, and confidence score of each detection box.

[0059] Optionally, the initial query features refer to the feature vectors obtained by mapping each category text in the query category set to a high-dimensional vector space through a pre-trained text encoder. Specifically, the open-set object detector receives image data and the query category set, calls the text encoder to independently encode each text description in the query category set, and generates initial query features that correspond one-to-one with each category text.

[0060] Optionally, the prompt word feature library is a set of semantic features pre-built and stored in the open-set target detector. This feature library is trained based on a target dataset containing business scenario data, and stores anchor features that can represent the semantics of each category.

[0061] Optionally, the target query feature refers to the final query vector that has been enhanced and integrates general semantics and business scenario-specific semantics.

[0062] Optionally, the open-set target detector uses semantic features from the cue word feature library as key-value pairs to perform multi-head attention calculation on the initial query features, extracting deep semantic information related to the business scenario to obtain enhanced query features. Subsequently, the system sets preset fusion weight coefficients, performs a weighted summation of the initial query features and enhanced query features, and normalizes the summation result to obtain the target query features. By using a weighted summation of the initial query features and enhanced query features, the system overcomes the problem of insufficient generalization ability of general text encoding in specific business scenarios. By introducing a cue word feature library trained on business data, the system enables query features to accurately match specific targets in images, improving the detection sensitivity for long-tail categories or combined attribute targets.

[0063] Optionally, multi-scale visual features refer to the set of feature maps extracted at different resolution levels after an image is processed by a backbone network and feature pyramid, which can simultaneously capture the approximate location and detailed information of the target.

[0064] Optionally, the detection box set refers to several candidate target regions output by the detection head, each region being defined by coordinates, confidence level, and associated visual features.

[0065] Optionally, the open-set object detector receives the input image, extracts multi-scale visual features through a backbone network, and generates a large number of candidate detection boxes by sliding or sampling on the feature map using a detection head. For each detection box, the system extracts its corresponding normalized visual features to reflect the visual semantic information of the detection box region, while recording its spatial coordinates and presence confidence.

[0066] Optionally, for each detection box in the detection box set, the system calculates the similarity score (such as cosine similarity) between its normalized visual features and the target query features corresponding to the category to which the detection box belongs. For each detection box, the similarity scores with all target query features in the query category set are compared, and the category with the highest score is selected as the predicted category of the detection box, wherein the similarity score is obtained by formula (1).

[0067] (1)

[0068] in, This represents the similarity score of the i-th detection box belonging to category c. Indicates the learnable temperature coefficient. Represents normalized visual features. This represents the normalized target query features. This represents the similarity calculation function.

[0069] Optionally, the system filters the detection boxes after nonmaximum suppression processing, retaining those with confidence scores higher than a preset threshold, and packages the predicted category, coordinate position, confidence score, and corresponding normalized visual features of each retained detection box to form structured object detection evidence.

[0070] In an optional embodiment, before generating target query features by attention enhancement and weighted fusion of the initial query features based on the prompt word feature library, the method further includes: acquiring an unlabeled set of business images; using a first model to pre-label each image in the business image set with candidate targets to obtain a candidate target set, wherein the candidate target set includes multiple candidate targets, and each candidate target includes a candidate bounding box and a pre-labeled category; for each candidate target in the candidate target set, constructing a complete evidence map, wherein the complete evidence map includes a left-side localization map and a right-side local evidence map, the left-side localization map is used to show the spatial position and contextual relationship of the candidate target in the corresponding image, and the right-side local evidence map is stitched together from the local cropped images of the candidate targets after area adaptive expansion; inputting the complete evidence map into a second model for correction to obtain a correction result, wherein the second model is used to perform target-by-target category discrimination for each candidate target, and the correction result includes the final category, retention status, and judgment reason; based on all candidate targets with a retention status and the final category corresponding to the candidate targets, constructing a labeled dataset, and merging the labeled dataset with a general open dataset to form a target dataset; and constructing a prompt word feature library based on the target dataset.

[0071] Optionally, the unlabeled business image set refers to a collection of raw image data from actual monitoring scenarios, such as high-point video of iron towers or drone inspections, that has not been manually labeled. .

[0072] Optionally, the first model refers to a multimodal large model or an open vocabulary detection model with high recall, used for preliminary identification of potential targets in an image.

[0073] Optionally, the candidate target set refers to a list of multiple potential targets generated by the first model, where each candidate target includes the coordinates of a candidate bounding box and a pre-labeled category output by the first model. Candidate Target Set ,in, Indicates the first A unique identifier for each candidate target Indicates the coordinates of the candidate box. This indicates the pre-labeled categories output by the first model. Representing an image The number of candidate targets.

[0074] Optionally, the system iterates through the set of unlabeled business images, inputs each image into the first model, and the first model outputs all possible target regions in the image and their corresponding category labels.

[0075] Optionally, the complete evidence map is a dual-view input format that provides comprehensive contextual information for the calibration model, including a left-side localization map and a right-side local evidence map. The left-side localization map is an image formed by plotting the original image alongside the candidate bounding boxes and their numbers for the current batch of candidate targets. It is used to show the spatial location, scale, and relative relationship of candidate targets within the overall scene. The right-side local evidence map is an image formed by stitching together local images of each candidate target (after adaptive area expansion) in a grid pattern. It is used to show the detailed features of the candidate targets. (Left-side localization map) ,in, This function represents drawing candidate bounding boxes and unique identifiers on the original image.

[0076] Optionally, the system selects a batch of candidate targets from the candidate target set, draws their candidate bounding boxes and unique identifiers on the original image, and generates the left-side localization map. Then, for each candidate target, an expansion coefficient is determined based on its area proportion, and the candidate bounding box is adaptively expanded to obtain richer contextual background, cropping out local images. Finally, these local images are stitched together to form the right-side local evidence map, which is then horizontally stitched with the left-side localization map to form a complete evidence map. This addresses the problem of multi-modal large models lacking global context or sufficient local details when judging a single target category, improving the accuracy of subsequent corrections by providing dual evidence of "global localization + local details."

[0077] Optionally, the second model refers to a large model with high-precision image understanding and structured output capabilities, used for fine discrimination of candidate targets.

[0078] Optionally, the correction result refers to the structured information output by the second model for each candidate target, including the final category, retention status, and reason for judgment.

[0079] Optionally, the system inputs the constructed complete evidence map into the second model. The second model analyzes each candidate target one by one according to the preset prompt word logic. The second model first determines whether a real target exists within the candidate box. If it does, it marks the target as retained; otherwise, it marks it as discarded. If marked as retained, the model further determines whether the pre-labeled category is accurate. If inaccurate, it corrects it to a more accurate final category and generates corresponding reasons for the judgment. Finally, the system collects all candidate targets in the retained state, along with their corrected categories and reasons, forming a correction result. This achieves the cleaning and correction of low-precision pre-labeled results using a high-precision model, eliminating false detection boxes, correcting incorrect categories, and obtaining high-quality training labels.

[0080] Optionally, the system filters out candidate targets whose status is retained from the correction results, and uses their candidate bounding box coordinates and final category as ground truth labels to construct an automatically labeled training set specifically for the business scenario. Simultaneously, the system acquires an open object detection dataset containing a wide range of general categories. The automatically labeled training set for the business scenario is merged with the general open dataset to form the final target dataset. By combining the specificity of business data with the breadth of general data, the system not only helps the open-set object detector adapt to long-tail categories and special attributes in specific monitoring scenarios, but also retains its basic generalization ability for general objects.

[0081] Finally, the system extracts the category names, region description texts, or detailed descriptions generated by multimodal large models for all targets from the target dataset to construct a description text set. A text encoder is used to encode this description text set, resulting in a semantic feature set. To avoid semantic redundancy, the system employs the furthest distance sampling method to select representative semantic anchors from the semantic feature set. Subsequently, using the semantic anchors as queries and the full set of semantic features as keys and values, multi-head attention aggregation is performed to obtain an initialized prompt word feature library. This enables the extraction of high-quality, evenly distributed semantic features from the mixed dataset, providing strong semantic support for subsequent open-set object detectors, allowing them to accurately understand and match various query categories.

[0082] In summary, this embodiment employs a dual-model collaborative annotation mechanism. The first model performs high-recall pre-annotation, followed by fine-tuning based on a complete evidence map using the second model, thus constructing a high-quality automated annotation dataset. By merging business data and general data for training, and building a feature library of prompt words containing rich semantic prototypes based on the target dataset, the problem of high data annotation costs and scarce samples in business scenarios is addressed.

[0083] In an optional embodiment, before constructing the complete evidence map, the method further includes: for each candidate target in the candidate target set, determining the proportion of the area of ​​the candidate box in the candidate target to the total area of ​​the corresponding image; determining a target expansion coefficient according to the proportion, wherein the target expansion coefficient is a first expansion coefficient or a second expansion coefficient, and the second expansion coefficient is less than the first expansion coefficient; if the proportion is less than a preset first threshold, expanding the candidate box using the first expansion coefficient to obtain a first local region; if the proportion is greater than or equal to the preset first threshold, expanding the candidate box using the second expansion coefficient to obtain a second local region; and stitching together the first local regions or second local regions corresponding to all candidate targets according to a preset stitching method to form the right-hand partial evidence map.

[0084] Optionally, the candidate target set refers to a list of multiple candidate targets output by the first model. A candidate bounding box refers to a rectangular bounding box surrounding a potential target.

[0085] Optionally, the ratio refers to the ratio of the area of ​​a single candidate box to the total area of ​​the entire input image, used to quantify the relative size of the candidate target in the image.

[0086] Optionally, the system reads the coordinates of the candidate bounding box for each candidate target and calculates the area S of the candidate bounding box. box Simultaneously, the image resolution is obtained to determine the total area S of the image. img The calculated ratio is Ratio = S box / S img The system can objectively evaluate the scale distribution of each candidate target in the scene by calculating the scale value, providing a quantitative basis for subsequent adaptive adjustment of the clipping range of local areas, and avoiding information redundancy or loss of detail caused by applying the same clipping strategy to large and small targets.

[0087] Optionally, the first expansion factor is typically used for small targets and has a larger value, aiming to expand the receptive field to obtain more context; the second expansion factor is typically used for large targets and has a smaller value, aiming to limit background interference to keep the subject clear.

[0088] Optionally, the system presets a threshold judgment logic. If the proportion of a candidate target is less than a first threshold, the target expansion coefficient is the first expansion coefficient; if the proportion is greater than or equal to the first threshold, the target expansion coefficient is the second expansion coefficient. Specifically, the system traverses the candidate target set. For each candidate target, if its proportion is less than the first threshold, it indicates that the target occupies a small portion of the image, possibly a distant object or a detailed feature. In this case, the system expands the candidate box outwards according to the first expansion coefficient (e.g., increasing it by 2 times or more), cropping out a first local region containing more background environment. Conversely, if the proportion is greater than or equal to the first threshold, it indicates that the target occupies a major part of the image. In this case, the system expands the candidate box according to the second expansion coefficient (e.g., increasing it by 1.2 times or less), cropping out a second local region closely surrounding the target. The system achieves scale-based adaptive context extraction by setting the expansion coefficient. For small targets, expanding the context helps identify their scene or state; for large targets, limiting the expansion range helps focus on target details and avoids irrelevant background noise interfering with the judgment.

[0089] Optionally, the preset stitching method refers to a layout rule that arranges multiple local images into a grid. Specifically, the system collects all local images of candidate targets in the current batch after expansion and cropping, fills them into the canvas sequentially according to the preset grid layout, and marks the corresponding candidate target's identifier and pre-labeled category below or above each local image, ultimately generating a combined image including multiple local views, i.e., the local evidence image on the right.

[0090] In summary, this embodiment intelligently selects either a first expansion coefficient or a second expansion coefficient based on an adaptive expansion mechanism of candidate box area ratio to generate either a first local region or a second local region, respectively. For small targets, a larger expansion coefficient is used to obtain more contextual information, addressing the semantic ambiguity of small targets; for large targets, a smaller expansion coefficient is used to focus on subject details, reducing background interference. Through differentiated local region generation strategies, the system improves the information quality and effectiveness of the local evidence map on the right, enabling the second model to more accurately classify individual candidate targets, thereby improving the construction quality of the automated labeled dataset and the performance of the final open-set object detector.

[0091] In one optional embodiment, the complete evidence map is input into the second model for correction to obtain the correction result, including: determining whether there is a real target in the candidate box of the candidate target; if there is a real target, the retention status is marked as retained; if there is no real target, the retention status is marked as discarded; when the retention status is retained, it is determined whether the real category of the target in the candidate box is consistent with the pre-labeled category of the candidate target; if the real category is inconsistent with the pre-labeled category, the pre-labeled category is corrected to the real category, and the real category is used as the final category of the candidate target, generating a judgment reason indicating category correction; if the real category is consistent with the pre-labeled category, the pre-labeled category is used as the final category of the candidate target, and a judgment reason indicating category consistency is generated; integrating the retention status, final category and judgment reason of the candidate target to generate the correction result.

[0092] Optionally, a true target refers to an actual object in the image that has clear visual characteristics, distinguished from background noise, occlusions, or false detection areas.

[0093] Optionally, the retained state is a binary label used to indicate whether the candidate target is considered a valid training sample.

[0094] Optionally, the second model receives the complete evidence map as input and analyzes the local cropped region corresponding to each candidate target in the right-hand local evidence map and the contextual information in the left-hand localization map. The second model uses visual feature extraction and semantic understanding to determine whether the region contains a target object that conforms to the business definition. If the region is only background, blurry, or a complete false positive, the model marks it as discarded; if the region contains a clearly identifiable target object, it is marked as retained. Through data cleaning, the system removes a large number of false positives generated by the first model, helping to ensure that subsequent training data only includes valid target samples, thereby improving the efficiency and quality of model training.

[0095] Optionally, the true category refers to the most accurate object category independently determined by the second model based on visual evidence.

[0096] Optionally, the pre-labeled category refers to the category label initially given by the first model.

[0097] Optionally, the reason for the judgment is the natural language explanation of the model output, explaining why the category is retained, discarded, or modified.

[0098] Optionally, for candidate targets marked as reserved, the second model performs in-depth analysis of local details and contextual relationships to determine their true category. The system compares the true category with the pre-labeled category. If they differ, the model updates the true category to the final category and explains the reason for the correction in the reasoning section (e.g., "originally labeled as vehicle, actually a motorcycle"). If they are the same, the pre-labeled category is directly used as the final category, and consistency is confirmed in the reasoning section. Through the above steps, potential category errors in the first model can be corrected, which helps improve the accuracy of training labels. At the same time, by generating reasoning, the interpretability and traceability of data labeling are increased, facilitating manual review or subsequent data analysis.

[0099] In summary, this embodiment uses a second model to discriminate candidate targets one by one, achieving a dual correction mechanism from "effective target selection" to "accurate category correction." By retaining state markers, false detection noise is removed; by comparing and correcting the true category with the pre-labeled category, the accuracy of category labels is improved; and by generating judgment reasons, rich semantic information is provided. Through the above processes, the system constructs a high-confidence automated annotation dataset, reducing the cost of manual annotation and improving the quality of training data, laying a solid data foundation for subsequent training of a high-precision open-set object detector.

[0100] In one optional embodiment, a prompt word feature library is constructed based on the target dataset, including: extracting category names from the target dataset to construct a set of descriptive texts; encoding the set of descriptive texts using a text encoder to obtain a set of semantic features; selecting K semantic features from the set of semantic features as semantic anchors, wherein the semantic anchors are used to characterize the most dispersed semantic features in the set of semantic features, and K is a positive integer greater than or equal to 1; and performing multi-head attention aggregation based on the semantic anchors and the set of semantic features to obtain the prompt word feature library.

[0101] Optionally, the description text set refers to a list of texts consisting of all deduplicated category names, used to represent all semantic categories that the detector needs to identify.

[0102] Alternatively, a text encoder refers to a pre-trained graph-text alignment model used to map natural language text to a high-dimensional vector space.

[0103] Optionally, the semantic feature set refers to the set of feature vectors obtained by encoding each category name in the text set, where each vector represents the semantic information of the corresponding category.

[0104] Optionally, the system will describe each text input text encoder in the text set. The encoder processes the text sequence through a multi-layer neural network and outputs a fixed-dimensional vector representation. Finally, all vectors are aggregated to form a semantic feature set.

[0105] Optionally, a semantic anchor point refers to a representative feature vector selected from the set of semantic features, and the semantic anchor point covers the main distribution area of ​​the semantic space.

[0106] Optionally, the most dispersed distribution means maximizing the distance between semantic anchors, covering diversity, and avoiding multiple anchors clustering in the same semantic cluster.

[0107] Optionally, the system employs a farthest distance sampling algorithm or a similar clustering strategy to iteratively select K feature vectors from the semantic feature set as anchors. First, a feature is randomly selected. Then, from the remaining features, the feature furthest from the selected anchor is chosen as the second semantic anchor, and so on, until K semantic anchors are selected. Subsequently, the semantic anchors are used as queries in the multi-head attention mechanism, with the semantic feature set serving as the key and value, and input into the multi-head attention module. The multi-head attention module calculates the attention weights of each anchor with all semantic features, and performs a weighted summation of the semantic features based on these weights to obtain updated anchor features. These updated anchor features constitute the final prompt word feature library.

[0108] In summary, this embodiment extracts categorized text from the target dataset and encodes it as semantic features. It then utilizes longest-distance sampling to select representative semantic anchors and employs a multi-head attention mechanism to fuse all semantic information for aggregation and updating, thus constructing an efficient and comprehensive prompt word feature library. This feature library represents a rich semantic space with a small number of semantic features, reducing storage and computational overhead while enhancing the expressive power of the features through semantic fusion. It provides dynamic query support for open-set object detectors, improving the model's robustness and adaptability in complex scenarios.

[0109] In an optional embodiment, after obtaining the prompt word feature library, the method further includes: for positive sample candidate boxes in the current batch, calculating the attention weight of the j-th semantic feature in the prompt word feature library for the positive sample candidate boxes in the current batch, wherein the positive sample candidate box represents a candidate box with a real category label and satisfying a preset confidence threshold, and j is a positive integer greater than or equal to 1; calculating the update amount of the j-th semantic feature in the prompt word feature library based on the attention weight, the fusion mapping result of the normalized visual features of the positive sample candidate box and the corresponding real category query features; and updating the j-th semantic feature in the prompt word feature library based on the update amount.

[0110] Optionally, the current batch refers to a small batch of data processed during each forward propagation in the training process of the target model.

[0111] Optionally, a positive sample candidate box refers to a candidate detection box that not only contains a real target in the image but also meets a pre-set confidence threshold.

[0112] Optionally, attention weight refers to the degree of matching or relevance score between semantic features and each positive candidate box in the current batch.

[0113] Optionally, the system iterates through all positive candidate boxes in the current batch. For each semantic feature (index j) in the prompt word feature library, it calculates the similarity (such as dot product or cosine similarity) between the feature and the normalized visual features of each positive candidate box, and obtains the attention weight of the semantic feature for each positive sample through normalization.

[0114] Optionally, the true category query feature refers to the query vector corresponding to the true label of the positive sample candidate box, which is usually generated by the text encoder and enhanced by the prompt lexicon.

[0115] Optionally, the fusion mapping result refers to the target representation obtained by fusing the visual features of the positive sample with its corresponding semantic query features, reflecting the ideal position of the sample in the semantic space.

[0116] Optionally, the update amount refers to the direction and magnitude of adjustment required for the j-th semantic feature.

[0117] Optionally, the system utilizes attention weights to perform a weighted summation of the fusion mapping results for each positive candidate box. Since the attention weights reflect the correlation between semantic features and samples, samples with higher weights can have a greater impact on the update amount. Finally, the j-th semantic feature at time t is obtained through weighted summation. The cumulative update amount under the current batch is shown in formula (2).

[0118] (2)

[0119] in, This indicates the total number of positive candidate boxes processed in the current batch. This represents the attention weight for sample i. This represents the fusion mapping between normalized visual features of positive samples and query features. This represents the normalized visual feature of the i-th positive sample in the current batch. This represents the query feature of the i-th real category.

[0120] Optionally, updating refers to applying the calculated update amount to the existing semantic features, typically using momentum updates or gradient descent. In practice, the system combines the j-th semantic feature with its update amount.

[0121] In summary, this embodiment proposes a dynamic update mechanism for the prompt word feature library based on positive sample candidate boxes. By calculating the attention weights of semantic features and positive samples, and combining them with real category query features for weighted updates, the feature library achieves online self-evolution, thereby reducing the pollution of the feature library by negative sample noise and contributing to the correctness of the update direction. At the same time, by independently calculating the update amount of each semantic feature, it prevents all features from being homogenized by batch average features, which contributes to the diversity and fine-grained discriminative ability of the feature library.

[0122] In one optional embodiment, based on the prompt word feature library, attention enhancement and weighted summation are performed on the initial query features to generate target query features, including: obtaining a preset fusion weight coefficient; using the prompt word feature library to enhance the attention of the initial query features to obtain enhanced query features; weighting and summing the initial query features and enhanced query features according to the fusion weight coefficient, and normalizing the summation result to obtain the target query features.

[0123] Optionally, the fusion weight coefficient is a pre-set numerical parameter or determined through hyperparameter optimization, used to control the contribution ratio of the initial query features and the enhanced query features in the final fusion result. This coefficient is typically a scalar between 0 and 1, or a learnable parameter vector.

[0124] Optionally, the initial query features refer to the raw query vector directly encoded by the text encoder, which mainly includes general linguistic semantic information.

[0125] Optionally, enhanced query features refer to query vectors that have been filtered and enhanced by the prompt word feature library, incorporating visual semantic prototype information from specific business scenarios.

[0126] Optionally, the system uses the initial query features as the query vector and the semantic features from the prompt word feature library as keys and values, inputting them into the multi-head attention mechanism module. The module calculates the attention score between the initial query features and the key vector, generates target weights through normalization, and then performs weighted aggregation on the value vectors based on these target weights to obtain the enhanced query features. Subsequently, the weighted summation result is normalized to obtain the target query features. , as in formula (3).

[0127] (3)

[0128] in, Indicates the initial query characteristics. This indicates enhanced query features. Indicates the fusion weight coefficient. This represents the normalization function.

[0129] In an optional embodiment, the training step of the open set object detector further includes: updating model parameters according to the overall loss, wherein the overall loss is a weighted sum of bounding box regression loss, object confidence loss, classification loss based on query feature similarity, and cue word feature library constraint term loss, the bounding box regression loss is used to optimize the deviation between the predicted coordinates and the true labeled coordinates of positive sample candidate boxes, the object confidence loss is used to optimize the judgment of object existence, the classification loss based on query feature similarity is used to optimize the matching of query features and visual features, and the cue word feature library constraint term is used to constrain the update direction of the cue word feature library.

[0130] Optionally, the system first calculates the losses for each component. For bounding box regression loss, CIoU (Complete Intersection over Union) loss or GIoU (Generalized Intersection over Union) loss is typically used to calculate the differences between the predicted and ground truth bounding boxes in terms of overlap area, center point distance, and aspect ratio, thereby optimizing the accuracy of coordinate prediction. For target confidence loss, binary cross-entropy loss is used to determine whether each candidate box contains a ground truth target, thereby optimizing the binary classification judgment of target existence. For classification loss based on query feature similarity, the cosine similarity or dot product between the normalized visual features of the detection box and the corresponding category query features is calculated and optimized using negative log-likelihood loss or contrastive loss, which helps to align visual features and query features of the same category in the vector space. For the cue word feature library constraint loss, based on the difference between the semantic prototype update amount calculated in the previous embodiments and the original features, regularization constraints are applied, such as limiting the update magnitude or maintaining the discreteness between prototypes, to optimize the stability of the feature library. Based on preset weight coefficients, the bounding box regression loss, target confidence loss, classification loss based on query feature similarity, and suggestion word feature library constraint term loss are linearly weighted and summed to obtain the total loss. , as in formula (4).

[0131] (4)

[0132] in, This represents the bounding box regression loss. Indicates the target confidence loss. This represents the classification loss based on query feature similarity. This represents the loss due to constraints in the feature library of prompt words. , These are the preset weighting coefficients.

[0133] Optionally, the system calculates the gradients of each parameter of the target model (including the backbone network, detector head, text encoder, and prompt word feature library) based on the overall loss using the backpropagation algorithm, and updates the parameters using an optimizer.

[0134] In summary, this embodiment achieves end-to-end joint training of the open-set object detector by constructing a multi-task overall loss function that includes bounding box regression, object confidence, query feature classification and matching, and feature library constraints. The bounding box regression loss and the optimized object confidence loss contribute to the spatial accuracy of detection and the reliability of existence judgment; the query feature similarity-based classification loss utilizes dynamically generated query features to improve category recognition capabilities under open vocabulary; and the cue word feature library constraint loss contributes to the stability and generalization of the feature library during training. Through this multi-task joint optimization mechanism, the system enables the object model to simultaneously consider geometric accuracy, semantic accuracy, and feature library robustness, improving the overall detection performance in complex scenarios.

[0135] Figure 2 This is an architecture diagram of an optional complex scene recognition system based on a multimodal intelligent agent, according to an embodiment of this application. For example... Figure 2 As shown, the system consists of three parts: the construction of the target dataset, the training of the open-set object detector, and online inference. The construction of the target dataset includes pre-annotating candidate targets based on the first model, constructing a complete evidence map, correcting the complete evidence map based on the second model to obtain the labeled dataset, and then mixing the labeled dataset with a pre-general open dataset to generate the target dataset. The training of the open-set object detector includes initializing the cue word feature library, extracting image features from the target training set and predicting candidate boxes, encoding and enhancing query words, updating the cue word feature library, query feature-driven candidate box classification, and updating the parameters of the cue word feature library. The online inference stage includes switching task types based on user input information, determining target events based on task types, thus determining the target category set, obtaining target detection evidence through the open-set object detector, and making comprehensive judgments and inferences about target events based on the target detection evidence.

[0136] Figure 3 This is a schematic diagram of an optional complex scene recognition device based on a multimodal intelligent agent according to an embodiment of this application. According to another aspect of an embodiment of this application, a complex scene recognition device based on a multimodal intelligent agent is also provided, including: a first determining unit 301, a second determining unit 302, a first detecting unit 303, and a second detecting unit 304.

[0137] The system comprises the following components: a first determining unit 301, used to determine the task type based on the user-input image and natural language request; a second determining unit 302, used to determine the target category set through a multimodal large model when the task type is a target event, wherein the target event represents a semantic task composed of multiple target objects, spatial relationships, scene context states, or temporal logic combinations, and the target category set represents the specific object category to be identified in the target event; a first detection unit 303, used to map the target category set to a query category set, and input the image and query category set into an open-set target detector to obtain target detection evidence, wherein the open-set target detector uses a dynamic query mechanism based on a prompt word feature library to predict the target category of the image; and a second detection unit 304, used to input the image, target detection evidence, and natural language request into the multimodal large model to obtain the judgment conclusion and supporting evidence for the target event.

[0138] Optionally, the second determining unit 302 includes: an extraction subunit, used to perform semantic analysis on the natural language request using a multimodal large model to extract key entities and state features related to the target event; a generation subunit, used to generate corresponding category description text based on the key entities and state features and in combination with a preset business semantic mapping table; and a processing subunit, used to perform deduplication and standardization processing on the category description text to form a target category set.

[0139] Optionally, the first detection unit 303 includes: an acquisition subunit, used to input the image and a query category set into an open-set target detector, and acquire initial query features corresponding to the query category set through the open-set target detector, wherein the initial query features are obtained by encoding any query category text in the query category set through a text encoder; a first generation subunit, used to perform attention enhancement and weighted summation on the initial query features based on a prompt word feature library, and generate target query features, wherein the prompt word feature library is trained based on a target dataset; a second generation subunit, used to acquire multi-scale visual features of the image, and generate a set of detection boxes based on the multi-scale visual features, wherein each detection box in the detection box set is associated with a normalized visual feature, coordinates, and confidence score; a determination subunit, used to determine the similarity between the target query features and the normalized visual features of each detection box in the detection box set, and determine the predicted category of the detection box based on the similarity; and an evidence generation subunit, used to obtain target detection evidence based on the predicted category, coordinates, and confidence score of each detection box.

[0140] Optionally, the complex scene recognition device based on multimodal intelligent agents further includes: a pre-labeling unit, used to acquire an unlabeled set of business images, and use a first model to pre-label each image in the set of business images to obtain a set of candidate targets, wherein the set of candidate targets includes multiple candidate targets, and each candidate target includes a candidate bounding box and a pre-labeled category; an evidence map construction unit, used to construct a complete evidence map for each candidate target in the set of candidate targets, wherein the complete evidence map includes a left-side localization map and a right-side local evidence map, the left-side localization map is used to show the spatial position and contextual relationship of the candidate target in the corresponding image, and the right-side local evidence map is stitched together from the local cropped images of the candidate targets after area adaptive expansion; a correction unit, used to input the complete evidence map into a second model for correction to obtain a correction result, wherein the second model is used to perform target-by-target category discrimination for each candidate target, and the correction result includes the final category, retention status and judgment reason; a dataset construction unit, used to construct a labeled dataset based on all candidate targets with a retention status and the final category corresponding to the candidate targets, and merge the labeled dataset with a general open dataset to form a target dataset; and a feature library construction unit, used to construct a prompt word feature library based on the target dataset.

[0141] Optionally, the complex scene recognition device based on multimodal intelligent agents further includes: a third determining unit, used to determine the proportion of the area of ​​the candidate box in each candidate target in the candidate target set to the total area of ​​the corresponding image; a fourth determining unit, used to determine a target expansion coefficient according to the proportion, wherein the target expansion coefficient is a first expansion coefficient or a second expansion coefficient, and the second expansion coefficient is less than the first expansion coefficient; a first expansion unit, used to expand the candidate box using the first expansion coefficient to obtain a first local region when the proportion is less than a preset first threshold; a second expansion unit, used to expand the candidate box using the second expansion coefficient to obtain a second local region when the proportion is greater than or equal to the preset first threshold; and a stitching unit, used to stitch together the first local regions or second local regions corresponding to all candidate targets according to a preset stitching method to form the right-side local evidence image.

[0142] Optionally, the correction unit includes: a first judgment subunit, used to determine whether there is a real target in the candidate box of the candidate target; if there is a real target, the retention status is marked as retained; if there is no real target, the retention status is marked as discarded; a second judgment subunit, used to determine whether the real category of the target in the candidate box is consistent with the pre-labeled category of the candidate target when the retention status is retained; if the real category is inconsistent with the pre-labeled category, the pre-labeled category is corrected to the real category, and the real category is used as the final category of the candidate target, generating a judgment reason indicating category correction; if the real category is consistent with the pre-labeled category, the pre-labeled category is used as the final category of the candidate target, and a judgment reason indicating category consistency is generated; and an integration subunit, used to integrate the retention status, final category, and judgment reason of the candidate target to generate a correction result.

[0143] Optionally, the feature library construction unit includes: a construction subunit for extracting category names from the target dataset and constructing a descriptive text set; an encoding subunit for encoding the descriptive text set using a text encoder to obtain a semantic feature set; a selection subunit for selecting K semantic features from the semantic feature set as semantic anchors, where the semantic anchors are used to represent the most dispersed semantic features in the semantic feature set, and K is a positive integer greater than or equal to 1; and an aggregation subunit for performing multi-head attention aggregation based on the semantic anchors and the semantic feature set to obtain a prompt word feature library.

[0144] Optionally, the complex scene recognition device based on multimodal intelligent agents further includes: a first calculation unit, used to calculate the attention weight of the j-th semantic feature in the prompt word feature library for the positive sample candidate boxes in the current batch, wherein the positive sample candidate box represents a candidate box with a real category label and satisfying a preset confidence threshold, and j is a positive integer greater than or equal to 1; a second calculation unit, used to calculate the update amount of the j-th semantic feature in the prompt word feature library based on the attention weight, the fusion mapping result of the normalized visual features of the positive sample candidate box and the corresponding real category query features; and a feature library update unit, used to update the j-th semantic feature in the prompt word feature library based on the update amount.

[0145] Optionally, the first generation subunit includes: an acquisition module for acquiring preset fusion weight coefficients; an attention enhancement module for enhancing the attention of the initial query features using a prompt word feature library to obtain enhanced query features; and a processing module for weighted summation of the initial query features and enhanced query features according to the fusion weight coefficients, and normalization of the summation result to obtain the target query features.

[0146] Optionally, the training steps of the open-set object detector further include: updating model parameters based on the overall loss, wherein the overall loss is a weighted sum of bounding box regression loss, object confidence loss, classification loss based on query feature similarity, and cue word feature library constraint term loss. The bounding box regression loss is used to optimize the deviation between the predicted coordinates and the true labeled coordinates of positive sample candidate boxes, the object confidence loss is used to optimize the judgment of object existence, the classification loss based on query feature similarity is used to optimize the matching of query features and visual features, and the cue word feature library constraint term is used to constrain the update direction of the cue word feature library.

[0147] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the above-described complex scene recognition method based on multimodal intelligent agents.

[0148] According to another aspect of the embodiments of this application, an electronic device is also provided, including one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors cause the one or more processors to perform the above-described complex scene recognition method based on multimodal intelligent agents.

[0149] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program or instructions, which, when executed by a processor, implement the above-described complex scene recognition method based on a multimodal intelligent agent.

[0150] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0151] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0152] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0153] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0154] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0155] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0156] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for complex scene recognition based on multimodal intelligent agents, characterized in that, include: Determine the task type based on user-input images and natural language requests; When the task type is a target event, a target category set is determined through a multimodal large model. The target event represents a semantic task based on a combination of multiple target objects, spatial relationships, scene context states, or temporal logic. The target category set represents the specific object category that needs to be identified in the target event. The target category set is mapped to the query category set, and the image and the query category set are input into the open set target detector to obtain target detection evidence. The open set target detector uses a dynamic query mechanism based on the prompt word feature library to predict the target category of the image. The image, the target detection evidence, and the natural language request are input into the multimodal large model to obtain the judgment conclusion and supporting evidence for the target event.

2. The method according to claim 1, characterized in that, When the task type is a target event, the target category set is determined using a multimodal large model, including: The multimodal large model is used to perform semantic analysis on the natural language request to extract key entities and state features related to the target event; Based on the key entities and state features, and combined with a preset business semantic mapping table, corresponding category description text is generated; The category description text is deduplicated and standardized to form the target category set.

3. The method according to claim 1, characterized in that, The target category set is mapped to a query category set, and the image and the query category set are input into an open-set target detector to obtain target detection evidence, including: The image and the query category set are input into the open set target detector, and the initial query features corresponding to the query category set are obtained by the open set target detector. The initial query features are obtained by encoding any query category text in the query category set by a text encoder. Based on the prompt word feature library, attention enhancement and weighted summation are performed on the initial query features to generate target query features, wherein the prompt word feature library is trained based on the target dataset; The multi-scale visual features of the image are obtained, and a set of detection boxes is generated based on the multi-scale visual features. Each detection box in the set of detection boxes is associated with a normalized visual feature, coordinates, and confidence score. Determine the similarity between the target query feature and the normalized visual features of each detection box in the detection box set, and determine the predicted category of the detection box based on the similarity. The target detection evidence is obtained based on the predicted category, coordinates, and confidence level of each detection box.

4. The method according to claim 3, characterized in that, Before generating target query features by performing attention enhancement and weighted fusion on the initial query features based on the prompt word feature library, the method further includes: A set of unlabeled business images is obtained, and a first model is used to pre-label each image in the set of business images to obtain a set of candidate targets. The set of candidate targets includes multiple candidate targets, and each candidate target includes a candidate bounding box and a pre-labeled category. For each candidate target in the candidate target set, a complete evidence map is constructed, wherein the complete evidence map includes a left positioning map and a right partial evidence map. The left positioning map is used to show the spatial position and contextual relationship of the candidate target in the corresponding image, and the right partial evidence map is formed by stitching together the partial cropped images of the candidate target that have been adaptively expanded in area. The complete evidence map is input into the second model for correction to obtain the correction result. The second model is used to perform target-by-target classification for each candidate target. The correction result includes the final category, retention status and judgment reason. Based on all candidate targets whose retention status is retained and the final category corresponding to the candidate targets, a labeled dataset is constructed, and the labeled dataset is merged with a general open dataset to form the target dataset; Based on the target dataset, the prompt word feature library is constructed.

5. The method according to claim 4, characterized in that, Before constructing the complete evidence map, the method further includes: For each candidate target in the candidate target set, determine the proportion of the area of ​​the candidate bounding box in that candidate target to the total area of ​​the corresponding image; The target expansion coefficient is determined according to the ratio, wherein the target expansion coefficient is a first expansion coefficient or a second expansion coefficient, and the second expansion coefficient is less than the first expansion coefficient; If the ratio is less than a preset first threshold, the candidate box is expanded using the first expansion coefficient to obtain a first local region; When the ratio is greater than or equal to a preset first threshold, the candidate box is expanded using the second expansion coefficient to obtain a second local region; All candidate targets are spliced ​​together according to a preset splicing method to form the right-side local evidence map.

6. The method according to claim 4, characterized in that, The complete evidence map is input into the second model for correction to obtain the correction result, including: Determine whether there is a real target in the candidate box of the candidate target. If there is a real target, mark the reserved state as reserved; if there is no real target, mark the reserved state as discarded. When the reserved state is reserved, it is determined whether the true category of the target in the candidate box is consistent with the pre-labeled category of the candidate target. If the true category is inconsistent with the pre-labeled category, the pre-labeled category is corrected to the true category, and the true category is taken as the final category of the candidate target, generating a judgment reason indicating category correction; if the true category is consistent with the pre-labeled category, the pre-labeled category is taken as the final category of the candidate target, and a judgment reason indicating category consistency is generated. The correction result is generated by integrating the retention status, final category, and judgment reasons of the candidate targets.

7. The method according to claim 4, characterized in that, Based on the target dataset, the prompt word feature library is constructed, including: Extract category names from the target dataset and construct a set of descriptive text. The set of descriptive texts is encoded using a text encoder to obtain a set of semantic features; K semantic features are selected from the set of semantic features as semantic anchors, wherein the semantic anchors are used to characterize the most dispersed semantic features in the set of semantic features, and K is a positive integer greater than or equal to 1; Based on the semantic anchors and the semantic feature set, multi-head attention aggregation is performed to obtain the prompt word feature library.

8. The method according to claim 7, characterized in that, After obtaining the prompt word feature library, the method further includes: For positive sample candidate boxes in the current batch, calculate the attention weight of the j-th semantic feature in the prompt word feature library for the positive sample candidate boxes in the current batch, where the positive sample candidate box represents a candidate box with a true category label and meets a preset confidence threshold, and j is a positive integer greater than or equal to 1. Based on the attention weight, the fusion mapping result of the normalized visual features of the positive sample candidate box and the corresponding real category query features, calculate the update amount of the j-th semantic feature in the prompt word feature library; Based on the update amount, update the j-th semantic feature in the prompt word feature library.

9. The method according to claim 3, characterized in that, Based on the prompt word feature library, attention enhancement and weighted summation are performed on the initial query features to generate target query features, including: Obtain the preset fusion weight coefficients; The initial query features are enhanced by using the aforementioned prompt word feature library to obtain enhanced query features; The initial query features and the enhanced query features are weighted and summed according to the fusion weight coefficients, and the summation result is normalized to obtain the target query features.

10. The method according to claim 1, characterized in that, The training steps for the open-set target detector also include: The model parameters are updated based on the overall loss, whereby the overall loss is a weighted sum of bounding box regression loss, target confidence loss, query feature similarity-based classification loss, and the prompt word feature library constraint term loss. The bounding box regression loss is used to optimize the deviation between the predicted coordinates and the true labeled coordinates of positive candidate boxes. The target confidence loss is used to optimize the judgment of target existence. The query feature similarity-based classification loss is used to optimize the matching of query features and visual features. The prompt word feature library constraint term is used to constrain the update direction of the prompt word feature library.

11. A complex scene recognition device based on multimodal intelligent agents, characterized in that, include: The first determining unit is used to determine the task type based on the user's input image and natural language request; The second determining unit is used to determine a set of target categories by using a multimodal large model when the task type is a target event. The target event represents a semantic task composed of multiple target objects, spatial relationships, scene context states, or temporal logic combinations, and the set of target categories represents the specific object categories that need to be identified in the target event. The first detection unit is used to map the target category set to a query category set, and input the image and the query category set into an open set target detector to obtain target detection evidence. The open set target detector uses a dynamic query mechanism based on a prompt word feature library to predict the target category of the image. The second detection unit is used to input the image, the target detection evidence, and the natural language request into the multimodal large model to obtain the judgment conclusion and supporting evidence of the target event.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device containing the computer-readable storage medium performs the complex scene recognition method based on a multimodal intelligent agent as described in any one of claims 1 to 10.

13. An electronic device, characterized in that, It includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the complex scene recognition method based on a multimodal intelligent agent as described in any one of claims 1 to 10.

14. A computer program product, characterized in that, It includes a computer program or instructions that, when executed by a processor, implement the complex scene recognition method based on a multimodal intelligent agent as described in any one of claims 1 to 10.

Images