Fine-grained visual target recognition expert knowledge agent generation method and device

Abstract

The invention discloses a fine-grained visual target recognition expert knowledge agent generation method and device, and the method comprises the steps: generating a first type of cue words of a to-be-recognized image based on the feature sorting and feature weight of diagnosis features, and the semantic description information of the diagnosis features in the to-be-recognized image, the feature sequence and the feature weight are determined according to subjective data and objective data when an expert identifies the same kind of targets; and inputting the field to which the to-be-recognized image belongs, the to-be-recognized image and the first type of cue words into a cognitive process reasoning system based on a large language model to obtain a reasoning text which is used for simulating a visual cognition process when an expert recognizes the to-be-recognized image. According to the method, the reasoning text for simulating the process of identifying the fine-grained target by human experts can be efficiently and automatically generated by utilizing the existing part of expert domain knowledge-driven large models.

Description

Technical Field

[0001] This invention belongs to the field of target recognition technology, specifically relating to a method and apparatus for generating a fine-grained target recognition expert knowledge agent. Background Technology

[0002] Traditional fine-grained object recognition methods mostly rely on training recognition models on general datasets. The image to be recognized is input into the trained model, which then outputs the object recognition result. This method is essentially a data-driven statistical fitting process, lacking semantic representation of the recognition process (such as regions of interest, diagnostic logic, etc.). The output result is merely a black-box classification label, which is difficult to support high-reliability scenarios, such as disease diagnosis and species identification, where the reliability of the recognition result is crucial.

[0003] To address this, some methods attempt to manually construct domain knowledge, with domain experts manually writing the reasoning process and key knowledge during the recognition process in text form. The decision-making chains formed by domain experts through long-term practice possess fine-grained traceability, and their structured cognitive abilities can provide reasonable and professional explanatory texts. However, such explanatory texts are mainly acquired through the long-term experience accumulation and systematic training of human experts. Human expert experience is often highly implicit, exhibits significant individual differences, and is difficult to quantify, severely limiting its reuse and transfer in intelligent systems.

[0004] In recent years, knowledge generation technologies based on Large Language Models (LLMs) and Large Multimodal Models (LMMs) have made significant progress in fields such as visual understanding, image and text interpretation, and question answering systems. Large Language Models, through pre-training on corpora of hundreds of billions of words, can acquire implicit knowledge distillation mechanisms, dynamically synthesizing semantic descriptions of visual features, greatly reducing the cost of semantic knowledge construction, and ensuring a unified output format and generation logic. With the rapid development of LLMs, the ability to recognize and understand visual information has been significantly improved.

[0005] However, current large language models (LLMs) still have significant limitations in vertical domain discrimination tasks (such as species identification and ship identification) where sufficient training samples and corpus knowledge are lacking, especially in scenarios requiring rich expert knowledge support, such as fine-grained target identification. On the one hand, current LLMs perform well in image description generation, but their output often remains at the level of generalizing the surface visual features of the image to be identified, rather than capturing the professional visual criteria used in the recognition process and generating a diagnostic description of the recognition process. On the other hand, existing methods usually rely on structured prompts or manual templates, making it difficult to simulate the step-by-step reasoning and visual attention mechanisms exhibited by human experts in recognition tasks, resulting in generated text content that fails to truly reflect expert-level cognitive logic.

[0006] Therefore, how to integrate the generation efficiency of LLM with the structured advantages of expert cognition to automatically generate expert knowledge that can describe the fine-grained visual target recognition process has become a key technical problem that urgently needs to be solved in the field of visual expert knowledge generation. Summary of the Invention

[0007] This invention provides a method and apparatus for generating a fine-grained visual target recognition expert knowledge agent, which can solve the above-mentioned technical problems.

[0008] In a first aspect, embodiments of the present invention provide a method for generating a fine-grained visual target recognition expert knowledge agent, the method comprising: Based on the feature ranking, feature weights, and semantic description information of the diagnostic features in the image to be identified, a first type of prompt word for the image to be identified is generated. The feature ranking and feature weights are determined based on subjective and objective data from experts when identifying similar targets. The domain to which the image to be identified belongs, the image to be identified, and the first type of prompt words are input into a cognitive process reasoning system based on a large language model to obtain reasoning text, wherein the reasoning text is used to simulate the visual cognitive process of an expert when identifying the image to be identified.

[0009] Secondly, embodiments of the present invention provide a fine-grained visual target recognition expert knowledge agent generation device, comprising: The first type of prompt word generation module is used to generate first type of prompt words for the image to be identified based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be identified. The feature ranking and feature weight are determined based on subjective and objective data of experts when identifying similar targets. A cognitive process reasoning system based on a large language model is used to generate reasoning text based on the domain of the image to be identified, the image to be identified, and the first type of prompt words. The reasoning text is used to simulate the visual cognitive process of an expert when identifying the image to be identified.

[0010] The beneficial effects of this invention compared to the prior art are as follows: This invention generates diagnostic prompts (first-class prompts) for the image to be identified by ranking, weighting, and semantic description of diagnostic features determined based on the subjective experience and objective data of domain experts. This enables the first-class prompts to internalize the dynamic decision-making preferences and reasoning logic of human experts when identifying such targets. By inputting the first-class prompts into a cognitive process reasoning system driven by a large language model, the system can simulate the expert's identification process under the constraints and guidance of the recognition criterion priority (i.e., feature ranking and weighting) framework obtained based on expert knowledge, and generate reasoning text according to the diagnostic logic of human experts. This efficiently and automatically generates a semantic representation of the fine-grained visual target recognition process. Attached Figure Description

[0011] Figure 1 A schematic diagram of an adjacency matrix provided in an embodiment of the present invention; Figure 2 A schematic diagram of an expert scan path network provided in an embodiment of the present invention; Figure 3 A schematic diagram of an original decision matrix provided for an embodiment of the present invention; Figure 4 A flowchart illustrating the implementation of a method for determining feature ranking and feature weights provided in an embodiment of the present invention; Figure 5 A schematic diagram of a sorting matrix provided in an embodiment of the present invention; Figure 6 This is a schematic diagram of the structure of a cognitive process reasoning system provided in an embodiment of the present invention; Figure 7 A flowchart illustrating the implementation of a fine-grained visual target recognition expert knowledge agent generation method provided in an embodiment of the present invention; Figure 8 A schematic diagram of an intermediate template for a first type of prompt word provided in an embodiment of the present invention; Figure 9 A schematic diagram illustrating the generation process of a first type of prompt word in JSON format provided in an embodiment of the present invention; Figure 10 This is a schematic diagram illustrating the evaluation results of reasoning text under an automatic index provided in an embodiment of the present invention; Figure 11 This is a schematic diagram illustrating the evaluation results of reasoning text under a model metric provided in an embodiment of the present invention. Detailed Implementation

[0012] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0013] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0014] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0015] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0016] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0017] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0018] The present invention will be further described in detail below with reference to specific embodiments, but the implementation of the present invention is not limited thereto.

[0019] Example 1 In some embodiments, before reasoning in the recognition process of the image to be recognized, subjective and objective data from experts when recognizing similar targets can be collected. Based on this data, the ranking of diagnostic features that generally play a key role in recognizing this type of image and the weight of each feature can be determined. Then, the diagnostic features, feature ranking, and feature weights of each target type are associated and stored in a database for easy retrieval during subsequent recognition reasoning.

[0020] For example, taking remote sensing images as an example, if the target in the image to be identified is a ship, and the purpose of identification is to identify the model or type of the target, then the diagnostic features of this type of target may include: hull shape, hull size, observation room, bow shape, stern shape, location and number of funnels, antenna array, etc.; if the target is an aircraft, and the purpose of identification is to determine the model or type of the aircraft, then the diagnostic features of this type of target may include: aircraft size, wing position, wing angle, wing planform shape, wingspan, winglets, tail shape, engine position, number of engines, engine shape, etc.

[0021] It should be understood that the target objects listed here are merely examples of targets, and the present invention does not limit the specific type of target in the image to be identified.

[0022] In one possible implementation, multiple features for target identification can be identified from target-related technical literature, such as literature summarizing the features of various ship hulls, aircraft, automobiles, and even biological targets, such as fish. Then, external morphology that is easy to observe can be identified from these features, and semantically redundant and spatially overlapping features can be eliminated to obtain the diagnostic features of this type of target.

[0023] In one possible implementation, experts in the target field can be convened and instructed to perform objective and subjective identification tasks in sequence to obtain objective and subjective data.

[0024] For example, subjective data may include a salience ranking of regions of interest and / or pairwise importance scores of adjacent regions of interest.

[0025] Specifically, the region of interest is the area in the sample image where the diagnostic features are located. For example, the region of interest could be the area in the sample image where the deck is located.

[0026] For example, objective data may include the number of fixations to the region of interest, the duration of fixations to the region of interest, the time of first fixation to the region of interest, and the fixation sequence of the region of interest.

[0027] In one example, during an objective recognition task, an expert can wear an eye tracker and be provided with a monitor and a standard keyboard. The expert sits in front of the monitor, observes sample images, and after identifying a target in the sample image, presses a key and verbally states the target's category / model, etc. While the expert is recognizing the target, the software automatically records the expert's accuracy and reaction time, while the eye tracker records the number of fixations on the region of interest, fixation duration, first fixation time, and fixation sequence, thus obtaining objective data.

[0028] Specifically, the objective recognition task requires experts to sit in front of an eye tracker with their heads fixed and their eyes approximately 100cm away from the display. A 9-point calibration is performed before the task begins to ensure the accuracy of the eye-tracking system. Before the formal task begins, a short practice session is conducted to help experts familiarize themselves with the task process and perform initial device calibration to ensure data accuracy. The image display time is 20 seconds; if the expert does not press the required key, the image is automatically skipped. In each task, images are presented in a random order, and the software automatically records the expert's eye movement data, including fixation points, number of fixations, and fixation duration for each region of interest. The collected eye movement data is then processed to obtain raw objective data, including the number of fixations, duration, first fixation time, and fixation sequence. After noise filtering of the raw objective data (e.g., saccades <40 ms or >1000 ms are excluded), the data is extracted according to the region of interest to obtain the final objective data.

[0029] In one example, the subjective recognition task presents rearranged sample images to experts sequentially. Experts recall and freely recount the recognition process for each image, completing corresponding scoring scales. These scales include scoring indicators for the extraction and judgment criteria of diagnostic features during the recognition process, further revealing the experts' recognition strategies and judgment standards. After each recall, experts are asked to rank the diagnostic features used in their recognition process by saliency, obtaining a saliency ranking of the diagnostic features. Then, they score the pairwise importance of adjacent features in the ranking, obtaining pairwise importance scores for adjacent regions of interest.

[0030] For example, for each aircraft model, the display presents a set of predefined candidate attributes (e.g., aircraft size, wing position, wing angle, wing planform, wingspan, winglets, tail shape, engine position, number of engines, engine shape, etc.). Participants can rank these features in descending order of perceived importance, referencing actual usage in the identification process. Then, a standardized interval scale is used to rate the pairwise importance between adjacent features, thereby quantifying the relative contribution of each feature to the decision.

[0031] This subjective data collection method not only captures the explicit content of expert decisions, but also the internal prioritization mechanisms that govern real-world diagnostic behavior.

[0032] In one possible implementation, an expert scan path network can be drawn first based on objective data. Then, feature evaluation indices for diagnostic features are calculated based on the expert scan path network and subjective data. An index vector for each region of interest is constructed based on the feature evaluation indices for each diagnostic feature, and these vectors are concatenated to obtain the original decision matrix. Finally, based on the improved VIKOR round-by-round extraction decision algorithm (the specific process can be found in patent publication number CN117422322A), the feature ranking and feature weights of the diagnostic features are determined according to the original decision matrix.

[0033] In one example, before drawing the expert scan path network, the adjacency matrix of each expert can be constructed based on objective data. Then, the adjacency matrices of all experts are concatenated to obtain the scan matrix, and the expert scan path network is drawn based on the scan matrix.

[0034] For example, see Figure 1 The rows and columns of the adjacency matrix represent the regions of interest (ROIs) for diagnostic features, and the numerical values ​​in the matrix represent the number of scans between two ROIs. The scan order can be indicated by the number of rows and columns. See, for example... Figure 1 The value of 5 in row P2 and column P3 indicates that there are 5 scan paths from P2 to P3.

[0035] In one example, nodes corresponding one-to-one with regions of interest (ROIs) can be created based on the saccade matrix. Then, edges are added between nodes based on each gaze sequence of the ROIs in the objective data. The weight of the edge reflects the number of saccades between the two ROIs it connects, and the direction indicates the saccade order. After adding the saccade paths for each expert, the expert saccade path network is obtained (see...). Figure 2 ).

[0036] For example, in an expert scan path network, the direction of scan can be reflected in the edges of the graph. Forward scans and backward scans can be distinguished by the direction of the edge edges (e.g., start edge, end edge).

[0037] In one example, feature evaluation metrics may include: degree centrality, betweenness centrality, proximity centrality, and weight coefficients for diagnostic features.

[0038] For example, degree centrality The degree (or value) of nodes in the expert scan path network is used to represent the region of interest; a higher degree indicates that the region of interest has been scanned more frequently. This is achieved through... This can highlight the importance of diagnostic features.

[0039] For example, betweenness centrality The number of shortest paths to a given node can be used as a metric, representing the number of shortest paths passing through that node in the expert scan path network. This metric reflects the crucial role of diagnostic features in the expert scan path.

[0040] For example, proximity centrality The average shortest path between a node and other nodes in the expert scan path network represents the connectivity between the diagnostic feature corresponding to that node and other diagnostic features, reflecting its centrality in the network.

[0041] For example, the importance weight vector of diagnostic features in different identification tasks involving the same expert can be calculated based on subjective data, and the weight coefficient of the diagnostic features can be obtained by averaging the weight vectors of each expert.

[0042] In one example, see Figure 3 In the original decision matrix, rows represent candidate points, i.e. regions of interest, and columns represent candidate indicators, i.e. feature evaluation indicators.

[0043] For example, the multi-criteria compromise solution ranking method (Vise Kriterijumska Optimizacija-Kompromisno Resenje, VIKOR) can be based on... Aggregation functions comprehensively evaluate candidates by examining the distance between the evaluated object and the "positive ideal solution" and the "negative ideal solution," enabling decision-makers to quickly obtain compromise solutions under complex criterion conditions and providing effective support for candidate selection. However, the final compromise ranking result of the VIKOR method is affected by the decision coefficient v. This coefficient is used to achieve a balance between "maximizing group utility" and "minimizing negative impact," and its value can continuously vary within the interval [0, 1]. In practical applications, it is difficult to clearly define the optimal expected value of v. In most cases, v is usually set to 0.5 to simultaneously consider both objectives, but this does not guarantee that the ranking result under other values ​​will necessarily be worse than the default value, nor does it guarantee that this ranking result is the optimal solution in the actual use case. To overcome this limitation, this invention adopts an improved VIKOR round-by-round extraction decision algorithm. This algorithm can select the optimal decision coefficient v within the interval [0, 1], and then obtain the optimal ranking result and weights based on the selected decision coefficient.

[0044] Example 2 Based on Example 1, Figure 4 The diagram illustrates a flowchart of a method for determining feature ranking and feature weights according to an embodiment of the present invention. As an example and not a limitation, the method may include steps S401-S404, which are described below: S401, calculate the group utility and individual regret value for each diagnostic feature based on the original decision matrix.

[0045] In one example, the original decision matrix can be normalized first, and then the group utility and individual regret value of each diagnostic feature can be calculated based on the normalized original decision matrix.

[0046] For example, the group utility of a diagnostic feature can satisfy the following formula:

[0047] in, For the group utility of the i-th diagnostic feature, Let the weight of the j-th feature evaluation index be . , These represent the maximum and minimum values ​​of the j-th column in the normalized original decision matrix (i.e., the maximum and minimum values ​​of the j-th feature evaluation index). Let be the value in the i-th row and j-th column of the original decision matrix after normalization (i.e., the j-th feature evaluation index value of the i-th diagnostic feature), and n be the total number of feature evaluation indicators, which can be 4 here.

[0048] For example, the individual regret value of a diagnostic feature can satisfy the following formula:

[0049] in, Let be the individual regret value for the i-th diagnostic feature.

[0050] S402, based on group utility and individual regret value, calculate the trade-off measure of each diagnostic feature at different v values, and obtain the ranking of diagnostic features at different v values.

[0051] In one example, the compromise measure of each diagnostic feature under different v values ​​can be calculated first, and then the diagnostic features can be sorted from largest to smallest according to the compromise measure to obtain the ranking of diagnostic features under different v values.

[0052] For example, a compromise measure can satisfy the following formula:

[0053] in, For the i-th diagnostic feature, a trade-off measure , These represent the maximum and minimum group utility values ​​among the various diagnostic features. , These represent the maximum and minimum individual regret values ​​among the various diagnostic features.

[0054] Optionally, in setting intermediate variables , Furthermore, the compromise measure can also be obtained through the formula calculate.

[0055] S403, based on the trade-off measure of each diagnostic feature under different v values, divide the decision value interval to obtain multiple critical rankings of the diagnostic features.

[0056] In one possible implementation, if the same decision coefficients are used... Under these conditions, the tradeoff measure of any two diagnostic features is equal. If the ranking of the two diagnostic features changes, then... Let be a critical decision coefficient. If there are a total of in the interval [0,1]... If there are critical decision coefficients, then the interval [0,1] can be divided into […]. +1 sub-intervals (i.e., decision value intervals). Then, the values ​​of the diagnostic feature at the interval endpoints, 0, or 1 can be used as v values ​​to calculate the diagnostic feature's... +1 critical sorting.

[0057] In one example, the critical decision coefficients can satisfy the following formula:

[0058] in:

[0059]

[0060] .

[0061] S404, determine the feature ranking and feature weights based on the critical ranking of diagnostic features.

[0062] In one example, a ranking matrix P can be constructed based on the critical ranking of each diagnostic feature and the length of its corresponding critical decision interval. Then, the ranking matrix P is extracted round by round to obtain the final feature ranking, and the total score of the diagnostic features during each round of extraction is used as the feature weight of the diagnostic features.

[0063] For example, see Figure 5 In the sorting matrix P, the c-th column corresponds to the length of the c-th critical decision interval, the i-th row corresponds to the i-th feature in the critical sorting, and the elements of the matrix represent the name or index of the i-th feature under the c-th critical sorting.

[0064] This invention determines feature ranking and feature weights based on an improved VIKOR round-by-round extraction decision algorithm. It can comprehensively consider the impact of different v values ​​within the [0,1] interval and can also use comparative analysis to select the ranking result that best balances overall utility and individual shortcomings. Compared with the traditional approach of fixing the v value (e.g., v=0.5), this method can be flexibly adjusted according to specific decision-making needs in practical applications, enhancing the objectivity and robustness of the ranking results.

[0065] Example 3 Figure 6 The diagram shown illustrates the structure of a cognitive process reasoning system provided in an embodiment of the present invention. As an example and not a limitation, the system may include a domain knowledge enhancement acquisition module 610, a feature selection module 620, an attribute extraction module 630, and a multimodal expert knowledge-guided model reasoning module 640.

[0066] In some embodiments, the domain knowledge enhancement acquisition module 610 can retrieve domain knowledge of the image to be identified based on the domain to which the image to be identified belongs. The feature selection module 620 can extract diagnostic features with diagnostic value in the process of identifying the image to be identified based on the first type of prompt words to obtain visually salient features; the attribute extraction module 630 can retrieve the structural attribute information of the visually salient features; and the model reasoning module 640 can integrate domain knowledge, visually salient features, and structural attribute information based on feature ranking to obtain reasoning text.

[0067] In one possible implementation, the domain knowledge enhancement acquisition module 610 can specifically be based on the Retrieval-Augmented Generation (RAG) mechanism, using sparse retrieval algorithms and dense retrieval algorithms to retrieve similar texts from a professional knowledge base related to the domain of the image to be identified, obtaining sparse similar texts and dense similar texts; then, based on Reciprocal Rank Fusion (RRF), these two types of similar texts are fused to obtain the fused text and the fusion score of each text; finally, the top k texts with the highest scores are selected as the domain knowledge of the image to be identified.

[0068] For example, the domain knowledge output by this module can be expressed in structured JSON format, including information source, knowledge text, relevance score, and category label. For instance, domain knowledge could include: "This civil aircraft adopts a low-wing configuration with two turbofan engine nacelles symmetrically suspended under the wings. This configuration helps improve fuel efficiency and reduce noise. The rear fuselage is equipped with a large T-tail, which is common in modern civil airliners and helps improve flight stability and handling performance. The main landing gear uses a dual-axle, six-wheel structure to ensure aircraft stability and runway adaptability during landing. These features combine to present the typical configuration characteristics of a mainline civil airliner." In one example, the knowledge base can be pre-built by the user based on heterogeneous data sources. Taking an aircraft-related knowledge base as an example, it can include aircraft identification manuals, the latest published academic papers on aircraft, structural description documents, online encyclopedia entries, and web page data in the aircraft field.

[0069] For example, data in original formats such as PDF, HTML, and TXT can be uniformly processed into text at the paragraph level using text recognition and segmentation tools, with semantic paragraphs as the smallest index unit, and stored as UTF-8 encoded text.

[0070] In one example, the jieba word segmentation tool can be used to segment the domain description of the image to be identified, and then combined with the BM25 algorithm to build an inverted index for sparse retrieval.

[0071] For example, the BM25 parameters can be k1=1.5 and b=0.75, supporting keyword position weighting and title field priority.

[0072] In one example, during intensive retrieval, the Sentence-BERT model can be used to semantically embed the knowledge base text segment and the domain description of the image to be identified, and all embedding vectors are stored in the Milvus vector database; then, cosine similarity is used as the distance function, and by default, the 50 most similar text records (i.e., densely similar text) are returned.

[0073] In one example, the fusion score of the merged text can be calculated using the following formula:

[0074] in, Let be the fusion score of the d-th fused text, and A be the set of retrieval algorithms (including sparse and vector algorithms). This indicates the text's sorting position in retrieval algorithm a. It is a smoothing constant, with a value range of 60 to 100.

[0075] The RRF fusion algorithm can effectively balance the coverage and ranking depth of different types of search results, and achieve unified optimization of multi-source ranking information.

[0076] The domain knowledge enhancement acquisition module 610 introduces the RAG mechanism to provide structured domain knowledge text to the large language model in the recognition task, thereby enhancing the professionalism, relevance and accuracy of the content generated by the model.

[0077] In one possible implementation, the feature selection module 620 may specifically call a multimodal large language model to parse the image to be recognized, determine the visibility, integrity and local salience of each diagnostic feature in the image to be recognized, and obtain candidate features; then determine whether each candidate feature exists in the image to be recognized and perform multiple rounds of verification to exclude hallucination features; and output the excluded candidate features as the visually salient recognition features.

[0078] For example, the feature selection module 620 can output visually salient features in the form of a structured feature list. Each feature is labeled with its visibility score, diagnostic importance level and expert confidence weight in the image to be identified, providing accurate input for subsequent attribute extraction and interpretation generation.

[0079] In one example, the Multimodal Large Language Model (MLLM) Gemini-2.5-pro can be called to perform deep analysis on the image to be recognized. Combined with feature ranking, multimodal reasoning is used to determine the visibility, completeness, and local saliency of each feature in the image to obtain candidate features.

[0080] In one example, for each candidate feature, MLLM not only determines whether it exists, but also evaluates its discriminative contribution in a specific task scenario based on expert experience priority, and performs multiple rounds of feature verification in combination with structured prompts to eliminate false detections and "illusion" features caused by image occlusion, blurring or noise, and obtain visually salient recognition features.

[0081] The feature selection module employs a structured and objective feature screening method to select visually salient features. This ensures that the model does not generate hallucinatory features during the feature selection process, avoiding misjudgments due to insufficient visual evidence. Furthermore, the results of feature selection directly impact the accuracy and effectiveness of subsequent attribute extraction, ensuring the robustness and professionalism of the overall process.

[0082] In one possible implementation, the attribute extraction module 630 can first call the structured semantic extension path based on the web search engine to construct a query template and query the structural description information of visually salient features online to obtain the first type of attribute information; then, a multimodal large model (such as Qwen2.5-VL, BLIP-2) is used to perform semantic querying and attribute recognition on the structural regions in the image to obtain the second type of attribute information; finally, the semantically similar multi-source information (i.e., the first type of attribute information and the second type of attribute information) is filtered and calibrated to form an accurate and consistent structural attribute description, thus obtaining the structural attribute information.

[0083] For example, structural attribute information may include four core elements: (1) the name of the structure (i.e., the visually salient identifiable feature), such as “airplane size”; (2) the attribute description, which includes fine-grained visual knowledge such as the location, size, length and width of the feature; (3) the source information, which identifies the URL of the source of the information or the name of the multimodal large model; and (4) the confidence score, which indicates the strength of consistency of the fused content.

[0084] In one example, the attribute extraction module 630 can initiate keyword-level requests to search engine interfaces (such as Google Search, Bing API, etc.) based on a query template to automatically collect structural description information from webpage summaries and encyclopedia entries. Then, the text segments returned after syntactic dependency analysis and keyword extraction are filtered for redundant content using a joint BERTScore and TF-IDF scoring mechanism, retaining only text fragments that are semantically highly relevant to the input structural items and have clear descriptions. The original source and extraction timestamp are automatically labeled to enhance verifiability, resulting in the final first-class attribute information.

[0085] In one example, the attribute extraction module 630 can crop the region of interest into sub-images and generate semantic prompts related to its structure, such as "What are the appearance features of this wing?" and "In which civil aircraft is this tail design commonly found?". The input is a multimodal large model, and the model combines the image and prompts to return a natural language answer, including content such as "The leading edge of this low-wing configuration has a 25-degree sweep angle and is equipped with winglets to optimize fuel efficiency" and "The vertical tail adopts a T-shaped design, and its smooth surface helps to improve aerodynamic performance. The vertical distance between the upper edge of the horizontal tail and the bottom of the fuselage is 2.5 meters", as the second type of attribute information.

[0086] In one example, the attribute extraction module 630 can semantically vectorize two types of attribute information related to the same visually salient recognition feature, and then calculate the cosine similarity between the two types of semantic information based on the semantic vectors to quantify the semantic proximity between different texts. Afterwards, texts with high similarity are merged, and different labels are added to texts with significant differences to obtain structural attribute information.

[0087] For example, the attribute extraction module 630 can use the multilingual sentence vector model Sentence-BERT to embed the text into the same semantic space to obtain the semantic vector of the attribute information.

[0088] For example, attribute information with a similarity higher than the merging threshold (e.g., 0.85) can be merged into a single semantic representation, retaining the version with richer semantic content and more complete syntactic structure.

[0089] For example, for attribute information with significant differences, two types of attribute information can be retained separately, and an "empirical attribute" label can be added to the first type of attribute information from expert text or web retrieval content; and a "structural attribute" label can be added to the second type of attribute information from image model inference content.

[0090] Optionally, to further improve the controllability and accuracy of structural attribute information, after obtaining the structural attribute information, the attribute extraction module 630 can introduce a secondary review mechanism after fusing the two types of attribute information. Based on the structural attribute information, prompt words are constructed, such as "Does this description contain speculative or inconsistent information about the aircraft type?" and "Does this structure have diagnostic significance in actual aircraft classification?". These are input into a large model, which scores and analyzes the professionalism, factuality, and redundancy of the structural attribute information, outputting a reliable score on a 5-point scale. When the score is higher than a set threshold (e.g., 4 points or above), the structural attribute information is included in the available attribute set; if the score is insufficient, it is returned to the original information source to reconstruct prompt words for correction.

[0091] By employing the aforementioned attribute extraction method based on multimodal, expert text, and large model cross-validation, this invention can effectively improve the quality control capability of structural semantic content, enabling the generated recognition text to possess stronger interpretability, consistency, and logical traceability, significantly different from traditional prompt generation methods based on simple keywords or template filling.

[0092] In one possible implementation, the model inference module 640 can generate a second type of prompt words based on feature ranking and structural attribute information. Then, using Beam Search (with a width of 3 to 5) as a decoding strategy, the second type of prompt words are input into the large language model to perform multiple rounds of content generation. The generated content is then subjected to duplicate detection and penalty, logical consistency verification, and structural integrity control to obtain the final generated text. Finally, the final generated text is converted into a standard structured format based on an expert language style converter to obtain the inference text.

[0093] For example, the large language model here can be a model that supports multi-turn reasoning capabilities, large-scale context processing capabilities, and uses a prompt structure to drive the generation process (such as GPT-4, Qwen2.5-72B).

[0094] In one example, the model inference module 640 can construct dynamically controlled multi-level prompt templates based on a predefined "expert prompt library". The prompt content of the second type of prompt words no longer uses static instructions or general descriptions, but combines feature ranking results and structural attribute information to generate language prompts with causal logic, following the idea of ​​"starting from the most discriminative structure and guiding it layer by layer to global type inference".

[0095] For example, the second type of prompt can consist of a structural description guidance module, an expert analogy reminder module, a reasoning task instruction module, and a final judgment prompt module.

[0096] For example, the second type of clue could be: its nose section features a unique streamlined fairing, a design commonly found in modern jet airliners that prioritize high-speed performance and fuel efficiency. Combined with this feature, further examine whether the wing-fuselage junction employs a smoothly transitioning blended wing-body design. This design not only reduces air resistance but also enhances flight stability and stealth capabilities. Next, conduct a comprehensive analysis based on the identifying markings on the vertical tail, including but not limited to special paint schemes or identifiers, as well as the specific layout of the engine nacelles and the shape of the air intakes. Based on all visible salient features, such as the design of the winglets, the unique tail arrangement, and the engine configuration, infer the possible category of the aircraft and the basis for this inference. This description avoids directly mentioning aircraft-related structures and analogies, focusing instead on the unique identifying features of civil aircraft.

[0097] In one example, to ensure the stability, logical consistency, and avoidance of redundancy in the generated language, the model inference module 640 can perform duplicate detection and penalty operations. It sets a duplicate penalty coefficient for repeated phrases and structures in the generated content to prevent language redundancy; it performs a logical consistency check operation, performing BERTScore semantic matching analysis on each piece of text and structural attribute information. If the deviation is too large, it is judged as "illusion content" and a regression reconstruction prompt is given; it performs a structural integrity control operation to ensure that the generated output contains no less than 3 feature-attribute corresponding contents and covers at least 1 recognition inference logic and 1 expert basis explanation.

[0098] In one example, an expert language style converter can transform the content generated by a large language model into an expression that closely resembles the writing style of human experts, while outputting the transformed text in a structured format.

[0099] For example, the transformed text can be expressed using analogical, judgmental, and inferential terms (such as "usually indicates", "more likely to belong to", "significantly different"), avoiding the use of vague semantics or uncertain structures.

[0100] For example, in the structured format, the first part is a description of the visually salient features, the second part is the reasoning chain, and the third part is the conclusion judgment.

[0101] Optionally, the final output reasoning text can be controlled within the range of 200 to 300 words to ensure a balance between information density and readability.

[0102] For example, the aircraft exhibits the following characteristics: a low-wing monoplane with a slight dihedral, a swept-back design, and winglets indicating an optimized focus on fuel efficiency; a wingspan of approximately 38 meters and a fuselage length of approximately 47.3 meters; and a typical commercial aircraft profile with a regularly arranged window layout. Based on these structural attributes, the tail system comprises a single large vertical tail and a low-mounted all-moving horizontal tail, providing excellent directional stability and control performance. Two turbofan engines are precisely positioned under the wings, with prominent air intakes further confirming this configuration. Combining these identifying features—particularly the winglets, the unique tail arrangement, and the specific location and number of engines—it can be clearly determined that this aircraft is a member of the Boeing B757 family, specifically conforming to the technical specifications and appearance characteristics of the 2024 B757-200 model.

[0103] The model inference module achieves highly controllable and highly interpretable recognition text output through the construction of a second type of prompt words based on expert prompt chains, as well as structured reasoning guidance, multi-level generation quality control mechanisms, and language style optimization strategies. It can make up for the shortcomings of traditional prompting methods, such as lack of logical constraints, semantic redundancy, and the occurrence of hallucinatory content, and significantly enhance the practicality and credibility of the model in professional image language generation scenarios.

[0104] The system provided by this invention significantly enhances the model's generative capabilities in scenarios lacking target corpora or in non-English contexts by introducing an expert-guided semantic prompting mechanism and embedding structured visual cognitive information. The system does not rely on training samples of the target category, but rather on general visual diagnostic logic and salient feature prompts extracted from expert cognition. Through semantic templates, it guides the large model to generate text according to a logical chain of "image evidence → feature induction → diagnostic description," enabling the construction of high-quality text with diagnostic logic and domain language even when faced with new communication types unseen by the model. Unlike general image-text description generation in typical scenarios, this invention focuses on professional tasks requiring visual recognition criteria (such as communication identification). The generated content not only needs to conform to language norms but also reflect key expert knowledge such as "classification criteria," "structural differences," and "detailed features." By embedding this content into the prompting structure and strongly binding it with image evidence, it can generate reasoning text with significant "traceability" and "decision support."

[0105] Furthermore, this invention, by introducing an expert-driven visual evidence modeling mechanism, achieves a one-to-one correspondence between diagnostic descriptions and key image regions during text generation, fundamentally alleviating the "illusion generation" problem common in large language models. Traditional large models often rely on language pattern-driven approaches in image-text generation tasks, lacking effective constraints on image content. This can lead to the model generating descriptive statements that do not conform to the facts of the image, especially when facing fine-grained, multi-category visual tasks, easily resulting in misjudgments of details, fabricated features, and logical conflicts. This invention, however, employs a salient feature extraction and expert decision path ranking mechanism to explicitly identify structural elements with classification and diagnostic value in the image before generation, and embeds them into the generation context through prompts, ensuring that the language output of the large model is controlled by the image evidence chain. This not only ensures that each diagnostic statement can be traced back to a specific image region, but also guides the language generation order with the help of expert reasoning paths, thereby improving the causality of reasoning and the accuracy of description. Simultaneously, this mechanism strengthens the visual alignment capability of the language model, giving the generated results higher interpretability, consistency, and professional credibility, demonstrating significant technical advantages in high-security or irreversible task scenarios (such as medical diagnosis).

[0106] Furthermore, this invention employs a modular architecture design, dividing the visual expert knowledge generation process into multiple independently operable and combinable functional sub-modules. This design significantly enhances the system's controllability and maintainability, allowing each module to be optimized or replaced according to the task scenario, providing excellent flexible configuration capabilities. For example, in different visual recognition tasks, the system can replace the criterion set used for saliency ranking based on domain characteristics, or update the inference path model based on expert sources, without needing to reconstruct the entire system's logical framework. Simultaneously, this modular architecture is easily extended to other visual interpretation tasks, such as remote sensing image interpretation, industrial defect detection, and lesion recognition; rapid migration and adaptation can be achieved simply by changing the input image and the corresponding domain knowledge base. Moreover, modules communicate collaboratively through standardized semantic interfaces, avoiding error propagation caused by data coupling and improving the overall robustness of the system. More importantly, this design supports accuracy evaluation and incremental iteration of individual modules, facilitating the enhancement of specific capabilities as needed during actual deployment, such as improving the granularity of the diagnostic chain logic or strengthening the consistency of language generation, greatly enhancing the system's engineering feasibility and practical expansion potential in multi-task scenarios.

[0107] Example 4 The fine-grained visual target recognition expert knowledge agent generation method provided in this embodiment of the invention can be applied to electronic devices such as mobile terminals, personal laptops, and supercomputers. This embodiment of the invention does not impose any restrictions on the specific type of electronic device.

[0108] Figure 7 The diagram shown illustrates an implementation flowchart of a fine-grained visual target recognition expert knowledge agent generation method provided by an embodiment of the present invention. As an example and not a limitation, the method may include steps S701-S702, which are described below.

[0109] S701, based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be identified, generate the first type of prompt words for the image to be identified.

[0110] In some embodiments, the first type of prompt words can consist of several levels of sub-templates, specifically including a top-level template, intermediate templates, and a bottom-level template. Placeholders in the templates can be filled with specific feature names, weights, and semantic description text. This hierarchical structure, which can be flexibly replaced and adjusted, enables modular design and reuse of the first type of prompt words.

[0111] For example, the content of the top-level template can be used to determine the overall framework and tone of the prompt, while the intermediate templates are responsible for the layout and connection of feature groups (see...). Figure 8 The underlying template generates semantic description fragments for a single feature.

[0112] Specifically, the top-level template may include fields such as task description, feature list, and prompts. The feature list may be an array containing multiple feature objects, each of which has nested sub-fields such as name, weight, and description.

[0113] Specifically, see Figure 8 The intermediate template can be filled in according to the feature order. First, describe the feature with the highest feature weight and add more detailed semantic fragment descriptions; then describe the features with lower feature weights and add relatively concise descriptions.

[0114] In one possible implementation, the semantic description information of each feature in the prompt word can be generated based on a semantic fragment enhancement mechanism.

[0115] For example, for each feature, its semantic description information may include not only the name, but also descriptive fragments rich in diagnostic significance generated by combining domain knowledge.

[0116] For example, for the diagnostic feature "aircraft size" with the highest feature weight, its semantic description information can emphasize the key information that the wing has a direct impact on flight performance. For the secondary feature "wing position", its semantic description information can briefly indicate the auxiliary role of the presence or absence of this feature in the identification.

[0117] Semantic fragment enhancement mechanisms ensure that prompts are not only a list of features, but also a set of natural language fragments based on expert interpretation experience.

[0118] In one example, templates with different features can be classified, such as structural appearance and electronic devices. The semantic description information can be enhanced with feature classification information based on intra-group cue weight control and cue interpretability control mechanisms.

[0119] For example, the intra-group prompt weight control mechanism refers to adjusting the length of semantic description information and emphasis words of similar features according to their feature weights.

[0120] For example, in equipment feature classification, if the feature weight of engine location is higher than that of wingspan, the emphasis word "primarily equipped" can be added to the semantic description information of the engine to lengthen its semantic description information; the emphasis word "secondarily equipped" can be added to the semantic description information of the wingspan to shorten its semantic description information. By controlling the sentence length, the importance of the former relative to the latter can be highlighted.

[0121] For example, the interpretability control mechanism for prompts is manifested in explicitly or implicitly identifying feature ordering in semantic description information, so that both the large model and the user can understand the logic of the prompts.

[0122] For example, use sequence conjunctions such as "firstly," "secondly," "in addition," and "finally" in the semantic descriptions of different features to implicitly indicate feature order; or use explicit feature order descriptions such as "ordered by importance as follows" to guide the model to reason along the order of expert cognition. See also Figure 8 If the first type of prompt words are output in JSON format, the primary and secondary relationships of the features can be reflected by field names or nesting levels to ensure that the meaning of the first type of prompt words is clear and interpretable.

[0123] In one possible implementation, the semantic description information of the diagnostic features can be provided by experts or generated by the recognition model. The obtained feature ranking, feature weights, and semantic description information can be stored in a database. When obtaining prompt words, a prompt word generation program is invoked, according to... Figure 9 The process shown involves retrieving relevant information from the database, filling in the template, and obtaining the first type of prompt words in JSON format.

[0124] The structured first-class cue words enable the model to obtain each feature, its weight, and meaning by parsing JSON, and then strictly follow the provided structure and order for reasoning. By explicitly including weight ordering information and analysis guidance in the JSON, the model can fully utilize the priority relationships inherent in expert knowledge and simulate the expert's judgment process.

[0125] S702, input the domain of the image to be identified, the image to be identified, and the first type of prompt words into the cognitive process reasoning system based on a large language model to obtain the reasoning text.

[0126] For example, the cognitive process reasoning system can be the system in Embodiment 3 above, which can be used to simulate the visual cognitive process of an expert when recognizing an image to be recognized.

[0127] This invention generates diagnostic prompts (first-class prompts) for images to be identified by ranking, weighting, and semantic description of diagnostic features determined based on the subjective experience and objective data of domain experts. This allows the first-class prompts to internalize the dynamic decision-making preferences and reasoning logic of human experts when identifying such targets. By inputting the first-class prompts into a cognitive process reasoning system driven by a large language model, the system can simulate the expert's recognition process under the constraints and guidance of the recognition criterion priority (i.e., feature ranking and weight) framework obtained based on expert knowledge, and generate reasoning text according to the diagnostic logic of human experts. This results in the efficient and automated generation of semantic representations of fine-grained target visual recognition processes.

[0128] Furthermore, the use of the inference text generated by this invention to explain the fine-grained target recognition process can significantly improve the transparency and interpretability of the recognition process, allowing users to clearly understand the key features and their weights, as well as the reasoning logic, on which the model's decisions are based. The reasoning results are based on feature interpretations that conform to domain consensus, which can significantly improve the credibility of the recognition conclusions and provide users with an easy-to-understand, verifiable, and interactive decision-making basis.

[0129] Furthermore, this invention proposes a "human-machine fusion" paradigm for expert knowledge generation. By systematically collecting and modeling the cognitive processes of human visual experts during fine-grained recognition, including visual attention trajectories (such as fixation areas and eye movement sequences) and reasoning logic structures (such as diagnostic judgment order and feature priority ranking), it transforms implicit expert experience into structured, inputtable prompts, forming semantic prompt templates. Unlike traditional prompting engineering that uses fixed sentence structures or general logical prompts, this method, based on expert cognitive modeling, constructs a prompt structure with greater domain knowledge depth and rationality of judgments. Essentially, it introduces an "expert semantic mediator layer," enabling large models not only to mimic surface text structures in language generation but also to simulate how experts recognize, judge, and express themselves at the semantic level. This human-machine fusion mechanism not only improves the semantic accuracy and logical coherence of the model in complex visual tasks but also significantly enhances the diagnostic, professional, and interpretive capabilities of the generated text, providing a fundamental framework for the reliable generation of large models in high-cognitive tasks. In addition, this paradigm has strong universality and can be extended to application scenarios with extremely high requirements for visual interpretability, such as medical care, security, and industrial quality inspection, thus broadening the transfer path and formal expression of expert knowledge in intelligent systems.

[0130] Example 5 This invention also provides a fine-grained visual target recognition expert knowledge agent generation device, which may include a first type of prompt word generation module and a cognitive process reasoning system based on a large language model.

[0131] For example, the first type of prompt word generation module is used to generate a first type of prompt word for the image to be identified based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be identified. The feature ranking and feature weight are determined based on subjective and objective data of experts when identifying similar targets. The cognitive process reasoning system is used to generate reasoning text based on the domain of the image to be identified, the image to be identified, and the first type of prompt word. The reasoning text is used to simulate the visual cognitive process of experts when identifying the image to be identified.

[0132] To better illustrate the beneficial effects of the present invention, the following simulation experiments were conducted: Simulation Experiment 1 For example, Experiment 1 employs a series of automated metrics, including ROUGE, BERTScore, MoverScore, MeteorScore, and BARTScore, to comprehensively evaluate the reasoning text generated by this invention and the reasoning text generated by traditional prompting engineering methods. These metrics are widely used in natural language processing tasks such as report generation and machine translation, covering character matching, semantic consistency, and distance metrics in high-dimensional semantic spaces, and can quantify the similarity between the generated text and the reference answer from multiple dimensions. Simulation Experiment 1 uses these metrics to comprehensively analyze the quality of the reasoning text.

[0133] See Figure 10 The light gray area represents the evaluation results of the present invention, while the dark gray area represents the evaluation results of the traditional method. It can be seen that the present invention has achieved stable superiority over the traditional method in almost all indicators, especially in the BERTS score, which measures semantic fidelity, and the BARTS score, which measures reasoning process and linguistic causal structure. The score difference is particularly significant, with an improvement of about 15% to 20% respectively. This fully demonstrates the technical advantages of the present invention in semantic control and expert logic reconstruction.

[0134] Simulation Experiment 2 For example, because metrics such as BERTScore and METEOR have a high tolerance for redundant information, the generated text may contain repetitive or irrelevant content without a significant decrease in score, thus failing to accurately reflect the actual quality of the text; therefore, these metrics are not effective in evaluating fine-grained recognition tasks. To minimize the presence of redundant and irrelevant information in the generated text, Experiment 2 uses the cutting-edge natural language processing model GPT-4o as the core evaluation tool. This model scores text quality based on three metrics: relevance, completeness, and redundancy, with each dimension ranging from 1 to 5 points.

[0135] Specifically, the relevance index is used to assess the degree of fit between the text and the recognition problem, the completeness index focuses on examining whether the reasoning text fully covers key elements and information, and the redundancy index measures whether the text contains redundant, repetitive, or valueless content.

[0136] In practice, GPT-4o can first be provided with guidance information that closely aligns with the research context, enabling it to conduct in-depth reviews of inference texts based on domain requirements. Subsequently, the model examines the logical structure, information content, and contextual consistency of the texts one by one according to pre-defined indicators, and provides refined scoring results.

[0137] See Figure 11As can be seen, this invention surpasses traditional methods in all three dimensions: relevance, completeness, and redundancy. The results show that traditional prompting engineering methods generally score low in completeness, averaging around 4.2, while the method of this invention averages 4.9, indicating that it more closely approximates the reasoning of real experts in terms of comprehensive information coverage. Regarding redundancy, traditional methods suffer from excessive repetition of structures or templated language. This invention, by introducing an expert ranking mechanism and semantically controlled prompt templates, effectively reduces invalid sentences, significantly improving the redundancy score. The relevance index also demonstrates that this invention can closely organize descriptive content around key image evidence, avoiding the "illusory statements" or "logical jumps" common in traditional CoT generation.

[0138] Simulation Experiment 3 In one example, to explore in depth the generation and evaluation mechanism of expert-level knowledge in fine-grained object recognition, Experiment 3 selected three experts with over 15 years of experience in aircraft recognition to conduct on-site evaluations of the texts shown in Table 1 below: texts written by the experts themselves, texts generated by traditional methods, and inference texts generated by this invention. These three experts possess rich practical experience and have accumulated profound knowledge in the field of aircraft recognition. The experts can evaluate the inference texts from two new perspectives: factual consistency and cognitive consistency.

[0139] Table 1

[0140] For example, factual consistency focuses on whether the diagnostic features in the reasoning text conform to the real-world fine-grained scenario, while cognitive consistency focuses on whether the reasoning text reflects the cognitive logic followed by experts in the actual judgment process.

[0141] Specifically, in the factual consistency assessment, researchers manually extracted feature information for each recognition task from the reasoning text and categorized them into valid and invalid features. Experts determined the validity of each feature using three key criteria: whether the feature name is accurate, whether the feature value is accurate, and whether the feature is helpful for interpretation. A feature was considered valid only if the answer to all three questions was "yes."

[0142] Specifically, in the cognitive consistency assessment, the complete recognized text output by the model is given to experts for scoring. The experts give a score of 1-100 based on their experience and judgment logic to measure the consistency and credibility of the model in cognitive logic.

[0143] Experimental results show that, in terms of information completeness, expert texts are rich in detail and comprehensively cover features, but due to differences in personal style, they lack a consistent organizational structure. Traditional reasoning text generation methods, such as cue engineering, can generate texts that meet professional standards to a certain extent, but their generalization of domain knowledge leads to the omission of some details. The text generated by this invention is more precise in its detailed expression, accurately preserving key features and ensuring a hierarchical presentation of information, making the recognition logic clearer. Regarding logical consistency, texts generated by cue engineering often suffer from disordered description order and a lack of logical connection between information. The reasoning text generated by this invention is structurally more consistent with the cognitive logic of human experts, with a description order from whole to part, successively covering dimensions, deck shape, key landmarks, equipment configuration, etc., allowing readers to more intuitively understand the target recognition process. In terms of feature accuracy, this invention has a more significant advantage. Cue engineering methods tend to introduce redundant or ambiguous expressions when generating feature descriptions, affecting the model's performance in practical tasks. The method of this invention integrates expert experience with structured knowledge extraction techniques, ensuring that all information in the reasoning text is task-relevant and accurately corresponds to the key features of expert interpretation. Notably, the structural similarity between the expert text and the reasoning text generated by this invention indicates that this patented method not only reproduces the expert's cognitive process but also optimizes the expert's description, making it more structured and universal.

[0144] Therefore, this invention generates diagnostic prompts (first-class prompts) for images to be identified by ranking, weighting, and semantic description of diagnostic features determined based on the subjective experience and objective data of domain experts. This allows the first-class prompts to internalize the dynamic decision-making preferences and reasoning logic of human experts when identifying such targets. By inputting the first-class prompts into a cognitive process reasoning system driven by a large language model, the system can simulate the expert's identification process under the constraints and guidance of the recognition criterion priority (i.e., feature ranking and weighting) framework obtained based on expert knowledge, and generate reasoning text according to the diagnostic logic of human experts. This efficiently and automatically generates a semantic representation of the fine-grained target visual recognition process.

[0145] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

Claims

1. A method for generating a fine-grained visual target recognition expert knowledge agent, characterized in that, include: Based on the feature ranking, feature weights, and semantic description information of the diagnostic features in the image to be identified, a first type of prompt word for the image to be identified is generated. The feature ranking and feature weights are determined based on subjective and objective data from experts when identifying similar targets. The domain to which the image to be identified belongs, the image to be identified, and the first type of prompt words are input into a cognitive process reasoning system based on a large language model to obtain reasoning text, wherein the reasoning text is used to simulate the visual cognitive process of an expert when identifying the image to be identified.

2. The method according to claim 1, characterized in that, The subjective data includes the saliency ranking of regions of interest and / or the pairwise importance scores of adjacent regions of interest; wherein, the region of interest is the region in the sample image where the diagnostic feature is located.

3. The method according to claim 2, characterized in that, The objective data includes the number of fixations to the region of interest, the duration of fixation to the region of interest, the time of the first fixation to the region of interest, and the fixation sequence of the region of interest.

4. The method according to claim 3, characterized in that, The feature weights and feature rankings are obtained by using the diagnostic features as candidate points in the improved VIKOR round-by-round extraction decision algorithm and the feature evaluation index as candidate indexes in the improved VIKOR round-by-round extraction decision algorithm. The feature evaluation indicators include: degree centrality, betweenness centrality, proximity centrality, and weight coefficients of diagnostic features; the degree centrality, betweenness centrality, and proximity centrality are determined based on the expert saccade path network, which is drawn based on the objective data, and the weight coefficients are determined based on the subjective data.

5. The method according to claim 1, characterized in that, The cognitive process reasoning system includes: A domain knowledge enhancement acquisition module is used to retrieve domain knowledge of the image to be identified based on the domain to which the image to be identified belongs; The feature selection module is used to extract diagnostic features with diagnostic value in the process of recognizing the image to be recognized based on the first type of prompt words, and obtain visually significant recognition features, wherein the visually significant recognition features include at least one of the diagnostic features. An attribute extraction module is used to retrieve the structural attribute information of the visually significant identifiable features. A multimodal expert knowledge-guided model reasoning module is used to integrate the domain knowledge, the visually salient features, and the structural attribute information based on the feature ranking in the first type of prompt words to obtain the reasoning text.

6. The method according to claim 5, characterized in that, The domain knowledge enhancement acquisition module is specifically used for: Similar texts are retrieved from the professional knowledge base of the domain to which the image to be identified belongs, based on sparse retrieval algorithm and dense retrieval algorithm, respectively, to obtain sparse similar texts and dense similar texts; The sparse and dense similar texts are fused based on the inverse ranking fusion algorithm to obtain the fused text and its fusion score. The top k fused texts with the highest fusion scores are selected as the domain knowledge, where k is a preset text selection threshold.

7. The method according to claim 5, characterized in that, The feature selection module is specifically used for: The multimodal large language model is invoked to parse the image to be identified, and the visibility, completeness and local saliency of each diagnostic feature in the image to be identified are determined to obtain candidate features; Determine whether each candidate feature exists in the image to be identified and perform multiple rounds of verification to exclude hallucination features; The excluded candidate features are output as the visually salient recognition features.

8. The method according to claim 5, characterized in that, The attribute extraction module is specifically used for: Based on the visually salient features, a query template is constructed to search online for the structural description information of the visually salient features, thereby obtaining the first type of attribute information; A multimodal large model is used to perform semantic querying and attribute recognition on the regions where visually significant identifiable features are located in the image to be identified, thereby obtaining the second type of attribute information; The structural attribute information is obtained by fusing the first type of attribute information and the second type of attribute information.

9. The method according to claim 5, characterized in that, The multimodal expert knowledge-guided model reasoning module is specifically used for: Based on the feature sorting and the structural attribute information, a second type of prompt words is generated, wherein the second type of prompt words consists of a structural description guidance module, an expert analogy reminder module, a reasoning task instruction module, and a final judgment prompt module; The second type of prompt words are input into the large language model for multiple rounds of content generation. The generated content is then subjected to duplicate detection and penalty, logical consistency verification, and structural integrity control to obtain the final generated text. The final generated text is converted into a structured format using an expert-level language style converter to obtain the inference text.

10. A fine-grained visual target recognition expert knowledge agent generation device, characterized in that, include: The first type of prompt word generation module is used to generate first type of prompt words for the image to be identified based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be identified. The feature ranking and feature weight are determined based on subjective and objective data of experts when identifying similar targets. A cognitive process reasoning system based on a large language model is used to generate reasoning text based on the domain of the image to be identified, the image to be identified, and the first type of prompt words. The reasoning text is used to simulate the visual cognitive process of an expert when identifying the image to be identified.

Images