A fine-grained visual target recognition expert knowledge intelligent agent generation method and device

Abstract

The application discloses a fine-grained visual target recognition expert knowledge intelligent agent generation method and device, and the method comprises the following steps: generating a first type of prompt word of a to-be-recognized image based on feature sorting, feature weight, and semantic description information of a diagnostic feature in the to-be-recognized image, wherein the feature sorting and the feature weight are determined according to subjective data and objective data of experts in identifying the same target; inputting a field to which the to-be-recognized image belongs, the to-be-recognized image and the first type of prompt word into a cognitive process reasoning system based on a large language model to obtain reasoning text, wherein the reasoning text is used for simulating a visual cognitive process of an expert in identifying the to-be-recognized image. The application can utilize existing part of expert field knowledge to drive a large model to efficiently and automatically generate reasoning text simulating a human expert identification fine-grained target process.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target identification, and particularly relates to a fine-grained target identification expert knowledge intelligent agent generation method and device. BACKGROUND

[0002] Traditional fine-grained target identification is mostly based on a general data set to train an identification model, and an image to be identified is input into the trained model, and the trained identification model outputs a target identification result. This method is essentially a data-driven statistical fitting process, lacks semantic representation of the identification process (such as attention areas and diagnosis logic), and the output result is only a black-box classification label, which is difficult to support high-reliability scenarios such as disease diagnosis and species identification, and the reliability of the identification result is required.

[0003] Therefore, some methods attempt to artificially construct domain knowledge, and domain experts manually write reasoning processes and key knowledge in the identification process in the form of text. The decision chain formed by the long-term practice of the domain expert has traceability for fine-grained discrimination, and the structured cognitive ability can provide reasonable and professional explanation texts. However, such explanation texts are mainly obtained through long-term experience accumulation and system training of human experts, and the experience of human experts often has characteristics such as high implicitness, large individual differences, and difficulty in quantification, which seriously restricts the reuse and migration of the experience in intelligent systems.

[0004] In recent years, knowledge generation technology based on large language models (LLMs) and large multimodal models (LMMs) has made significant progress in visual understanding, image-text explanation, and question-answering systems. Large language models can obtain an implicit knowledge distillation mechanism through pre-training of hundreds of billions of corpus, dynamically synthesize semantic descriptions of visual features, greatly reduce the cost of semantic knowledge construction, and unify the output format and generation logic. With the rapid development of LLMs, the recognition and understanding ability of visual information has been significantly improved.

[0005] However, current large language models still have great limitations when applied to vertical field discrimination tasks (such as species identification and ship identification) that lack sufficient training samples and corpus knowledge, especially in scenarios such as fine-grained target identification that require rich expert knowledge support. On the one hand, current LLMs perform well in image description generation, but their output often stays at the generalization of the surface visual features of the image to be identified, rather than capturing professional visual criteria used in the identification process to generate diagnostic descriptions of the identification process. On the other hand, existing methods usually rely on structured prompts or artificial templates, and are difficult to simulate the step-by-step reasoning and visual attention mechanism exhibited by human experts in the identification task, resulting in text content that is difficult to truly reflect expert-level cognitive logic.

[0006] Therefore, how to integrate the generation efficiency of LLM and the structured advantages of expert cognition, automatically generate expert knowledge that can describe the fine-grained visual target recognition process, has become a key technical problem that needs to be solved in the current visual expert knowledge generation field. SUMMARY

[0007] The embodiment of the application provides a fine-grained visual target recognition expert knowledge intelligent agent generation method and device, which can solve the above technical problems.

[0008] In a first aspect, the embodiment of the application provides a fine-grained visual target recognition expert knowledge intelligent agent generation method, which comprises the following steps:

[0009] generating first type prompt words of the to-be-recognized image based on feature ranking, feature weight, and semantic description information of diagnostic features in the to-be-recognized image, wherein the feature ranking and the feature weight are determined according to subjective data and objective data of experts in identifying the same type of target;

[0010] inputting the field to which the to-be-recognized image belongs, the to-be-recognized image and the first type prompt words into a cognitive process reasoning system based on a large language model to obtain reasoning text, wherein the reasoning text is used to simulate the visual cognitive process of the expert in identifying the to-be-recognized image.

[0011] In a second aspect, the embodiment of the application provides a fine-grained visual target recognition expert knowledge intelligent agent generation device, which comprises:

[0012] The first type prompt word generation module is configured to generate first type prompt words of the to-be-recognized image based on feature ranking, feature weight, and semantic description information of diagnostic features in the to-be-recognized image, wherein the feature ranking and the feature weight are determined according to subjective data and objective data of experts in identifying the same type of target;

[0013] The cognitive process reasoning system based on a large language model is configured to generate reasoning text according to the field to which the to-be-recognized image belongs, the to-be-recognized image and the first type prompt words, wherein the reasoning text is used to simulate the visual cognitive process of the expert in identifying the to-be-recognized image.

[0014] Compared with the prior art, the embodiment of the present application has the beneficial effects that: the present application generates the diagnostic prompt words (first type of prompt words) of the to-be-recognized image by the diagnostic feature ranking, weight and semantic description determined based on the subjective experience and objective data of the field experts, so that the first type of prompt words can internalize the dynamic decision preference and reasoning logic of the human experts when identifying such targets; by inputting the first type of prompt words into the cognitive process reasoning system based on the large language model, the system can simulate the identification process of the experts and generate reasoning texts according to the diagnostic logic of the human experts under the constraint and guidance of the identification criterion priority (i.e., feature ranking, weight) framework obtained based on the expert knowledge; and the semantic expression of the fine-grained visual target identification process is generated efficiently and automatically. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 A schematic diagram of an adjacency matrix is provided for the embodiment of the present application;

[0016] Figure 2 A schematic diagram of an expert saccade path network is provided for the embodiment of the present application;

[0017] Figure 3 A schematic diagram of an original decision matrix is provided for the embodiment of the present application;

[0018] Figure 4 An implementation flowchart of a feature ranking and feature weight determination method is provided for the embodiment of the present application;

[0019] Figure 5 A schematic diagram of a ranking matrix is provided for the embodiment of the present application;

[0020] Figure 6 A structural schematic diagram of a cognitive process reasoning system is provided for the embodiment of the present application;

[0021] Figure 7 An implementation flowchart of a fine-grained visual target identification expert knowledge intelligent agent generation method is provided for the embodiment of the present application;

[0022] Figure 8 A schematic diagram of an intermediate template of the first type of prompt words is provided for the embodiment of the present application;

[0023] Figure 9 A generation flowchart of the first type of prompt words in JSON format is provided for the embodiment of the present application;

[0024] Figure 10 An evaluation result schematic diagram of the reasoning text under the automatic index is provided for the embodiment of the present application;

[0025] Figure 11This is a schematic diagram illustrating the evaluation results of reasoning text under a model metric provided in an embodiment of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The application will be described in further detail below with reference to specific embodiments. The embodiments of the application are not limited to this.

[0033] Embodiment 1

[0034] In some embodiments, before the inference of the identification process of the image to be identified, subjective data and objective data of experts in identifying the same type of target can be collected first, and the ranking of the diagnostic features that play a key role in identifying this type of image and the weight of each feature are determined according to these data. Then the diagnostic features, feature ranking and feature weight of each type of target are associated and stored in the database for easy calling in the subsequent identification inference process.

[0035] For example, taking a remote sensing image as an example, if the target in the image to be identified is a ship target, and the purpose of identification is to identify the model or type of the target, then the diagnostic features of this type of target can include: ship shape, ship size, observation room, bow shape, stern shape, chimney position and number, antenna array, etc. If the target is an airplane, and the purpose of identification is to determine the model or type of the airplane, then the diagnostic features of this type of target can include: airplane size, wing position, wing angle, wing plane shape, wing span size, wing tip winglet, tail shape, engine position, engine number, engine shape, etc.

[0036] It should be understood that the target identification objects listed here are only examples of targets, and the application is not limited to the specific types of targets in the image to be identified.

[0037] In a possible implementation, a plurality of features for target identification can be found from technical literature related to the target, such as literature that summarizes the features of various types of ship bodies, airplanes, cars, and even biological targets such as fish, and then the external morphology that is easy to observe is found from these features, and the features with semantic redundancy and spatial overlap are excluded to obtain the diagnostic features of this type of target.

[0038] In a possible implementation, experts in the field of the target can be invited to perform objective identification tasks and subjective identification tasks in turn to obtain objective data and subjective data.

[0039] For example, the subjective data can include the saliency ranking of the region of interest and / or the pairwise importance score of the adjacent region of interest.

[0040] Specifically, the region of interest is the region where the diagnostic feature is located in the sample image. For example, the region of interest can be the region where the deck is located in the sample image.

[0041] For example, the objective data can include the number of fixations on the region of interest, the duration of fixations on the region of interest, the first fixation time on the region of interest, and the fixation sequence on the region of interest.

[0042] In one example, in the objective recognition task, the expert can be equipped with an eye tracker and provided with a display, a standard keyboard. The expert sits in front of the display, observes the sample image, and after identifying the target in the sample image, presses the button and orally states the category / model of the target, etc. At the same time when the expert identifies, the software automatically records the accuracy and reaction time of the expert, and the eye tracker records the number of gazes, gaze duration, first gaze time, gaze sequence of the expert when identifying the target, to obtain objective data.

[0043] Specifically, the objective recognition task requires the expert to sit in front of the eye tracker with the head fixed and the eyes kept at a distance of about 100 cm from the display. Before the task starts, 9-point calibration will be performed to ensure the accuracy of the eye tracking system. Before the formal task starts, a short practice will be carried out first to help the expert familiarize with the task process and perform preliminary calibration of the equipment to ensure the accuracy of the data. The image display time is 20 seconds, and if the expert does not press the required key, the image will automatically skip. In each task, the images are presented in random order, and the eye movement data of the expert, including gaze points, gaze times and gaze times of each interest area, are automatically recorded by the software. Then the collected eye movement data is processed to obtain the original objective data, including the number of gazes, the duration, the first gaze time and the gaze sequence, etc. After noise filtering of the original objective data (for example, saccades <40 ms or >1000 ms are excluded), the objective data is extracted according to the interest area.

[0044] In one example, the subjective recognition task presents the sample images to the expert in a rearranged order. The expert recalls and freely reproduces the recognition process of each image and fills in the corresponding rating scales. The scoring indicators in these scales include the extraction and judgment basis of diagnostic features in the recognition process, thereby further revealing the recognition strategy and judgment standard of the expert. After each recall, the expert is required to significantly sort the diagnostic features used in the recognition process, to obtain the significance ranking of the diagnostic features; then score the pair-wise importance of adjacent features in the ranking, to obtain the pair-wise importance score of adjacent interest areas.

[0045] For example, for each type of aircraft, a set of predefined candidate attributes (e.g., aircraft size, wing position, wing angle, wing planform shape, wing span size, winglet, tail shape, engine position, engine number, engine shape, etc.) will be presented on the display. Participants can refer to the actual recognition process and rank these features in descending order of perceived importance. Then, the pair-wise importance between adjacent features is rated using a standardized interval scale, thereby quantifying the relative contribution of each feature to its decision.

[0046] Through this subjective data collection method, not only the explicit content of expert decision-making can be captured, but also the internal priority mechanism that dominates the real-world diagnosis behavior can be captured.

[0047] In one possible implementation, the expert saccade path network can be first drawn according to the objective data, and then the feature evaluation index of the diagnosis feature is calculated based on the expert saccade path network and the subjective data, the index vector of each region of interest is constructed according to the feature evaluation index of each diagnosis feature, and the original decision matrix is obtained by splicing these vectors. Finally, based on the improved VIKOR round-by-round extraction decision algorithm (for specific process, please refer to the patent with publication number CN117422322A), the feature ranking and feature weight of the diagnosis feature are determined according to the original decision matrix.

[0048] In one example, before drawing the expert saccade path network, the adjacency matrix of each expert can be first constructed according to the objective data, and then the saccade matrix is obtained by splicing the adjacency matrices of all experts, and the expert saccade path network is drawn according to the saccade matrix.

[0049] For example, referring to Figure 1 , the row and column of the adjacency matrix are used to represent the region of interest of the diagnosis feature, and the value in the matrix is used to represent the number of saccades between two regions of interest. For example, referring to Figure 1 , the value of P2 row and P3 column is 5, which means that there are 5 saccade paths from P2 to P3.

[0050] In one example, nodes corresponding to the regions of interest can be created according to the saccade matrix, and then edges between the nodes are added according to each fixation sequence of the region of interest in the objective data, the weight of the edge reflects the number of saccades between the two regions of interest connected by the edge, and the direction represents the saccade order; after adding the saccade path of each expert, the expert saccade path network is obtained (see Figure 2 ).

[0051] For example, in the expert saccade path network, the direction of saccade can be reflected in the edge of the graph, and the forward saccade and the backward saccade can be distinguished by the direction of the edge edge (such as the starting edge and the ending edge).

[0052] In one example, the feature evaluation index can include: degree centrality, betweenness centrality, closeness centrality and weight coefficient of the diagnosis feature.

[0053] For example, the degree centrality is the value (i.e. degree) on the node in the expert saccade path network, and the larger the value, the more times the region of interest is saccaded. Through , the importance of the diagnosis feature can be reflected.

[0054] Exemplarily, betweenness centrality The betweenness centrality can be measured by the number of shortest paths passing through the node, which reflects the key position of the diagnostic feature in the expert scan path network.

[0055] Exemplarily, closeness centrality The closeness centrality is the average shortest path between the node and other nodes in the expert scan path network, which reflects the connection density between the diagnostic feature corresponding to the node and other diagnostic features, and reflects the centrality of the node in the network.

[0056] Exemplarily, the importance weight vector of the diagnostic feature in different identification tasks participated by the same expert can be calculated according to the subjective data, and the weight vector of each expert is averaged to obtain the weight coefficient of the diagnostic feature.

[0057] In one example, referring to Figure 3 In the original decision matrix, the row represents the candidate point, i.e., the region of interest, and the column represents the candidate index, i.e., the feature evaluation index.

[0058] Exemplarily, the multi-criteria compromise solution ranking method (Vise Kriterijumska Optimizacija-Kompromisno Resenje, VIKOR) can be based on The VIKOR method can be used to quickly obtain a compromise solution for the decision maker under complex criteria conditions, and can provide effective support for the selection of candidate options. However, the final compromise ranking result of the VIKOR method is affected by the decision coefficient v. The coefficient is used to balance the "maximization of group utility" and the "minimization of negative impact", and its value can be continuously changed in the interval [0, 1]. In practical applications, it is difficult to clearly define the optimal expected value of v. In most cases, v is usually set to 0.5 to take into account both aspects, but this cannot guarantee that the ranking result under other values is always worse than the default value, nor can it ensure that this ranking result is the optimal solution in the actual use scenario. To overcome this limitation, the improved VIKOR round-by-round decision algorithm is used, which can select the optimal decision coefficient v in the interval [0, 1], and then obtain the optimal ranking result and weight based on the selected decision coefficient.

[0059] Example 2

[0060] Based on example 1, Figure 4An implementation flowchart of a feature ranking and feature weight determination method provided by an embodiment of the application is shown. By way of example but not limitation, the method can include steps S401-S404, which are described as follows:

[0061] S401, calculating a group utility and an individual regret value of each diagnostic feature according to an original decision matrix.

[0062] In one example, the original decision matrix can be normalized first, and then the group utility and the individual regret value of each diagnostic feature are calculated based on the normalized original decision matrix.

[0063] By way of example, the group utility of the diagnostic feature can satisfy the following formula:

[0064]

[0065] wherein, is the group utility of the i-th diagnostic feature, is the weight of the j-th feature evaluation index, , is the maximum value and the minimum value of the j-th column in the normalized original decision matrix (i.e., the maximum value and the minimum value of the j-th feature evaluation index), is the value of the i-th row and the j-th column in the normalized original decision matrix (i.e., the value of the j-th feature evaluation index of the i-th diagnostic feature), and n is the total number of feature evaluation indexes, which can be 4 here.

[0066] By way of example, the individual regret value of the diagnostic feature can satisfy the following formula:

[0067]

[0068] wherein, is the individual regret value of the i-th diagnostic feature.

[0069] S402, calculating a compromise measure of each diagnostic feature at different v values according to the group utility and the individual regret value, to obtain a ranking of the diagnostic features at different v values.

[0070] In one example, the compromise measure of each diagnostic feature at different v values can be calculated first, and then the diagnostic features are ranked from large to small according to the compromise measure, to obtain the ranking of the diagnostic features at different v values.

[0071] By way of example, the compromise measure can satisfy the following formula:

[0072]

[0073] wherein, is the compromise measure of the i-th diagnostic feature, , are the maximum and minimum of the group utility between each pair of diagnostic features, respectively, , are the maximum and minimum of the individual regret value between each pair of diagnostic features, respectively.

[0074] Optionally, after the intermediate variable , , the compromise measure can also be calculated by the formula .

[0075] S403, according to the compromise measure of each diagnostic feature at different v values, the decision value interval is divided, and multiple critical orders of the diagnostic feature are obtained.

[0076] In a possible implementation, if the compromise measures of any two diagnostic features are equal at the same decision coefficient , the order of the two diagnostic features before and after the change is changed, then the critical decision coefficient can be recorded as . If there are critical decision coefficients in the interval [0, 1], then the interval [0, 1] can be divided into +1 sub-intervals (i.e. decision value intervals) according to the critical decision coefficients. Then the value of the diagnostic feature at the interval endpoint, 0 or 1, can be taken as the v value, and +1 critical orders of the diagnostic feature can be calculated.

[0077] In an example, the critical decision coefficient can satisfy the following formula:

[0078]

[0079] Wherein:

[0080]

[0081]

[0082] .

[0083] S404, according to the critical order of the diagnostic feature, the feature order and the feature weight are determined.

[0084] In an example, the order matrix P can be constructed according to the critical order of each diagnostic feature and the length of the critical decision interval corresponding to the critical order, and then the order matrix P is extracted round by round to obtain the final feature order, and the total score of the diagnostic feature during the round-by-round extraction is taken as the feature weight of the diagnostic feature.

[0085] For example, see Figure 5 ​In the ranking matrix P, the cth column of the matrix corresponds to the interval length of the cth critical decision interval, the ith row of the matrix corresponds to the ith feature in the critical ranking, and the element of the matrix represents the name or index of the ith feature under the cth critical ranking.

[0086] The application determines feature ranking and feature weight based on improved VIKOR round-by-round extraction decision algorithm, can comprehensively consider the influence of different v values in the interval [0, 1], and can also screen out the ranking result that can best consider the overall utility and individual regret by using comparative analysis. Compared with the traditional fixed v value (such as v = 0.5) method, the method can be flexibly adjusted according to the specific decision demand in practical application, and the objectivity and robustness of the ranking result are enhanced.

[0087] Embodiment 3

[0088] Figure 6 A structural schematic diagram of a cognitive process reasoning system provided by an embodiment of the application is shown. As an example but not limitation, the system can include a domain knowledge enhanced acquisition module 610, a feature selection module 620, an attribute extraction module 630, and a multi-modal expert knowledge guided model reasoning module 640.

[0089] In some embodiments, the domain knowledge enhanced acquisition module 610 can retrieve the domain knowledge of the to-be-recognized image according to the domain to which the to-be-recognized image belongs. The feature selection module 620 can extract diagnostic features having diagnostic value in recognizing the to-be-recognized image according to the first type of prompt word, to obtain visual salient recognition features. The attribute extraction module 630 can retrieve structural attribute information of the visual salient recognition features. The model reasoning module 640 can integrate the domain knowledge, the visual salient recognition features, and the structural attribute information based on feature ranking to obtain reasoning text.

[0090] In a possible implementation, the domain knowledge enhanced acquisition module 610 can specifically generate retrieval-augmented generation (RAG) mechanism, retrieve similar texts from a professional knowledge base related to the domain to which the to-be-recognized image belongs through sparse retrieval algorithm and dense retrieval algorithm respectively, to obtain sparse similar texts and dense similar texts; then fuse the two kinds of similar texts based on reciprocal rank fusion (RRF) to obtain fused texts and fusion scores of each text; finally, select the top k texts with higher scores as the domain knowledge of the to-be-recognized image.

[0091] For example, the domain knowledge output by the module can be expressed in structured JSON form, containing information sources, knowledge text, relevance scores, and category labels. For example, the domain knowledge can include: "This civil aircraft adopts a low-wing layout, with two turbofan engine nacelles symmetrically suspended under the wing. This configuration helps improve fuel efficiency and reduce noise. The rear section of the fuselage is equipped with a large T-shaped tail, which is common in modern civil aviation passenger aircraft and helps improve flight stability and handling performance. The main landing gear adopts a double-axis six-wheel structure to ensure the stability and runway adaptability of the aircraft during landing. These features combine to present the configuration characteristics of a typical civil aviation trunk passenger aircraft."

[0092] In one example, the professional knowledge base can be constructed by the user in advance according to heterogeneous data sources. For example, the professional knowledge base for the aircraft class can include aircraft identification manuals, newly published academic papers related to aircraft, structural specification documents, web encyclopedia entries, and aircraft-related web page data.

[0093] For example, data in original formats such as PDF, HTML, and TXT can be uniformly processed by text recognition and segmentation tools at the sentence level, with semantic paragraphs as the minimum indexing unit and stored as UTF-8 encoded text.

[0094] In one example, the jieba word segmentation tool can be used to perform word segmentation on the domain description of the image to be recognized, and an inverted index can be constructed using the BM25 algorithm for sparse retrieval operations.

[0095] For example, the BM25 parameters can be k1 = 1.5 and b = 0.75, supporting keyword position weighting and title field priority.

[0096] In one example, during dense retrieval, the Sentence-BERT model can be used to perform semantic embedding of the knowledge base text segment and the domain description of the image to be recognized, and all embedding vectors are stored in the Milvus vector database. Then, cosine similarity is used as the distance function, and the top 50 similar text records (i.e., dense similar texts) are returned by default.

[0097] In one example, the fusion score of the fused text can be calculated by the following formula:

[0098]

[0099] wherein, is the fusion score of the dth fused text, A is a set of retrieval algorithms (including sparse and vector types), represents the ranking position of the text in retrieval algorithm a, is a smoothing constant, with a value range of 60-100.

[0100] The RRF fusion algorithm can effectively balance the coverage ability and sorting depth of different types of retrieval results, and realize unified optimization of multi-source sorting information.

[0101] The domain knowledge enhancement acquisition module 610 provides structured domain knowledge text to the large language model in the identification task by introducing the RAG mechanism, to enhance the professionalism, relevance and accuracy of the model-generated content.

[0102] In one possible implementation, the feature selection module 620 can specifically call a multi-modal large language model to analyze the to-be-identified image, judge the visibility, integrity and local saliency of each diagnostic feature in the to-be-identified image, and obtain candidate features. Then, it is judged whether each candidate feature exists in the to-be-identified image and multiple rounds of verification are performed to exclude illusory features. The candidate features after exclusion are output as the visually salient identification features.

[0103] For example, the feature selection module 620 can output the visually salient identification features in the form of a structured feature list, with each feature labeled with its visibility score in the to-be-identified image, diagnostic importance level and expert confidence weight, providing accurate input for subsequent attribute extraction and explanation generation.

[0104] In one example, a multi-modal large language model (MLLM) Gemini-2.5-pro can be called to analyze the to-be-identified image in depth, and through multi-modal reasoning, judge the visibility, integrity and local saliency of each feature in the image to obtain candidate features.

[0105] In one example, for each candidate feature, the MLLM not only judges whether it exists, but also evaluates its discriminative contribution in a specific task scenario according to the priority of expert experience, and performs multiple rounds of feature verification combined with structured prompts to exclude false positives and "illusory" features caused by image occlusion, blur or noise, etc., to obtain visually salient identification features.

[0106] The feature selection module uses a structured and objective feature screening method to screen visually salient identification features, which can ensure that the model does not produce illusory features in the feature selection link, and avoid misjudgment problems caused by insufficient visual evidence. At the same time, the result of feature selection directly affects the accuracy and effectiveness of subsequent attribute extraction, ensuring the robustness and professionalism of the overall process.

[0107] In one possible implementation, the attribute extraction module 630 can specifically first call a structured semantic extension path based on a network search engine, construct a query template, and query online the structural description information of the visual salient feature to obtain first-class attribute information; then, a multi-modal large model (such as Qwen2.5-VL or BLIP-2) is used to perform semantic inquiry and attribute identification on the structural region in the image to obtain second-class attribute information; finally, accurate and consistent structural attribute expressions are formed by screening and calibrating the semantically similar multi-source information (i.e., the first-class attribute information and the second-class attribute information), and the structural attribute information is obtained.

[0108] For example, the structural attribute information can include four core elements: (1) structure (i.e., visual salient feature) name, such as “aircraft size”; (2) attribute description, including location, size, length, width, and other fine-grained visual knowledge of the feature; (3) source information, identifying the website or multi-modal large model name of the information source; and (4) confidence score, indicating the consistency strength of the fused content.

[0109] In one example, the attribute extraction module 630 can initiate a keyword-level request to a search engine interface (such as Google Search or Bing API) based on the query template, and automatically collect the structural description information in the web abstract and the encyclopedia entry. Then, after performing syntax dependency analysis and keyword extraction on the returned text paragraphs, a BERTScore and TF-IDF joint scoring mechanism is used to filter redundant content, and only text segments with high semantic relevance to the input structure item and clear description are retained, and their original source and extraction timestamp are automatically labeled to enhance verifiability, and the final first-class attribute information is obtained.

[0110] In one example, the attribute extraction module 630 can crop the region of interest into a subgraph, generate semantic prompt words related to the structure, such as “What are the appearance features of this wing?” and “What civilian aircraft are this tail design commonly used for?” and the like; input into a multi-modal large model, and the model returns a natural language answer combining the image and the prompt, which includes, for example, “The lower single-wing layout has a wing leading edge with a 25-degree sweep angle and is equipped with winglet to optimize fuel efficiency” and “The vertical tail has a T-shaped design, and its smooth surface helps to improve aerodynamic performance, and the vertical distance between the upper edge of the horizontal tail and the bottom of the fuselage is 2.5 meters”, as second-class attribute information.

[0111] In one example, the attribute extraction module 630 can perform semantic vectorization processing on the two types of attribute information related to the same visual salient feature, and then calculate the cosine similarity of the two types of semantic information based on the semantic vectors to quantify the semantic proximity between different texts. Then, the texts with high similarity are merged, and different labels are added to the texts with large differences, and the structural attribute information is obtained.

[0112] Exemplarily, the attribute extraction module 630 can use a multilingual sentence vector model Sentence-BERT to embed the text into the same semantic space to obtain semantic vectors of the attribute information.

[0113] Exemplarily, attribute information with a similarity higher than a merging threshold (for example, 0.85) can be merged into one semantic expression, and the version with richer semantic content and more complete syntax structure is retained.

[0114] Exemplarily, for attribute information with too large difference, two types of attribute information can be retained respectively, and the first type of attribute information from expert text or network retrieval content is added with an "empirical attribute" label; the second type of attribute information from image model inference content is added with a "structural attribute" label.

[0115] Optionally, in order to further improve the controllability and accuracy of the structural attribute information, after obtaining the structural attribute information, the attribute extraction module 630 can introduce a secondary review mechanism after fusing the two types of attribute information. Prompt words are constructed based on the structural attribute information, such as "Does the description exist speculation or information inconsistent with the aircraft type?" and "Does the structure have diagnostic significance in the actual aircraft classification?" and the like; a large model is input, the large model scores and analyzes the professionalism, factuality and redundancy of the structural attribute information, and outputs a credibility score in the form of 5 points. When the score is higher than a set threshold (such as 4 points and above), the structural attribute information is included in the available attribute set; if the score is insufficient, the original information source is returned to re-construct the prompt word for correction.

[0116] Through the above attribute extraction method based on multi-modal, expert text and large model cross verification, the application can effectively improve the quality control ability of the structural level semantic content, so that the generated recognition text has stronger explainability, consistency and logical traceability, which is significantly different from the traditional prompt generation method based on simple keywords or template filling.

[0117] In one possible implementation, the model inference module 640 can generate the second type of prompt word according to the feature ranking and the structural attribute information; then, input the second type of prompt word into the large language model by taking Beam Search (the width can be 3 to 5) as the decoding strategy, perform multiple rounds of content generation, and perform repeated detection and punishment on the generated content, logical consistency verification, and structural integrity control to obtain the final generated text; finally, the final generated text is converted into a standard structured format based on an expert language style converter to obtain the inference text.

[0118] Exemplarily, the large language model here can adopt a model (such as GPT-4, Qwen2.5-72B) that supports multi-round reasoning capability, large-scale context processing capability, and prompt structure driven generation process.

[0119] In one example, the model inference module 640 can construct a dynamically controlled multi-level prompt template based on a pre-defined "expert prompt library". The prompt content of the second type of prompt words no longer adopts static instructions or general descriptions, but combines feature ranking results and structural attribute information to generate language prompts with causal logic according to the idea of "starting from the most discriminative structure and guiding to global type inference layer by layer".

[0120] Exemplarily, the second type of prompt words can be composed of a structural description guiding module, an expert analogy reminding module, an inference task instruction module, and a final judgment prompt module.

[0121] For example, the second type of prompt words can be: Its nose section is designed with a unique streamlined fairing, which is commonly seen in modern jet passenger aircrafts that pursue high-speed performance and fuel efficiency. Combined with this feature, further observe whether the wing-body junction adopts a smooth transition blended wing-body design, which not only reduces air resistance, but also improves flight stability and stealth performance. Next, based on the identification marks on the vertical tail, including but not limited to special painting or identifiers, as well as the specific layout of the engine nacelle and the shape of the air intake, conduct a comprehensive analysis. According to all visible significant identification features, such as winglet design, unique tail arrangement, and engine configuration, infer the possible category of the aircraft and its reasoning basis. Such a description avoids directly mentioning the structures and analogies related to the aircraft, and focuses on the identification features unique to civil aircrafts.

[0122] In one example, in order to ensure the stability, logical consistency and avoid repetition and redundancy of the generated language, the model inference module 640 can perform repetition detection and penalty operations, set a repetition penalty coefficient for phrases and structures that repeatedly appear in the generated content to prevent language redundancy; perform logical consistency verification operations, perform BERTScore semantic matching analysis on each piece of text and structural attribute information, if the deviation is too large, it is determined as "hallucination content" and the prompt is rolled back and reconstructed; perform structure integrity control operations to ensure that the generated content output contains feature-attribute corresponding content no less than 3 items, and at least covers 1 identification inference logic and 1 expert basis explanation.

[0123] In one example, the expert language style converter can convert the generated content of the large language model into an expression form close to the writing style of human experts, and output the converted text in a structured format.

[0124] For example, the transformed text can use analogies, judgments, and inferential words (such as "usually indicates", "more likely to belong to", "with significant differences") to express, avoiding the use of ambiguous semantics or uncertain structures.

[0125] For example, the first part of the structured format is a description of the visual significant identification features, the second part is the reasoning chain, and the third part is the conclusion judgment.

[0126] Optionally, the number of words in the final output reasoning text can be controlled within the range of 200-300 words, ensuring a balance between information density and readability.

[0127] For example, the aircraft exhibits the following features: the lower wing has a slight upward angle, the swept-back design and the wingtip winglets indicate an optimization pursuit for fuel efficiency, the wingspan is about 38 meters, the fuselage length is about 47.3 meters, and it has the appearance of a typical commercial aircraft with regularly arranged porthole layout; based on these structural attribute information, it can be further described: the tail system includes a single large vertical tail and a low-mounted all-moving horizontal tail, providing excellent directional stability and control performance. Two turbofan engines are precisely placed under the wings, and the prominent air intakes further confirm this configuration. Combined with these identification features, especially the wingtip winglets, unique tail arrangement, and specific location and number of engines, it can be determined that this model is a member of the Boeing B757 series, specifically conforming to the technical specifications and appearance characteristics of the B757-200 model of 2024.

[0128] The model reasoning module realizes the output of high-controllable and high-explainable identification text through the second type of prompt words based on the expert prompt chain, structural reasoning guidance, multi-level generation quality control mechanism, and language style optimization strategy, which can make up for the lack of logical constraints, semantic redundancy, and illusion content in traditional prompt methods, significantly enhancing the practicality and credibility of the model in professional image language generation scenarios.

[0129] The system provided by the present application can significantly enhance the generation capability of the model in the absence of target corpus or non-English scenarios by introducing expert knowledge guided semantic prompting mechanism and structured visual cognitive information embedding. The system does not rely on training samples of target categories, but relies on general visual diagnostic logic and significant feature prompts extracted by expert cognition. Through semantic template, the large model generates text according to the logic chain of "image evidence → feature induction → diagnostic expression", so that even in the face of new propagation types that the model has never seen, high-quality text with diagnostic logic and domain language can still be constructed. Unlike general image-text description generation in general scenarios, the present application focuses on professional tasks with visual recognition criteria requirements (such as propagation recognition), and the generated content not only needs to conform to language specifications, but also needs to reflect key expert knowledge such as "classification criteria", "structural differences", "detail features", etc. By embedding these contents into the prompt structure and strongly binding them with the image evidence, reasoning text with obvious "traceability" and "decision support" can be generated.

[0130] Further, the present application introduces an expert-driven visual evidence modeling mechanism to realize a one-to-one correspondence between diagnostic description and key image regions during text generation, thereby fundamentally alleviating the "hallucination generation" problem commonly encountered by large language models. Traditional large models often rely on language pattern driving in image-text generation tasks, lacking effective constraints on image content, resulting in the generation of descriptive sentences that do not conform to the facts of the image. Especially when facing fine-grained and multi-category visual tasks, it is easy to misjudge details, fabricate features, and have logical conflicts. However, the present application uses saliency feature extraction and expert decision path ordering mechanism to explicitly identify structural elements with classification diagnostic value in the image before generation, and embed them into the generation context through prompt structure, so that the language output of the large model is controlled by the image evidence chain. This not only ensures that each diagnostic sentence can be traced back to a specific image region, but also guides the language generation order with the help of expert reasoning path, thereby improving the causality of reasoning and the accuracy of description. At the same time, this mechanism enhances the visual alignment capability of the language model, making the generated results have higher interpretability, consistency, and professional credibility, and has obvious technical advantages in high safety requirements or task irreversible scenarios (such as medical diagnosis).

[0131] In addition, the present application adopts a modular architecture design, divides the visual expert knowledge generation process into a plurality of functional sub-modules which can be independently run and combined. This design greatly enhances the controllability and maintainability of the system, so that each module can be optimized or replaced according to the task scene, and has good flexible configuration capability. For example, in different visual recognition tasks, the system can replace the criterion set for saliency ranking or update the reasoning path model according to the expert source without reconstructing the entire system logic framework. At the same time, this modular architecture is easy to extend to other visual interpretation tasks, such as remote sensing image interpretation, industrial defect detection, lesion identification, etc. Only the input image and the corresponding domain knowledge base need to be replaced, and rapid migration and adaptation can be completed. In addition, the modules communicate with each other through standardized semantic interfaces, avoiding error propagation caused by data coupling and improving the overall robustness of the system. More importantly, this design supports precision evaluation and incremental iteration of a single module, facilitating on-demand enhancement of specific capabilities in actual deployment, such as improving the fineness of the diagnosis chain logic or strengthening the consistency of language generation, greatly improving the engineering landing nature of the system and the practical expansion potential in multi-task scenarios.

[0132] Embodiment 4

[0133] The fine-grained visual target recognition expert knowledge agent generation method provided by the embodiments of the present application can be applied to electronic devices such as mobile terminals, personal notebook computers, supercomputers, etc. The embodiments of the present application do not make any limitation on the specific type of electronic device.

[0134] Figure 7 The implementation flowchart of the fine-grained visual target recognition expert knowledge agent generation method provided by the embodiments of the present application is shown. As an example but not limitation, the method can include steps S701-S702, which will be described below.

[0135] S701, generating a first type of prompt word of the to-be-recognized image based on the feature ranking of the diagnosis feature, the feature weight, and the semantic description information of the diagnosis feature in the to-be-recognized image.

[0136] In some embodiments, the first type of prompt word can be composed of several levels of sub-templates, which can specifically include a top-level template, an intermediate template, and a bottom-level template. The placeholders in the templates can be filled with specific feature names, weights, and semantic description information texts. Through this hierarchical structure which can be flexibly replaced and adjusted, modular design and reuse of the first type of prompt word can be realized.

[0137] For example, the content of the top-level template can be used to determine the overall framework and tone of the prompt, the intermediate template is responsible for the layout and connection of the feature group (see Figure 8 ), and the bottom-level template generates semantic description fragments for a single feature.

[0138] Specifically, the top-level template can include a task description, a feature list, and a prompt guide, etc. The feature list can be an array containing multiple feature objects, each of which is nested with name, weight, description, etc.

[0139] Specifically, referring to Figure 8 The middle-level template can be filled in the order of features. First, the feature with the highest weight is described, and a more detailed semantic segment description is added; then the feature with a lower weight is described, and a relatively concise description is added.

[0140] In a possible implementation, the semantic description information of each feature in the prompt word can be generated based on a semantic segment enhancement mechanism.

[0141] For example, for each feature, the semantic description information can not only include the name, but also include a diagnostic meaning-rich description segment generated in combination with domain knowledge.

[0142] For example, for the diagnostic feature with the highest weight "aircraft size", the semantic description information can emphasize the key information that the wing directly affects the flight performance, and for the secondary feature "wing position", the semantic description information can briefly point out the auxiliary role of the existence of the feature in identification.

[0143] The semantic segment enhancement mechanism can ensure that the prompt word not only includes a feature list, but also includes a set of natural language segments of expert interpretation experience.

[0144] In an example, the templates of different features can be classified, such as structural appearance, electronic equipment, etc., and the classification information of the features can be added in the semantic description information based on the in-group prompt weight control and prompt interpretability control mechanisms.

[0145] For example, the in-group prompt weight control mechanism refers to adjusting the length and emphasis of the semantic description information of the same type of features according to their feature weights.

[0146] For example, in the equipment feature classification, if the feature weight of the engine position is higher than that of the wingspan size, the emphasis "mainly equipped" can be added to the semantic description information of the engine, and the length of the semantic description information is lengthened; the emphasis "secondarily equipped" is added to the semantic description information of the wingspan, and the length of the semantic description information is shortened, so as to highlight the importance of the former relative to the latter through length control.

[0147] For example, the prompt interpretability control mechanism is embodied in the display or implicit identification of the feature order in the semantic description information, so that the large model and the user can understand the logic of the prompt.

[0148] For example, use sequential conjunctions in the semantic information description of different features; "first", "second", "in addition", "last", to implicitly identify the feature order; or use explicit feature order instructions: "the order of importance is as follows" to guide the model to reason along the expert cognitive order. See Figure 8 If the first type of prompt word is output in JSON format, the primary and secondary relationships of the features can also be represented by field names or nested levels, ensuring that the first type of prompt word is clear and interpretable.

[0149] In a possible implementation, the semantic description information of the diagnostic features can be given by an expert or generated by a recognition model. The obtained feature order, feature weight, and semantic description information can be saved in a database. When obtaining the prompt word, a prompt word generation program is called to obtain the relevant information from the database to fill in the template to obtain the first type of prompt word in JSON format. Figure 9

[0150] The structured first type of prompt word enables the model to obtain each feature and its weight and meaning in the form of parsing JSON, and then strictly follows the provided structure and order for reasoning. By explicitly including weight order information and analysis instructions in JSON, the model can make full use of the emphasis relationship implied in the expert knowledge and simulate the judgment process of the expert.

[0151] S702, input the domain to which the to-be-recognized image belongs, the to-be-recognized image, and the first type of prompt word into a cognitive process reasoning system based on a large language model to obtain a reasoning text.

[0152] Illustratively, the cognitive process reasoning system can be the system in Embodiment 3 described above, which can be used to simulate the visual cognitive process of an expert when identifying the to-be-recognized image.

[0153] The present application generates a diagnostic prompt word (first type of prompt word) for the to-be-recognized image by determining the diagnostic feature order, weight, and semantic description based on the subjective experience and objective data of the domain expert, so that the first type of prompt word can internalize the dynamic decision preference and reasoning logic of the human expert when identifying such targets; by inputting the first type of prompt word into a cognitive process reasoning system driven by a large language model, the system can simulate the identification process of the expert under the constraint and guidance of the identification criterion priority (i.e., feature order, weight) framework obtained based on expert knowledge, and generate a reasoning text according to the diagnostic logic of the human expert; efficiently and automatically generate a semantic representation of the fine-grained target visual recognition process.

[0154] ​Further, the fine-grained target identification process generated by the reasoning text explanation of the application can significantly improve the transparency and explainability of the identification process, allowing users to clearly understand the key features and their weights on which the model decision is based and the reasoning logic. The reasoning result is based on feature explanations consistent with the field consensus, which can significantly improve the credibility of the identification conclusion and provide users with easy-to-understand, verifiable and interactive decision-making basis.

[0155] Furthermore, the application proposes a "human-computer integration" expert knowledge generation paradigm. By systematically collecting and modeling the cognitive processes of visual experts in fine-grained identification, including visual attention trajectories (such as fixation areas and eye movement sequences) and reasoning logic structures (such as diagnostic judgment sequences and feature priority rankings), the application converts implicit expert experience knowledge into structured and inputable prompt information, forming a semantic prompt template. Unlike traditional prompt engineering, which uses fixed sentence patterns or general logic prompts, the application is based on expert cognitive modeling and constructs a prompt structure with greater depth of domain knowledge and rationality of criteria. Essentially, the application introduces an "expert semantic intermediate layer", enabling large models to not only mimic surface text structures in language generation, but also to simulate how experts recognize, judge and express on the semantic level. This human-computer integration mechanism not only improves the semantic accuracy and logical coherence of large models in complex visual tasks, but also significantly enhances the diagnostic, professional and explanatory capabilities of generated text, providing a basic framework for reliable generation of large models in high-cognitive tasks. In addition, this paradigm has strong universality and can be applied to medical, security, industrial inspection and other application scenarios with high visual explanation requirements, broadening the migration path and form of expression of expert knowledge in intelligent systems.

[0156] Embodiment 5

[0157] The embodiment of the application also provides a fine-grained visual target identification expert knowledge intelligent agent generation device, which can include a first type of prompt word generation module and a cognitive process reasoning system based on a large language model.

[0158] For example, the first type of prompt word generation module is used to generate the first type of prompt word of the image to be identified based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be identified. The feature ranking and feature weight are determined according to the subjective data and objective data of the experts in identifying the same target; the cognitive process reasoning system is used to generate a reasoning text according to the domain of the image to be identified, the image to be identified and the first type of prompt word, wherein the reasoning text is used to simulate the visual cognitive process of the expert in identifying the image to be identified.

[0159] In order to better illustrate the beneficial effects of the application, the following simulation experiments are carried out:

[0160] Simulation Experiment 1

[0161] Exemplarily, a series of automatic indicators including ROUGE, BERTScore, MoverScore, MeteorScore and BARTScore are used in Experiment 1 to comprehensively evaluate the reasoning texts generated by the present application and the reasoning texts generated by the traditional prompt engineering method. These indicators are widely used in natural language processing tasks such as report generation and machine translation, covering character matching, semantic consistency and distance measurement in high-dimensional semantic space, and can quantify the similarity between generated text and reference answer from multiple dimensions. Simulation Experiment 1 can comprehensively analyze the quality of the reasoning text through these indicators.

[0162] Referring to Figure 10 , wherein the light gray area is the evaluation result of the present application, and the dark gray area is the evaluation result of the traditional method. It can be seen that the present application has stably surpassed the traditional method in almost all indicators, especially in the BERTScore which measures semantic fidelity and the BARTScore which measures reasoning process and language causal structure. The score difference is particularly significant, with an increase of about 15% to 20%. This can fully reflect the technical advantages of the present application in semantic control and expert logic reconstruction.

[0163] Simulation Experiment 2

[0164] Exemplarily, since the indicators such as BERTScore and METEOR have high tolerance for redundant information, the generated text may contain repeated or irrelevant content without a significant decrease in score, thus failing to accurately reflect the actual quality of the text. Therefore, these indicators are not effective in evaluating fine-grained recognition tasks. In order to avoid the generation of redundant and irrelevant information in the generated text, Experiment 2 uses the cutting-edge model GPT-4o in the field of natural language processing as the core evaluation tool. This model can score the text quality from three model indicators: relevance, completeness and redundancy. The score of each dimension ranges from 1 to 5.

[0165] Specifically, the relevance indicator is used to evaluate the degree of fit between the text and the recognition problem, the completeness indicator focuses on whether the reasoning text fully covers key elements and information, and the redundancy indicator measures whether the text contains redundant, repetitive or worthless content.

[0166] In specific operation, first, GPT-4o can be provided with guidance information highly consistent with the research context, so that it can conduct in-depth review of the reasoning text according to the field requirements. Subsequently, the model checks the logical structure, information content and context consistency of the text according to the pre-set indicators one by one, and gives a refined scoring result.

[0167] Referring to Figure 11It can be seen that the present application exceeds the traditional method in the three dimensions of relevance, integrity and redundancy. The results show that the traditional prompting engineering method generally scores low in integrity, with an average of about 4.2, while the present application method can reach an average of 4.9, indicating that it can be closer to the real expert reasoning expression in terms of comprehensive information coverage. In terms of redundancy, the traditional method has the problem of excessive repetition of structure or templating language, and the present application effectively reduces invalid sentence patterns by introducing expert sorting mechanism and semantic control prompting template, greatly improving the redundancy score. The relevance index also shows that the present application can closely organize the description content around the key evidence of the image, avoiding the common "hallucination sentence" or "logical jump" in the traditional CoT generation.

[0168] Simulation experiment 3

[0169] In one example, in order to further explore the generation and evaluation mechanism of expert-level knowledge in fine-grained object recognition, experiment 3 selects three experts with more than 15 years of experience in aircraft recognition to evaluate the following Table 1: the text written by the experts themselves, the text generated by the traditional method and the reasoning text generated by the present application. The three experts have rich practical experience and have accumulated deep knowledge reserve in the field of aircraft recognition. Experts can evaluate the reasoning text based on two new angles: fact consistency and cognitive consistency.

[0170] Table 1

[0171]

[0172] For example, fact consistency focuses on whether the diagnostic features in the reasoning text conform to the real fine-grained scene, while cognitive consistency focuses on whether the reasoning text can reflect the cognitive logic followed by experts in the actual judgment process.

[0173] Specifically, in the fact consistency evaluation, the experimenters manually extracted the feature information of each recognition task from the reasoning text and divided it into two categories: valid recognition features and invalid recognition features. Experts determine whether each feature is valid through three key judgments (whether the feature name is accurate, whether the feature value is accurate, and whether the feature is helpful for judgment), and only when the answers to all three questions are "yes", the feature is considered valid.

[0174] Specifically, in the cognitive consistency evaluation, the complete recognition text output by the model is given to the experts for scoring, and the experts give a score of 1-100 according to their experience and judgment logic to measure the consistency and credibility of the model in cognitive logic.

[0175] The experimental results show that, in terms of information integrity, the expert text description is rich in details and comprehensive in feature coverage, but lacks a unified organizational structure due to individual style differences; traditional reasoning text generation methods, such as the prompting engineering method, can generate texts that conform to professional standards to some extent, but due to the generalization of domain knowledge, some details are missed; and the text generated by the present application is more accurate in detail expression, not only accurately retaining key features, but also ensuring the hierarchical presentation of information, making the identification logic clearer. In terms of logical consistency, the text generated by the prompting engineering method often has problems such as disordered description sequence and lack of logical connection between information; the reasoning text generated by the present application is more in line with the cognitive logic of human experts in structure, with a description sequence from the whole to the part, covering size, deck form, key markers, equipment configuration, etc., enabling readers to more intuitively understand the identification process of the target. In terms of feature accuracy, the present application has more obvious advantages, and the prompting engineering method is prone to introduce redundant or ambiguous expressions when generating feature descriptions, affecting the performance of the model in actual tasks. The method of the present application can integrate expert experience and structured knowledge extraction technology to ensure that all information in the reasoning text is task-related and accurately corresponds to the key features of expert interpretation. It is worth noting that the similarity in structure between the expert text and the reasoning text generated by the present application shows that the present patent method not only reproduces the cognitive process of experts, but also optimizes the expert description, making it more structured and universal.

[0176] Therefore, the present application generates diagnostic prompts (first type of prompts) for the image to be identified by determining the diagnostic feature ranking, weight and semantic description based on the subjective experience and objective data of the domain expert, so that the first type of prompts can internalize the dynamic decision-making preferences and reasoning logic of human experts when identifying such targets; by inputting the first type of prompts into the cognitive process reasoning system based on the large language model, the system can simulate the identification process of the expert under the constraint and guidance of the identification criterion priority framework (i.e. feature ranking, weight) obtained based on expert knowledge, and generate reasoning text according to the diagnostic logic of human experts; efficiently and automatically generate semantic expressions of the visual identification process of fine-grained targets.

[0177] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

Claims

1. A method for generating a fine-grained visual target recognition expert knowledge agent, characterized in that, The method comprises the following steps: based on the feature ranking, feature weight, and semantic description information of the diagnostic features in the image to be recognized, a first type of prompt word of the image to be recognized is generated, wherein the feature ranking and feature weight are determined according to subjective data and objective data of experts in identifying the same type of target; inputting the domain of the image to be recognized, the image to be recognized, and the first type of prompt word into a cognitive process reasoning system based on a large language model to obtain a reasoning text, wherein the reasoning text is used to simulate the visual cognitive process of the expert in identifying the image to be recognized; The cognitive process reasoning system comprises: a domain knowledge enhancement acquisition module, which is used to retrieve the domain knowledge of the image to be recognized according to the domain of the image to be recognized; a feature selection module, which is used to extract diagnostic features with diagnostic value in the process of identifying the image to be recognized according to the first type of prompt word, and obtain visual salient identification features, wherein the visual salient identification features comprise at least one diagnostic feature; an attribute extraction module, which is used to retrieve structural attribute information of the visual salient identification features; a multi-modal expert knowledge guided model reasoning module, which is used to integrate the domain knowledge, the visual salient identification features, and the structural attribute information based on the feature ranking in the first type of prompt word to obtain the reasoning text.

2. The method of claim 1, wherein, The subjective data comprises saliency ranking of the region of interest and / or pair-wise importance score of adjacent regions of interest; wherein the region of interest is a region in which the diagnostic feature is located in the sample image.

3. The method of claim 2, wherein, The objective data comprises the number of fixations on the region of interest, the duration of fixations on the region of interest, the first fixation time on the region of interest, and the fixation sequence on the region of interest.

4. The method of claim 3, wherein, The feature weight and the feature ranking are obtained by taking the diagnostic feature as a candidate point in an improved VIKOR round-by-round extraction decision algorithm and taking a feature evaluation index as a candidate index in the improved VIKOR round-by-round extraction decision algorithm. The feature evaluation index comprises degree centrality, betweenness centrality, closeness centrality, and a weight coefficient of the diagnostic feature; the degree centrality, betweenness centrality, and closeness centrality are determined according to an expert saccade path network, the expert saccade path network is drawn according to the objective data, and the weight coefficient is determined according to the subjective data.

5. The method of claim 1, wherein, The domain knowledge enhancement acquisition module is specifically configured to: retrieve similar texts from a professional knowledge base of the domain of the image to be recognized based on a sparse retrieval algorithm and a dense retrieval algorithm respectively to obtain sparse similar texts and dense similar texts; fuse the sparse similar texts and the dense similar texts based on a reverse ranking fusion algorithm to obtain fused texts and fusion scores thereof; select the top k fused texts with higher fusion scores as the domain knowledge, wherein k is a preset text selection threshold.

6. The method of claim 1, wherein, The feature selection module is specifically configured to: The multi-modal large language model is called to analyze the to-be-identified image, visibility, integrity and local saliency of each diagnostic feature in the to-be-identified image are judged, and candidate features are obtained; It is judged whether each candidate feature exists in the to-be-identified image and multiple rounds of verification are performed to exclude illusory features; The candidate features after exclusion are output as the visually salient identification features.

7. The method of claim 1, wherein, The attribute extraction module is specifically configured to: Based on the visually salient identification features, a query template is constructed to search for structural description information of the visually salient identification features, obtaining first-class attribute information; A multi-modal large model is used to perform semantic inquiry and attribute identification on the region where the visually salient identification features are located in the to-be-identified image, obtaining second-class attribute information; The first-class attribute information and the second-class attribute information are fused to obtain the structural attribute information.

8. The method of claim 1, wherein, The multi-modal expert knowledge guided model reasoning module is specifically configured to: According to the feature ranking and the structural attribute information, a second type of prompt word is generated, wherein the second type of prompt word is composed of a structural description guiding module, an expert analogy reminding module, a reasoning task instruction module, and a final judgment prompting module; The second type of prompt word is input into the large language model for multiple rounds of content generation, and the generated content is subjected to repeated detection and punishment, logical consistency verification, and structural integrity control, obtaining a final generated text; Based on an expert language style converter, the final generated text is converted into a structured format, obtaining the reasoning text.

9. A fine-grained visual target recognition expert knowledge intelligent agent generation apparatus, characterized in that, It includes: A first type of prompt word generation module, which is configured to generate a first type of prompt word for a to-be-identified image based on feature ranking, feature weight, and semantic description information of diagnostic features in the to-be-identified image, wherein the feature ranking and feature weight are determined according to subjective data and objective data of experts in identifying similar targets; A cognitive process reasoning system based on a large language model, which is configured to generate a reasoning text according to the domain of a to-be-identified image, the to-be-identified image, and the first type of prompt word, wherein the reasoning text is used to simulate the visual cognitive process of an expert in identifying the to-be-identified image; The cognitive process reasoning system includes: A domain knowledge enhancement acquisition module, which is configured to retrieve domain knowledge of the to-be-identified image according to the domain of the to-be-identified image; A feature selection module, which is configured to extract diagnostic features with diagnostic value in the process of identifying the to-be-identified image according to the first type of prompt word, obtaining visually salient identification features, wherein the visually salient identification features include at least one diagnostic feature; An attribute extraction module, which is configured to retrieve structural attribute information of the visually salient identification features; A multi-modal expert knowledge guided model reasoning module, which is configured to integrate the domain knowledge, the visually salient identification features, and the structural attribute information based on the feature ranking in the first type of prompt word, obtaining the reasoning text.

Images