Image semantic retrieval method and device based on multimodal large model

Abstract

This invention provides an image semantic retrieval method and apparatus based on a multimodal large model. The method includes: using a preset prompt word engineering template to generate a structured semantic description of the image to be retrieved according to a preset multidimensional structural framework, and storing this description in a semantic database; semantically deconstructing the user's natural language query text, extracting multiple dimensional elements corresponding to the element dimensions in the multidimensional structural framework, and generating dynamic retrieval keys based on these elements; based on the dynamic retrieval keys, performing semantic matching and scoring on the structured semantic description of the image to be retrieved at each element dimension of the dynamic retrieval keys; fusing the scores of each element dimension into a final matching score according to a preset weight fusion strategy; sorting the final matching scores to obtain a ranking result, and returning the retrieved image based on the ranking result. The method and apparatus of this invention can overcome the semantic gap and achieve accurate multidimensional image retrieval.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an image semantic retrieval method and apparatus based on a multimodal large model. Background Technology

[0002] With the explosive growth of image data, traditional image retrieval methods based on keyword tags or manual annotation are no longer sufficient to meet the needs of accurate understanding of complex semantic content.

[0003] In existing technologies, retrieval systems based on vision-language pre-trained models (such as CLIP), while capable of cross-modal matching, still have significant limitations: First, the free text descriptions or global vectors they rely on lack structured parsing, making it difficult to meticulously depict the semantic relationships between "entities-events-environment" in images, resulting in insufficient understanding of fine-grained, multi-element composite queries; second, the retrieval process mostly uses static embedding vector similarity calculations, failing to dynamically adjust matching strategies based on query intent, resulting in poor flexibility; furthermore, the inherent semantic gap between natural language query text and image content remains significant in complex scenarios, especially when actions, relationships, and contextual reasoning are involved, making it difficult to achieve both recall and precision. Although recent multimodal large-scale models (such as BLIP and Qwen-VL) have performed well in generating descriptions, their outputs are mostly unstructured text, failing to form semantic representations that machines can efficiently compute and combine, thus hindering the realization of accurate and intelligent retrieval in large-scale image databases.

[0004] Therefore, there is an urgent need in the existing technology for an image semantic retrieval method and device that can deeply integrate multimodal understanding capabilities and support structured semantic generation and dynamic matching. Summary of the Invention

[0005] Therefore, it is necessary to provide an image semantic retrieval method and apparatus based on a multimodal large model to address the aforementioned technical problems.

[0006] In a first aspect, the present invention provides an image semantic retrieval method based on a multimodal large model, the method comprising:

[0007] The image to be retrieved is input into the multimodal large model. The multimodal large model generates a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework through the preset prompt word engineering template, and stores the structured semantic description in the semantic database. The element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension.

[0008] Receive the user's natural language query text, perform semantic deconstruction on the natural language query text, extract multiple dimensional elements corresponding to the element dimensions in the multidimensional structure framework, and generate dynamic search key values ​​based on the multiple dimensional elements, including search logic and field requirements.

[0009] Based on dynamic retrieval keys, the structured semantic descriptions of the images to be retrieved in the semantic database are semantically matched and scored on each element dimension of the dynamic retrieval keys.

[0010] According to the preset weight fusion strategy, the scores of each element dimension are merged into the final matching score;

[0011] The final matching scores are sorted to obtain the sorted results, and the search result images are returned based on the sorted results.

[0012] Optionally, the image to be retrieved is input into a multimodal large model, and the multimodal large model generates a structured semantic description of the image to be retrieved according to a preset multidimensional structural framework using a preset prompt word engineering template, including:

[0013] The image to be retrieved is input into the multimodal big model. The multimodal big model is guided by the preset prompt word engineering template to perform a structured semantic description of the image to be retrieved according to the entity element dimension, event element dimension and environmental element dimension in the preset multidimensional structural framework.

[0014] The prompt word engineering template includes a coordinate specification layer, a content constraint layer, a text processing layer, and a format guarantee layer, which are used to force the output of structured semantic descriptions that conform to the preset format and ensure the consistency of references between the dimensions of each element.

[0015] Optionally, the system receives the user's natural language query text, performs semantic deconstruction on the natural language query text, extracts multiple dimensional elements corresponding to the element dimensions in the multidimensional structural framework, and generates dynamic search key values ​​based on the multiple dimensional elements, including search logic and field requirements, including:

[0016] The system utilizes a multimodal large model to receive users' natural language query text and performs semantic understanding on the natural language query text, extracting the dimension elements corresponding to entity element dimension, event element dimension, and environment element dimension respectively.

[0017] Based on the extracted multiple dimensions of elements, a dynamic retrieval key value is generated, which includes retrieval logic and field requirements; among which, the field requirements include hard constraint fields and core matching fields.

[0018] Optionally, based on the extracted multiple dimensional elements, dynamic search key values ​​are generated, including search logic and field requirements, including:

[0019] A hierarchical thesaurus is pre-defined; wherein the hierarchical thesaurus includes at least a general semantic layer, a domain-specific semantic layer and a dynamically updated layer, and the hierarchical mapping relationship between a word and one or more of its superordinate concepts is defined in the hierarchical thesaurus;

[0020] For the extracted multiple dimensional elements, query the hierarchical thesaurus to obtain one or more corresponding superordinate concept words, so as to form the first-level extended set corresponding to each dimensional element;

[0021] For each dimension element, based on the multidimensional structural framework, pre-selected associated fields related to each dimension element are determined, and the corresponding replacement strategy is selected based on the pre-selected associated fields related to each dimension element.

[0022] Based on the replacement strategy selected for each pre-selected associated field, the hierarchical thesaurus is queried to obtain one or more extended words corresponding to each pre-selected associated field, so as to form a second-level extended set;

[0023] The dynamic retrieval key values ​​include: the first-level extended set corresponding to each dimension element, the second-level extended set of pre-selected associated fields corresponding to each dimension element, and the confidence decay coefficient preset for extended terms at different levels.

[0024] Optionally, the replacement strategy includes:

[0025] If the pre-selected associated fields represent the attributes of the entity, an exact matching strategy is adopted, and no expansion is performed;

[0026] If the pre-selected associated fields represent actions, behaviors, or interaction relationships, a conservative replacement strategy is adopted, and the core verbs are expanded to higher levels.

[0027] If the pre-selected associated fields represent text content, an aggressive replacement strategy is adopted to expand upwards and to related terms.

[0028] Optionally, the confidence attenuation coefficients preset for extended terms at different levels include: a first attenuation coefficient corresponding to the original dimension element and / or a second attenuation coefficient corresponding to a superordinate concept word with a direct hierarchical relationship and / or a third attenuation coefficient corresponding to a superordinate concept word with an indirect hierarchical relationship; wherein the first attenuation coefficient is greater than the second attenuation coefficient, and the second attenuation coefficient is greater than the third attenuation coefficient.

[0029] Optionally, based on the dynamic retrieval key, the structured semantic description of the image to be retrieved in the semantic database is semantically matched and scored on each element dimension of the dynamic retrieval key, including:

[0030] In response to the dynamic retrieval key, the dimension elements corresponding to the entity element dimension, event element dimension, and environment element dimension in the structured semantic description of the image to be retrieved in the semantic database are obtained respectively.

[0031] The obtained elements of each dimension are semantically compared with the corresponding first-level extended set in the dynamic search key value to obtain the first-level matching result;

[0032] For the dimensional elements that yield the first-level matching results, the associated pre-selected associated fields are semantically compared with the corresponding second-level extended sets in the dynamic retrieval key values ​​to obtain the second-level matching results;

[0033] Using a pre-defined semantic similarity algorithm, based on the first-level matching results and the second-level matching results, the matching scores of the structured semantic descriptions on each element dimension are calculated respectively.

[0034] When matching with a higher-level concept word, the matching score is determined based on semantic similarity and the confidence decay coefficient corresponding to the dynamic retrieval key value.

[0035] Optionally, according to a preset weighting fusion strategy, the scores of each element dimension are fused into a final matching score, including:

[0036] Determine the query type of the natural language query text; the query type should include at least action queries and scenario queries.

[0037] In response to the query type, obtain the basic weight allocation strategy corresponding to the query type; the basic weight allocation strategy includes: for action-type queries, the weight of the event element dimension is higher than that of the entity element dimension and the environment element dimension; for scenario-type queries, the weight of the environment element dimension is higher than that of the entity element dimension and the event element dimension.

[0038] Receive user-defined configurations for the weights of at least one dimension in the basic weight allocation strategy, and form the final weight allocation strategy;

[0039] Based on the final weighting strategy, the scores of the entity element dimension, event element dimension, and environmental element dimension are weighted and summed to obtain the final matching score.

[0040] Optionally, the final matching scores are sorted to obtain a sorted result, and based on the sorted result, the retrieval result image is returned, including:

[0041] The final match score is compared with a preset threshold.

[0042] Images with a final matching score greater than or equal to a preset threshold are selected to form a candidate image set.

[0043] The images in the candidate image set are sorted in descending order according to their final matching scores to obtain the sorting results. The top N images in the sorting results are then selected as the search result images and returned.

[0044] Secondly, the present invention provides an image semantic retrieval device based on a multimodal large model, the device comprising:

[0045] The structured description generation module is used to input the image to be retrieved into the multimodal large model, and to generate a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework through the preset prompt word engineering template; wherein, the element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension.

[0046] A semantic database, connected to the structured description generation module, is used to store structured semantic descriptions;

[0047] The query parsing and key-value generation module is used to receive the user's natural language query text, perform semantic deconstruction on the natural language query text, extract multiple dimensional elements corresponding to the element dimensions in the multidimensional structure framework, and generate dynamic search key-values ​​based on the multiple dimensional elements, including search logic and field requirements.

[0048] The multi-dimensional semantic matching and scoring module is connected to the semantic database and the query parsing and key value generation module, respectively. It is used to perform semantic matching and scoring on each element dimension of the structured semantic description of the image to be retrieved in the semantic database based on the dynamic retrieval key value.

[0049] The weight fusion module, connected to the multi-dimensional semantic matching and scoring module, is used to fuse the scores of each element dimension into the final matching score according to the preset weight fusion strategy.

[0050] The results sorting and return module, connected to the weight fusion module, is used to sort the final matching scores, obtain the sorted results, and return the search result image based on the sorted results.

[0051] The image semantic retrieval method and apparatus based on a multimodal large model provided by this invention firstly transforms the content of the image to be retrieved into a semantic representation that can be accurately parsed and computed by a machine by introducing a three-dimensional structural framework of "entity-event-environment". This fundamentally solves the problem of fragmented semantic expression in traditional methods and significantly improves the ability to characterize complex semantics and retrieval accuracy. Secondly, the innovative dynamic retrieval key generation and multi-dimensional semantic matching mechanism can dynamically adjust the matching strategy according to the user's query intent. Combined with synonym expansion and confidence decay mechanisms, it effectively controls false positives while ensuring high recall, achieving flexible and accurate semantic retrieval. In addition, the configurable weight fusion strategy allows the system to adapt or allow users to define the importance of different dimensions, making the retrieval results more suitable for diverse query needs and greatly improving the user experience. Attached Figure Description

[0052] Figure 1a This is a flowchart illustrating an image semantic retrieval method based on a multimodal large model provided in an embodiment of the present invention.

[0053] Figure 1b This is another flowchart illustrating the image semantic retrieval method based on a multimodal large model provided in an embodiment of the present invention;

[0054] Figure 1c This is another flowchart illustrating the image semantic retrieval method based on a multimodal large model provided in this embodiment of the invention;

[0055] Figure 2 This is a schematic diagram of a circuit module structure of an image semantic retrieval device based on a multimodal large model provided in an embodiment of the present invention;

[0056] Figure 3 This is an internal structural diagram of a computer device according to one embodiment of the present invention. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0058] like Figure 1a As shown, this invention provides an image semantic retrieval method based on a multimodal large model, the method comprising:

[0059] Step S11: Input the image to be retrieved into the multimodal large model, and use the preset prompt word engineering template to make the multimodal large model generate a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework, and store the structured semantic description in the semantic database; wherein, the element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension.

[0060] Optionally, step S11, which involves inputting the image to be retrieved into a multimodal large model and using a preset prompt word engineering template to generate a structured semantic description of the image to be retrieved according to a preset multidimensional structural framework, includes:

[0061] The image to be retrieved is input into the multimodal big model. The multimodal big model is guided by the preset prompt word engineering template to perform a structured semantic description of the image to be retrieved according to the entity element dimension, event element dimension and environmental element dimension in the preset multidimensional structural framework.

[0062] The prompt word engineering template includes a coordinate specification layer, a content constraint layer, a text processing layer, and a format guarantee layer, which are used to force the output of structured semantic descriptions that conform to the preset format and ensure the consistency of references between the dimensions of each element.

[0063] In this invention, the multimodal large model can receive and fuse multiple inputs such as images and text, understand and align them in a unified semantic space, and generate corresponding outputs or complete cross-modal reasoning tasks. It can be an existing multimodal large model, such as Qwen3-VL 8B, Qwen3-VL 32B, Qwen2.5-VL 72B, InternVL3.5-241B-A28B, InternVL3.5-2B, Keye-VL-8B, MiMo-VL-RL-7B, Kimi-VL-2506-16B-A3B, etc. Those skilled in the art can flexibly choose according to actual needs, and there is no limitation here.

[0064] In this invention, the coordinate standardization layer is used to standardize spatial positioning, forcibly unifying the bounding box coordinate output format and numerical range of entities in the image to be retrieved, ensuring that spatial information can be accurately parsed and calculated by the machine; the content constraint layer is used to ensure the evidentiary and specific nature of the description, forcing all descriptions to be based on visible visual features in the image (such as color, action, state, etc.), avoiding vague, speculative, or generalized output, and ensuring the objectivity and fine granularity of semantic description; the text processing layer is used to structurally process the text information in the image to be retrieved, distinguishing the text in the image to be retrieved into "independent text carriers" (such as slogans, signs, etc.) and "entity-attached text" (such as vehicle markings), and using different structural strategies for recording and association, realizing deep integration and precise management of image and text information; the format guarantee layer is used to force the output of structured data and maintain internal consistency, requiring the multimodal large model to output clean data that strictly conforms to a predetermined architecture (such as JSON), and through the ID reference mechanism, ensuring that the logical association between descriptions of different element dimensions such as entities, events, and environments is correct and unambiguous. In addition, it should be noted that for the prompt word engineering template, those skilled in the art can flexibly add or delete various constraints according to actual needs, which is not limited here.

[0065] In this invention, the number and types of element dimensions in the multidimensional structural framework can be flexibly selected by those skilled in the art according to actual needs, and are not limited here.

[0066] Optionally, the multidimensional structural framework is a three-dimensional structural framework, which includes entity element dimension, event element dimension, and environmental element dimension. The entity element dimension is used to describe the specific objects and attributes of the specific objects in the image to be retrieved. The specific objects include one or more of people, animals, and objects (such as motor vehicles, non-motor vehicles, and text carriers). The attributes of the specific objects include one or more of color, position, and state. The event element dimension is used to describe the actions, behaviors, or interactions that are taking place in the image to be retrieved. The environmental element dimension is used to describe the contextual information of the image to be retrieved. The contextual information includes not only lighting, weather, and season, but also scene background and / or time.

[0067] In one specific embodiment of the present invention, a multimodal large model is guided by a preset prompt word engineering template to generate a structured semantic description of the image to be retrieved. The complete field composition and generation mechanism within its three-dimensional structural framework are as follows:

[0068] (1) Preset structured semantic description constituent fields.

[0069] ① The entity dimension (entities) may include the following fields:

[0070] ID field: A unique identifier for a specific object (i.e., an entity), used for cross-dimensional referencing;

[0071] Entity type field: The semantic category of the specific object, such as person, motor vehicle, text carrier, etc.;

[0072] Entity Name field: The generic name or label of the specific object;

[0073] Appearance fields: Visual attributes of a specific object, including entity subtype fields, content fields, color fields, license plate fields, etc.

[0074] First Evidence Field: Visual evidence supporting the identification and location of the specific object, including the first coordinate field, the first text description field, and the first confidence field.

[0075] ② The event element dimension (events) may include the following fields:

[0076] Event type field (event_type): The broad category of the event;

[0077] Event subtype field: A more refined classification of the event;

[0078] Related entity field (actors): Specifies the entities participating in this event by referencing the entity's ID;

[0079] The judgment field describes the specific manifestation of the event in natural language.

[0080] Second Evidence Field: Visual evidence supporting the judgment of this event, including the second coordinate field, the second text description field, and the second confidence field.

[0081] ③ The environment dimension may include the following fields:

[0082] The lighting field (light_env): The lighting conditions of the scene;

[0083] Weather field: Weather conditions, including the first specific value and the third evidence field;

[0084] Season field: Seasonal information, including the second specific value and the fourth evidence field.

[0085] It should be noted that the types and number of fields included in each element dimension of the structured semantic description can be flexibly set by those skilled in the art according to different application scenarios, and are not limited here.

[0086] (2) By using the constraints of the prompt word engineering template, guide the multimodal large model to generate the above fields and corresponding content.

[0087] ① The coordinate specification layer in the prompt word project template guides the generation of all bounding box coordinate fields in the multimodal large model; Template constraints: All coordinates must use the normalized COCO[x,y,w,h] format, and the values ​​must be in the range [0,1].

[0088] ② The content constraint layer in the prompt word engineering template guides the multimodal large model to generate all text description fields; template constraint: all descriptions must be based on visible visual evidence to avoid speculation.

[0089] ③ The text processing layer in the prompt word engineering template guides the multimodal large model to generate text content in the appearance dimension of the entity element; Template constraints: the entity type field of independent text carriers (such as signs) is text; the attached text (such as vehicle body text) is the content (i.e., content field) of the entity appearance.

[0090] ④ The format guarantee layer in the prompt word project template guides the generation of ID fields and JSON format for multimodal large models; Template constraints: the output must be pure JSON, entities must be assigned unique IDs, and events must reference entity IDs through the actors field.

[0091] The following demonstrates the structured semantic description of the scenario "a fire breaks out in a warehouse area during rainy weather," generated by a multimodal large model, guided by constraints from a four-layer cue word engineering template:

[0092] json

[0093] {

[0094] "environment":{

[0095] "light_env":"dim", / / Lighting field is set to dim

[0096] "weather":{"value":"rain","basis":"The ground is wet with obvious reflections, and rain streaks are visible in the image"}, / / The first specific value in the weather field is rainy, and the third piece of evidence is that the ground is wet with obvious reflections, and rain streaks are visible in the image.

[0097] "season":{"value":"winter","basis":"All personnel on site were wearing heavy cotton-padded coats or down jackets"}

[0098] }, / / The second specific value in the season field is winter, and the fourth piece of evidence is that all personnel on site were wearing heavy cotton-padded clothes or down jackets.

[0099] "entities":[

[0100] {

[0101] "id":"t1", / / The ID field of the first entity is t1

[0102] "type":"text", / / The entity type field of the first entity is text.

[0103] "name":"Hazardous Goods Warning Sign", / / The name field of the first entity is "Hazardous Goods Warning Sign"

[0104] "appearance":{"text_type":"warning","content":"Caution: Fire Prevention, No Open Flames","colors":["Red","White","Black"]}, / / The entity subtype field in the appearance field of the first entity is "warning", the content field in the appearance field of the first entity is "Caution: Fire Prevention, No Open Flames", and the color field in the appearance field of the first entity is "red", "white", and "black".

[0105] "evidence":[{"bbox":[0.15,0.70, 0.18,0.06],"desc":"Red background with white border, triangular warning sign, black icon and text, fixed to the fence","confidence":0.98}] / / The first coordinate field in the first evidence field of the first entity is [0.15,0.70,0.18,0.06]; the first text description field in the first evidence field of the first entity is "Red background with white border, triangular warning sign, black icon and text, fixed to the fence"; the first confidence field in the first evidence field of the first entity is 0.98

[0106] },

[0107] {

[0108] "id":"v1", / / The ID field of the second entity is v1

[0109] "type":"vehicle_motor", / / The entity type field of the second entity is a motor vehicle.

[0110] "name":"Hazardous Goods Transport Vehicle", / / The name field of the second entity is "Hazardous Goods Transport Vehicle"

[0111] "appearance": { / / Appearance field of the second entity

[0112] "vehicle_type":"tanker_truck", / / The entity subtype field in the appearance field of the second entity is a transport vehicle.

[0113] "color": "orange", / / The color field in the appearance field of the second entity is orange

[0114] "plate_text": "Su C·WX123", / / The license plate field in the appearance field of the second entity is Su C·WX123

[0115] "content": "爆" / / The content field in the appearance field of the second entity is 爆

[0116] },

[0117] "evidence": [{"bbox": [0.60, 0.55, 0.25, 0.20], "desc": "Orange tanker truck with a huge black '爆' character sprayed in the middle of the body", "confidence": 0.97}] / / The first coordinate field in the first evidence field of the second entity is [0.60, 0.55, 0.25, 0.20]; the first text description field in the first evidence field of the second entity is Orange tanker truck with a huge black '爆' character sprayed in the middle of the body; the first confidence field in the first evidence field of the second entity is 0.97

[0118] }

[0119] ,

[0120] "events": [{{END]]

[0121] "event_type": "other_anomaly", / / The event type field is other anomaly

[0122] "subtype": "fire", / / The event subtype field is fire

[0123] "actors": [], / / The associated entity field is empty

[0124] "judgement": "There is an open fire accompanied by black smoke continuously emitting from the exhaust vent on the side of the warehouse", / / The description field is There is an open fire accompanied by black smoke continuously emitting from the exhaust vent on the side of the warehouse

[0125] "evidence":[{"bbox":[0.40,0.30,0.15,0.25],"desc":"Orange-red flames and upward-blowing black smoke are visible at the metal ventilation opening of the building","confidence":0.99}] / / The second coordinate field in the second evidence field is [0.40,0.30,0.15,0.25]; the second text description field in the second evidence field is "Orange-red flames and upward-blowing black smoke are visible at the metal ventilation opening of the building"; the second confidence level field in the second evidence field is 0.99

[0126] }]

[0127] }

[0128] Optionally, the multidimensional structural framework is a five-dimensional framework, which includes a first entity element dimension, an event element dimension, a second entity element dimension, an environmental element dimension, and an emotional element dimension; wherein the first entity element dimension and the second entity element dimension together constitute the entity element dimension. Specifically, the first entity element dimension describes the initiating subject and its attributes in the image to be retrieved; the event element dimension describes the specific actions, behaviors, or interactive relationships between the subject and the object; the second entity element dimension describes the receiving object, related objects, and their attributes in the image to be retrieved; the environmental element dimension describes the lighting, weather, season, scene background, time, etc., presented in the image to be retrieved; and the emotional element dimension describes the overall emotional atmosphere, emotional tendency, or subjective evaluation inferred from the content of the image to be retrieved.

[0129] Step S12: Receive the user's natural language query text, perform semantic deconstruction on the natural language query text, extract multiple dimensional elements corresponding to the element dimensions in the multidimensional structure framework, and generate dynamic search key values ​​including search logic and field requirements based on the multiple dimensional elements.

[0130] In one optional embodiment of the present invention, such as Figure 1b As shown, step S12 specifically includes:

[0131] Step S121: Receive the user's natural language query text using a multimodal large model, and perform semantic understanding on the natural language query text to extract the dimension elements corresponding to the entity element dimension, event element dimension, and environment element dimension respectively.

[0132] Step S122: Based on the extracted multiple dimensional elements, generate dynamic search key values ​​that include search logic and field requirements; wherein, the field requirements include hard constraint fields and core matching fields.

[0133] In an optional embodiment of the present invention, step S122, generating a dynamic retrieval key value including retrieval logic and field requirements based on the extracted multiple dimensional elements, includes:

[0134] A hierarchical thesaurus is pre-defined; wherein the hierarchical thesaurus includes at least a general semantic layer, a domain-specific semantic layer and a dynamically updated layer, and the hierarchical mapping relationship between a word and one or more of its superordinate concepts is defined in the hierarchical thesaurus;

[0135] For the extracted multiple dimensional elements, query the hierarchical thesaurus to obtain one or more corresponding superordinate concept words, so as to form the first-level extended set corresponding to each dimensional element;

[0136] For each dimension element, based on the multidimensional structural framework, pre-selected associated fields related to each dimension element are determined, and the corresponding replacement strategy is selected based on the pre-selected associated fields related to each dimension element.

[0137] Based on the replacement strategy selected for each pre-selected associated field, the hierarchical thesaurus is queried to obtain one or more extended words corresponding to each pre-selected associated field, so as to form a second-level extended set;

[0138] The dynamic retrieval key values ​​include: the first-level extended set corresponding to each dimension element, the second-level extended set of pre-selected associated fields corresponding to each dimension element, and the confidence decay coefficient preset for extended terms at different levels.

[0139] In one specific embodiment of the present invention, the replacement strategy may include: if the pre-selected associated field represents the attribute of the entity, an exact matching strategy is adopted and no expansion is performed; if the pre-selected associated field represents an action, behavior or interaction relationship, a conservative replacement strategy is adopted and its core verb is expanded to the upper level; if the pre-selected associated field represents text content, an aggressive replacement strategy is adopted and the upper level and associated terms are expanded.

[0140] Specifically, entity attributes include entity type fields, color fields in the appearance field of entity element dimensions, etc.; actions, behaviors, or interaction relationships include description fields in the event element dimension, etc.; and text content includes content fields in the appearance field of entity element dimensions, etc.

[0141] In this invention, the depth (number of layers) of upward hierarchical expansion can be flexibly set by those skilled in the art according to actual needs, and is not limited here. Related term expansion refers to the expansion of related terms at the same level. Furthermore, it should be noted that if the fields included in the core matching fields cannot represent the entity's attributes, actions, behaviors, or interaction relationships, or text content, then no expansion will be performed. Of course, those skilled in the art can also flexibly choose according to actual needs, and is not limited here.

[0142] It should be noted that the pre-selected association fields can be one or more fields from the multidimensional structural framework corresponding to each dimension element; this is not limited here. Furthermore, those skilled in the art can flexibly choose the replacement strategy corresponding to the pre-selected association fields according to actual needs; this is also not limited here.

[0143] Optionally, the confidence attenuation coefficients preset for extended terms at different levels include: a first attenuation coefficient corresponding to the original dimension element and / or a second attenuation coefficient corresponding to a superordinate concept word with a direct hierarchical relationship and / or a third attenuation coefficient corresponding to a superordinate concept word with an indirect hierarchical relationship; wherein, the first attenuation coefficient is greater than the second attenuation coefficient, and the second attenuation coefficient is greater than the third attenuation coefficient.

[0144] Specifically, the first attenuation coefficient for the original dimensional element is 1.0, the second attenuation coefficient for the superordinate concept word with a direct hierarchical relationship is 0.85, and the third attenuation coefficient for the superordinate concept word with an indirect hierarchical relationship is 0.7. Of course, those skilled in the art can flexibly set each attenuation coefficient according to actual needs, and no limitation is made here.

[0145] To enable those skilled in the art to clearly understand step S12, a detailed explanation will be given using the example of a user-input natural language query text that reads "Find all warehouse fire monitoring footage that occurred on rainy days".

[0146] ① Utilize a multimodal large model to perform semantic understanding on the natural language query text input by the user, and decompose and classify it into a three-dimensional structural framework, as follows:

[0147] Entity element dimension: "Warehouse";

[0148] Event element dimension: "Fire";

[0149] Environmental element dimension: "Rainy day".

[0150] ② Based on the extracted multiple dimensional elements (warehouse, fire, rainy day), a dynamic search key-value pair that can directly drive the search engine is automatically generated, as follows:

[0151] json

[0152] {

[0153] "logic":"AND", / / The search logic is AND, meaning that all search conditions in the subjects must simultaneously satisfy the logical AND condition.

[0154] "subjects":[ / / Search topics

[0155] {

[0156] "mode":"event", / / Search mode: event element dimension

[0157] "field_plan":{ / / Field requirements

[0158] "hard_require":["events[].subtype|fire"], / / Hard constraint field: The event subtype field in the event feature dimension is fire.

[0159] "priority":["events[].judgement","events[].evidence.desc"] / / The core matching field is the second text description field in the description field and the second evidence field.

[0160] },

[0161] "keywords_groups":[["fire", "open flame", "fire situation", "burning"]] / / The second text description field in the description field or second evidence field includes the keyword groups "fire", "open flame", "fire situation", or "burning".

[0162] },

[0163] {

[0164] "mode":"environment", / / Search mode: environmental element dimension

[0165] "field_plan":{ / / Field requirements

[0166] "hard_require":["environment.weather.value|rain"], / / Hard constraint field: the first specific value in the environment element dimension or rainy day.

[0167] "priority":["environment.weather.basis", "environment.light_env"] / / Core matching fields: third evidence field and lighting field

[0168] },

[0169] "keywords_groups":[["rainy day","rain", "cloudy"]] / / The third evidence field or lighting field contains the keyword groups "rainy day", "rain", and "cloudy".

[0170] },

[0171] {

[0172] "mode":"entity", / / Search mode: entity element dimension

[0173] "field_plan":{ / / Field requirements

[0174] "hard_require": ["entities[].type|building"], / / Hard constraint field: Entity type field or building in the entity feature dimension.

[0175] "priority":["entities[].name","entities[].appearance.content"] / / Core matching fields: content fields in the entity name field and appearance field.

[0176] },

[0177] "keywords_groups":[["warehouse","storeroom","storage room"]] / / Content fields in the entity name or appearance field include the keyword groups "warehouse", "storeroom", and "storage room".

[0178] } ]

[0180] }

[0181] In this invention, by dynamically retrieving hard constraint fields from key values, precise "veto" conditions are provided for subsequent rapid filtering, quickly eliminating the vast majority of irrelevant data and achieving a rapid improvement in retrieval efficiency. Simultaneously, the core matching field, combined with a hierarchical synonym expansion mechanism, is responsible for fine-grained semantic matching and scoring within the filtered candidate set. This ensures high recall through synonym expansion and high accuracy through semantic similarity calculation. Furthermore, the combined design of the retrieval logic and the requirements of each element's dimensional fields enables this invention to accurately express and execute complex composite query intents, achieving a balance between efficiency and accuracy at the architectural level.

[0182] Step S13: Based on the dynamic retrieval key value, perform semantic matching and scoring on each element dimension of the structured semantic description of the image to be retrieved in the semantic database.

[0183] Optionally, step S13 specifically includes:

[0184] In response to the dynamic retrieval key, the dimension elements corresponding to the entity element dimension, event element dimension, and environment element dimension in the structured semantic description of the image to be retrieved in the semantic database are obtained respectively.

[0185] The obtained elements of each dimension are semantically compared with the corresponding first-level extended set in the dynamic search key value to obtain the first-level matching result;

[0186] For the dimensional elements that yield the first-level matching results, the associated pre-selected associated fields are semantically compared with the corresponding second-level extended sets in the dynamic retrieval key values ​​to obtain the second-level matching results;

[0187] Using a pre-defined semantic similarity algorithm, based on the first-level matching results and the second-level matching results, the matching scores of the structured semantic descriptions on each element dimension are calculated respectively.

[0188] When matching with a higher-level concept word, the matching score is determined based on semantic similarity and the confidence decay coefficient corresponding to the dynamic retrieval key value.

[0189] Taking the semantic matching and scoring process of event element dimension as an example. Assuming the user inputs a natural language query text of "find images of a fire," the dynamic retrieval key value information for the event element dimension generated after step S12 is as follows:

[0190] Event element dimension: Fire; First-level extended set: By querying the hierarchical thesaurus for "fire", the superordinate concept words are obtained, forming the first-level extended set as ["fire", "fire", "on fire", "burning"]; Pre-selected associated fields: According to the multi-dimensional structural framework, the pre-selected associated fields related to the event element dimension are events[].judgement (description field); Replacement strategy: Since events[].judgement represents actions and behaviors, a conservative replacement strategy is adopted to extend its core verbs to the upper level; Second-level extended set: The pre-selected associated field events[].judgement is expanded using a conservative replacement strategy to obtain: ["fire", "burning", "dense smoke", "open flame"]; Confidence decay coefficient: 1.0 for the original dimension element, 0.85 for directly hierarchically associated superordinate concept words, and 0.7 for indirectly hierarchically associated superordinate concept words; The structured semantic description of a searchable image stored in the semantic database is as follows (only the part related to the event element dimension is shown):

[0191] json

[0192] {

[0193] "events":[

[0194] {

[0195] "event_type":"other_anomaly",

[0196] "subtype": "fire",

[0197] "judgement": "A serious fire broke out inside the building, with thick smoke billowing out."

[0198] "evidence": [{"desc":"Flames and black smoke spewed from the building's windows","confidence":0.98}]

[0199] } ]

[0201] }

[0202] ① First-level matching: Dimensional element matching

[0203] In response to the dynamic retrieval key, the event element dimension is first obtained from the structured semantic description of the image to be retrieved in the semantic database and from the dynamic retrieval key. Based on the dynamic retrieval key, the field corresponding to the event element dimension is events[].subtype. The value of events[0].subtype is obtained from the structured semantic description of the image to be retrieved as fire.

[0204] The obtained "fire" is semantically compared with the first-level extended set of event element dimensions in the dynamic retrieval key values ​​["fire", "fire", "on fire", "burning"].

[0205] Using a pre-defined semantic similarity algorithm (such as a BERT-based semantic encoding model), the semantic similarity between "fire" and each word in the first-level extended set is calculated: "fire" has a semantic similarity of 0.99 with "fire"; "fire" has a semantic similarity of 0.95 with "fire"; "fire" has a semantic similarity of 0.90 with "fire"; and "fire" has a semantic similarity of 0.88. The word with the highest semantic similarity of 0.99 (the first semantic similarity) is taken as "fire".

[0206] Since "fire" is a raw element (a raw word extracted from the natural language query text), the matching type is raw element matching, and the corresponding confidence decay coefficient is 1.0 (first confidence decay coefficient). Therefore, the first-level matching score = first semantic similarity × first confidence decay coefficient = 0.99 × 1.0 = 0.99.

[0207] ② Second-level matching: Matching of pre-selected related fields

[0208] For event dimension elements that have obtained first-level matching results, their associated pre-selected association fields are matched for second-level matching. Based on the dynamic retrieval key value, the pre-selected association field for the event element dimension is events[].judgement. The value of the pre-selected association field is obtained from the structured semantic description of the image to be retrieved: events[0].judgement="A serious fire occurred inside the building, with thick smoke billowing out".

[0209] Perform a semantic comparison between this field value and the second-level extended set of the event element dimension in the dynamic retrieval key value ["fire", "burning", "dense smoke", "open flame"].

[0210] Calculate the semantic similarity between the description "a serious fire broke out in the building, with thick smoke billowing" and each word in the second-level extended set: Semantic similarity between this description and fire: 0.85; Semantic similarity between this description and burning: 0.90; Semantic similarity between this description and thick smoke: 0.95 (the description contains "thick smoke"); Semantic similarity between this description and open flame: 0.80. Take the highest semantic similarity of 0.95 (the second semantic similarity), which corresponds to the word "thick smoke".

[0211] Since "dense smoke" is an indirect hierarchical related term to the original dimensional element "fire" (fire produces dense smoke), the matching type is indirect hierarchical association matching, and the corresponding confidence decay coefficient is 0.7 (second confidence decay coefficient). Second-level matching score = second semantic similarity × second confidence decay coefficient = 0.95 × 0.7 = 0.665.

[0212] ③ Event element dimension matching score calculation

[0213] Based on the first-level and second-level matching results, a preset weighted formula is used to calculate the matching score of the event element dimension.

[0214] Assume the preset first-level matching weight is 0.6 and the second-level matching weight is 0.4 (the first-level and second-level matching weights can be adjusted according to actual needs).

[0215] Event element dimension matching score = first-level matching score × first-level matching weight + second-level matching score × second-level matching weight = 0.99 × 0.6 + 0.665 × 0.4 = 0.594 + 0.266 = 0.86.

[0216] The matching scores for other element dimensions are calculated as described above, and will not be repeated here.

[0217] Step S14: According to the preset weight fusion strategy, the scores of each element dimension are fused into the final matching score;

[0218] In one optional embodiment of the present invention, such as Figure 1cAs shown, step S14 specifically includes:

[0219] Step S141: Determine the query type of the natural language query text; wherein, the query type includes at least action queries and scenario queries;

[0220] Step S142: In response to the query type, obtain the basic weight allocation strategy corresponding to the query type; wherein, the basic weight allocation strategy includes: for action-type queries, the weight of the event element dimension is higher than that of the entity element dimension and the environment element dimension; for scenario-type queries, the weight of the environment element dimension is higher than that of the entity element dimension and the event element dimension.

[0221] Step S143: Receive the user's custom configuration for the weight of at least one dimension in the basic weight allocation strategy, and form the final weight allocation strategy;

[0222] Step S144: Based on the final weight allocation strategy, the scores of the entity element dimension, event element dimension, and environmental element dimension are weighted and summed to obtain the final matching score.

[0223] The basic weight allocation strategy can be flexibly set by those skilled in the art according to actual needs, and no limitation is made here.

[0224] Step S15: Sort the final matching scores to obtain the sorting results, and return the retrieval result image based on the sorting results.

[0225] In an optional embodiment of the present invention, step S15 specifically includes:

[0226] Step S151: Compare the final matching score with a preset threshold;

[0227] Step S152: Select images whose final matching score is greater than or equal to a preset threshold to form a candidate image set;

[0228] Step S153: Sort the images in the candidate image set in descending order according to their final matching scores to obtain the sorting results, and select the top N images from the sorting results as the retrieval result images to return.

[0229] The preset threshold and N (where N is a positive integer) can be flexibly set by those skilled in the art according to actual needs, and are not limited here.

[0230] It should be noted that steps S151 to S153 are not shown in the figure for ease of explanation only.

[0231] This invention presents an image semantic retrieval method based on a multimodal large model. Firstly, by introducing a three-dimensional "entity-event-environment" framework, it transforms the content of the image to be retrieved into a semantic representation that can be accurately parsed and computed by the machine. This fundamentally solves the problem of fragmented semantic expression in traditional methods, significantly improving the ability to characterize complex semantics and retrieval accuracy. Secondly, the innovative dynamic retrieval key generation and multi-dimensional semantic matching mechanism can dynamically adjust the matching strategy according to the user's query intent. Combined with synonym expansion and confidence decay mechanisms, it effectively controls false positives while ensuring high recall, achieving flexible and accurate semantic retrieval. Furthermore, the configurable weight fusion strategy allows the system to adapt or allows users to define the importance of different dimensions, making the retrieval results more aligned with diverse query needs and greatly improving the user experience.

[0232] Based on the same inventive concept, embodiments of the present invention also provide a multimodal large-model-based image semantic retrieval device for implementing the above-mentioned multimodal large-model-based image semantic retrieval method. The solution provided by this device is similar to the implementation scheme described in the above-described method. Therefore, the specific limitations of one or more embodiments of the multimodal large-model-based image semantic retrieval device provided below can be found in the limitations of the multimodal large-model-based image semantic retrieval method described above, and will not be repeated here.

[0233] like Figure 2 As shown, this invention provides an image semantic retrieval device based on a multimodal large model. The device includes: a structured description generation module 21, a semantic database 22, a query parsing and key-value generation module 23, a multi-dimensional semantic matching and scoring module 24, a weight fusion module 25, and a result sorting and return module 26; wherein,

[0234] The structured description generation module 21 is used to input the image to be retrieved into the multimodal large model, and to generate a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework through the preset prompt word engineering template; wherein, the element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension.

[0235] Semantic database 22, connected to structured description generation module 21, is used to store structured semantic descriptions;

[0236] The query parsing and key value generation module 23 is used to receive the user's natural language query text, perform semantic deconstruction on the natural language query text, extract multiple dimensional elements corresponding to the element dimensions in the multidimensional structure framework, and generate dynamic retrieval key values ​​based on the multiple dimensional elements, including retrieval logic and field requirements.

[0237] The multi-dimensional semantic matching and scoring module 24 is connected to the semantic database 22 and the query parsing and key value generation module 23, respectively. It is used to perform semantic matching and scoring on each element dimension of the structured semantic description of the image to be retrieved in the semantic database based on the dynamic retrieval key value.

[0238] The weight fusion module 25 is connected to the multi-dimensional semantic matching and scoring module 24, and is used to fuse the scores of each element dimension into the final matching score according to the preset weight fusion strategy.

[0239] The result sorting and return module 26, connected to the weight fusion module 25, is used to sort the final matching scores, obtain the sorting results, and return the retrieval result image based on the sorting results.

[0240] Optionally, the structured description generation module 21 is specifically used to: input the image to be retrieved into the multimodal large model, and guide the multimodal large model to perform a structured semantic description of the image to be retrieved according to the entity element dimension, event element dimension, and environment element dimension in the preset multidimensional structural framework through a preset prompt word engineering template; wherein, the prompt word engineering template includes a coordinate specification layer, a content constraint layer, a text processing layer, and a format guarantee layer, which are used to force the output of a structured semantic description that conforms to the preset format and ensure the consistency of references between the various element dimensions.

[0241] Optionally, the query parsing and key-value generation module 23 is specifically used to: receive the user's natural language query text using a multimodal large model, and perform semantic understanding on the natural language query text to extract the dimension elements corresponding to the entity element dimension, event element dimension, and environment element dimension respectively; and generate dynamic retrieval key-values ​​including retrieval logic and field requirements based on the extracted multiple dimension elements; wherein, the field requirements include hard constraint fields and core matching fields.

[0242] Optionally, based on the extracted multiple dimensional elements, a dynamic retrieval key value is generated, including retrieval logic and field requirements. This includes: pre-setting a hierarchical thesaurus; wherein the hierarchical thesaurus includes at least a general semantic layer, a specific domain semantic layer, and a dynamically updated layer, and the hierarchical thesaurus defines a hierarchical mapping relationship between words and one or more superordinate concept words; for the extracted multiple dimensional elements, the hierarchical thesaurus is queried to obtain one or more corresponding superordinate concept words to form a first-level extended set corresponding to each dimensional element; for each dimensional element, according to the multidimensional structural framework, pre-selected associated fields are determined, and a corresponding replacement strategy is selected based on the pre-selected associated fields; based on the replacement strategy selected for each pre-selected associated field, the hierarchical thesaurus is queried to obtain one or more extended words corresponding to each pre-selected associated field to form a second-level extended set; wherein the dynamic retrieval key value includes: the first-level extended set corresponding to each dimensional element, the second-level extended set of the pre-selected associated fields corresponding to each dimensional element, and a confidence decay coefficient preset for extended words at different levels.

[0243] Optionally, the replacement strategy includes: if the pre-selected associated field represents the attribute of the entity, an exact matching strategy is adopted and no expansion is performed; if the pre-selected associated field represents an action, behavior or interaction relationship, a conservative replacement strategy is adopted and its core verb is expanded to the upper level; if the pre-selected associated field represents text content, an aggressive replacement strategy is adopted and the upper level and associated terms are expanded.

[0244] Optionally, the confidence attenuation coefficients preset for extended terms at different levels include: a first attenuation coefficient corresponding to the original dimension element and / or a second attenuation coefficient corresponding to a superordinate concept word with a direct hierarchical relationship and / or a third attenuation coefficient corresponding to a superordinate concept word with an indirect hierarchical relationship; wherein the first attenuation coefficient is greater than the second attenuation coefficient, and the second attenuation coefficient is greater than the third attenuation coefficient.

[0245] Optionally, the multi-dimensional semantic matching and scoring module 24 is specifically used for: responding to the dynamic retrieval key value, obtaining the dimensional elements corresponding to the entity element dimension, event element dimension, and environment element dimension in the structured semantic description of the image to be retrieved in the semantic database and the dynamic retrieval key value; semantically comparing the obtained dimensional elements with the corresponding first-level extended set in the dynamic retrieval key value to obtain the first-level matching result; for the dimensional elements that have obtained the first-level matching result, semantically comparing the associated pre-selected association fields with the corresponding second-level extended set in the dynamic retrieval key value to obtain the second-level matching result; using a preset semantic similarity algorithm, based on the first-level matching result and the second-level matching result, calculating the matching score of the structured semantic description on each element dimension; wherein, when matching with a higher-level concept word, the matching score is determined based on semantic similarity and the confidence decay coefficient corresponding to the dynamic retrieval key value.

[0246] Optionally, the weight fusion module 25 is specifically used to determine the query type to which the natural language query text belongs; wherein, the query type includes at least action-type queries and scenario-type queries; in response to the query type, it obtains the basic weight allocation strategy corresponding to the query type; wherein, the basic weight allocation strategy includes: for action-type queries, the weight of the event element dimension is higher than that of the entity element dimension and the environment element dimension; for scenario-type queries, the weight of the environment element dimension is higher than that of the entity element dimension and the event element dimension; it receives the user's custom configuration of the weight of at least one dimension in the basic weight allocation strategy to form the final weight allocation strategy; and according to the final weight allocation strategy, it performs a weighted summation of the scores of the entity element dimension, the event element dimension, and the environment element dimension to obtain the final matching score.

[0247] Optionally, the result sorting and return module 26 is specifically used to: compare the final matching score with a preset threshold; filter out images whose final matching score is greater than or equal to the preset threshold to form a candidate image set; sort the images in the candidate image set in descending order according to their final matching scores to obtain the sorting result, and select the top N images from the sorting result as the search result images to return.

[0248] The image semantic retrieval device based on a multimodal large model of the present invention firstly transforms the content of the image to be retrieved into a semantic representation that can be accurately parsed and computed by a machine by introducing a three-dimensional structural framework of "entity-event-environment". This fundamentally solves the problem of fragmented semantic expression in traditional methods and significantly improves the ability to characterize complex semantics and the accuracy of retrieval. Secondly, the innovative dynamic retrieval key generation and multi-dimensional semantic matching mechanism can dynamically adjust the matching strategy according to the user's query intent. Combined with synonym expansion and confidence decay mechanisms, it effectively controls false positives while ensuring high recall, achieving flexible and accurate semantic retrieval. In addition, the configurable weight fusion strategy allows the system to adapt or allow users to define the importance of different dimensions, making the retrieval results more suitable for diverse query needs and greatly improving the user experience.

[0249] It should be noted that "multiple" in this invention includes two or more.

[0250] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0251] Each module in the devices of this invention can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0252] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 3As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores data required for or generated by the aforementioned image semantic retrieval method based on a multimodal large model. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements an image semantic retrieval method based on a multimodal large model.

[0253] In one embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 3 As shown, the computer device includes a processor, memory, communication interface, display screen, and input devices connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements an image semantic retrieval method based on a multimodal large model. The display screen can be an LCD screen or an e-ink display screen. The input devices can be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the computer device casing, or an external keyboard, touchpad, or mouse.

[0254] Those skilled in the art will understand that Figure 3 The structure shown is merely a block diagram of a portion of the structure related to the present invention and does not constitute a limitation on the computer device to which the present invention is applied. A specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0255] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0256] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0257] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0258] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties.

[0259] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided by this invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided by this invention may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided by this invention may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0260] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0261] The embodiments described above are merely examples of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention.

Claims

1. An image semantic retrieval method based on a multimodal large model, characterized in that, include: The image to be retrieved is input into the multimodal large model. The multimodal large model generates a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework through the preset prompt word engineering template, and stores the structured semantic description in the semantic database. The element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension. The system receives natural language query text from users, performs semantic deconstruction on the query text, extracts multiple dimensional elements corresponding to the element dimensions in the multidimensional structural framework, and generates dynamic search key values ​​based on these multiple dimensional elements, including search logic and field requirements. This includes: receiving natural language query text from users using a multimodal large model, performing semantic understanding on the query text, and extracting dimensional elements corresponding to entity element dimensions, event element dimensions, and environment element dimensions; generating dynamic search key values ​​based on the extracted multiple dimensional elements, including search logic and field requirements; wherein, the field requirements include hard constraint fields and core matching fields. Based on dynamic retrieval keys, the structured semantic descriptions of the images to be retrieved in the semantic database are semantically matched and scored on each element dimension of the dynamic retrieval keys. This includes: in response to the dynamic retrieval keys, obtaining the dimension elements corresponding to the entity element dimension, event element dimension, and environment element dimension in the structured semantic descriptions of the images to be retrieved in the semantic database; semantically comparing each obtained dimension element with the corresponding first-level extended set in the dynamic retrieval keys to obtain first-level matching results; for the dimension elements that have obtained first-level matching results, semantically comparing their associated pre-selected association fields with the corresponding second-level extended set in the dynamic retrieval keys to obtain second-level matching results; using a preset semantic similarity algorithm, calculating the matching score of the structured semantic description on each element dimension based on the first-level and second-level matching results; wherein, when matching with a higher-level concept word, the matching score is determined based on semantic similarity and the corresponding confidence decay coefficient in the dynamic retrieval keys. According to the preset weight fusion strategy, the scores of each element dimension are merged into the final matching score; The final matching scores are sorted to obtain the sorted results, and the search result images are returned based on the sorted results. The process of generating dynamic search keys based on extracted multiple dimensional elements, including search logic and field requirements, includes: pre-setting a hierarchical thesaurus; wherein the hierarchical thesaurus includes at least a general semantic layer, a specific domain semantic layer, and a dynamically updated layer, and the hierarchical thesaurus defines a hierarchical mapping relationship between words and one or more superordinate concept words; for each extracted multiple dimensional element, querying the hierarchical thesaurus to obtain one or more corresponding superordinate concept words to form a first-level extended set corresponding to each dimensional element; for each dimensional element, determining pre-selected associated fields based on the multi-dimensional structural framework, and selecting a corresponding replacement strategy based on the pre-selected associated fields; based on the replacement strategy selected for each pre-selected associated field, querying the hierarchical thesaurus to obtain one or more extended words corresponding to each pre-selected associated field to form a second-level extended set; wherein the dynamic search keys include: the first-level extended set corresponding to each dimensional element, the second-level extended set of the pre-selected associated fields corresponding to each dimensional element, and a confidence decay coefficient preset for extended words at different levels.

2. The method according to claim 1, characterized in that, The step of inputting the image to be retrieved into a multimodal large model, and using a preset prompt word engineering template to enable the multimodal large model to generate a structured semantic description of the image to be retrieved according to a preset multidimensional structural framework, includes: The image to be retrieved is input into the multimodal big model. The multimodal big model is guided by the preset prompt word engineering template to perform a structured semantic description of the image to be retrieved according to the entity element dimension, event element dimension and environmental element dimension in the preset multidimensional structural framework. The prompt word engineering template includes a coordinate specification layer, a content constraint layer, a text processing layer, and a format guarantee layer, which are used to force the output of structured semantic descriptions that conform to the preset format and ensure the consistency of references between the dimensions of each element.

3. The method according to claim 1, characterized in that, The replacement strategy includes: If the pre-selected associated fields represent the attributes of the entity, an exact matching strategy is adopted, and no expansion is performed; If the pre-selected associated fields represent actions, behaviors, or interaction relationships, a conservative replacement strategy is adopted, and the core verbs are expanded to higher levels. If the pre-selected associated fields represent text content, an aggressive replacement strategy is adopted to expand upwards and to related terms.

4. The method according to claim 1, characterized in that, The confidence attenuation coefficients preset for extended terms at different levels include: a first attenuation coefficient corresponding to the original dimension element and / or a second attenuation coefficient corresponding to a superordinate concept word with a direct hierarchical relationship and / or a third attenuation coefficient corresponding to a superordinate concept word with an indirect hierarchical relationship; wherein, the first attenuation coefficient is greater than the second attenuation coefficient, and the second attenuation coefficient is greater than the third attenuation coefficient.

5. The method according to claim 1, characterized in that, The step of fusing the scores of each element dimension into a final matching score according to a preset weight fusion strategy includes: Determine the query type of the natural language query text; the query type should include at least action queries and scenario queries. In response to the query type, obtain the basic weight allocation strategy corresponding to the query type; the basic weight allocation strategy includes: for action-type queries, the weight of the event element dimension is higher than that of the entity element dimension and the environment element dimension; for scenario-type queries, the weight of the environment element dimension is higher than that of the entity element dimension and the event element dimension. Receive user-defined configurations for the weights of at least one dimension in the basic weight allocation strategy, and form the final weight allocation strategy; Based on the final weighting strategy, the scores of the entity element dimension, event element dimension, and environmental element dimension are weighted and summed to obtain the final matching score.

6. The method according to claim 1, characterized in that, The process of sorting the final matching scores to obtain a sorting result, and returning the retrieval result image based on the sorting result, includes: The final match score is compared with a preset threshold. Images with a final matching score greater than or equal to a preset threshold are selected to form a candidate image set. The images in the candidate image set are sorted in descending order according to their final matching scores to obtain the sorting results. The top N images in the sorting results are then selected as the search result images and returned.

7. An image semantic retrieval device based on a multimodal large model, characterized in that, include: The structured description generation module is used to input the image to be retrieved into the multimodal large model, and to generate a structured semantic description of the image to be retrieved according to the preset multidimensional structural framework through the preset prompt word engineering template; wherein, the element dimensions in the multidimensional structural framework include at least: entity element dimension, event element dimension and environment element dimension. A semantic database, connected to the structured description generation module, is used to store structured semantic descriptions; The query parsing and key-value generation module receives the user's natural language query text, performs semantic deconstruction on the natural language query text, extracts multiple dimensional elements corresponding to the element dimensions in the multidimensional structural framework, and generates dynamic search key-values ​​that include retrieval logic and field requirements based on the multiple dimensional elements. This includes: receiving the user's natural language query text using a multimodal large model, performing semantic understanding on the natural language query text, and extracting dimensional elements corresponding to entity element dimensions, event element dimensions, and environment element dimensions respectively; generating dynamic search key-values ​​that include retrieval logic and field requirements based on the extracted multiple dimensional elements; wherein, the field requirements include hard constraint fields and core matching fields. A multi-dimensional semantic matching and scoring module, connected to both the semantic database and the query parsing and key-value generation module, performs semantic matching and scoring on each element dimension of the structured semantic description of the image to be retrieved in the semantic database based on dynamic retrieval keys. This includes: responding to dynamic retrieval keys, acquiring the dimensional elements corresponding to the entity element dimension, event element dimension, and environment element dimension in the structured semantic description of the image to be retrieved in the semantic database; semantically comparing each acquired dimensional element with the corresponding first-level extended set in the dynamic retrieval key to obtain a first-level matching result; for the dimensional elements that have obtained the first-level matching result, semantically comparing their associated pre-selected association fields with the corresponding second-level extended set in the dynamic retrieval key to obtain a second-level matching result; and using a preset semantic similarity algorithm, calculating the matching score of the structured semantic description on each element dimension based on the first-level and second-level matching results. When matching with a higher-level concept word, the matching score is determined based on semantic similarity and the corresponding confidence decay coefficient in the dynamic retrieval key. The weight fusion module, connected to the multi-dimensional semantic matching and scoring module, is used to fuse the scores of each element dimension into the final matching score according to the preset weight fusion strategy. The result sorting and return module, connected to the weight fusion module, is used to sort the final matching scores, obtain the sorted results, and return the search result image based on the sorted results. The process of generating dynamic search keys based on extracted multiple dimensional elements, including search logic and field requirements, includes: pre-setting a hierarchical thesaurus; wherein the hierarchical thesaurus includes at least a general semantic layer, a specific domain semantic layer, and a dynamically updated layer, and the hierarchical thesaurus defines a hierarchical mapping relationship between words and one or more superordinate concept words; for each extracted multiple dimensional element, querying the hierarchical thesaurus to obtain one or more corresponding superordinate concept words to form a first-level extended set corresponding to each dimensional element; for each dimensional element, determining pre-selected associated fields based on the multi-dimensional structural framework, and selecting a corresponding replacement strategy based on the pre-selected associated fields; based on the replacement strategy selected for each pre-selected associated field, querying the hierarchical thesaurus to obtain one or more extended words corresponding to each pre-selected associated field to form a second-level extended set; wherein the dynamic search keys include: the first-level extended set corresponding to each dimensional element, the second-level extended set of the pre-selected associated fields corresponding to each dimensional element, and a confidence decay coefficient preset for extended words at different levels.

Images