Multimodal data query method and system

Abstract

The application provides a multi-modal data query method and system, which comprises the following steps: cutting a multi-modal original object into minimum controlled sub-objects; generating a corresponding joint signature according to a uniform boundary vector, a permission tuple and a version generation identifier corresponding to each minimum controlled sub-object; establishing a prefix security routing bucket; the joint signatures of each minimum controlled sub-object corresponding to each prefix security routing bucket meet a preset proximity condition; constructing a restricted semantic graph with each minimum controlled sub-object as a node based on the semantic vectors corresponding to the minimum controlled sub-objects; generating a user capability signature according to a query text, a query boundary and a user permission context in a query request; determining a prefix security routing bucket matched with the user capability signature as a target routing bucket; and traversing nodes meeting a legality verification condition on the restricted semantic graph from an entry node in the target routing bucket, and outputting a query result. The method can reduce the computing cost.

Description

Technical Field

[0001] This application relates to the field of information retrieval technology, and in particular to a multimodal data query method and system. Background Technology

[0002] With the widespread deployment of enterprise-level large-scale model applications, retrieval enhancement generation systems, intelligent question answering systems, audio and video auditing platforms, and industrial intelligent analysis platforms, multimodal data such as images, videos, audio, documents, tables, and structured records are gradually being unified into an AI foundation for vectorized indexing and semantic retrieval. In this process, the system typically needs to support unified cross-modal retrieval and relevant result recall.

[0003] Traditional methods often employ a "retrieve first, filter later" approach. This involves first constructing a query vector based on user input to recall a batch of candidate objects; then filtering these candidates to remove those for which the user has no access rights. This approach can easily lead to unnecessary computational overhead. For multimodal large-scale indexes, recalling a large number of objects first and then discarding those for which the user has no access rights results in the filtered-out objects still consuming significant retrieval and sorting computational resources, leading to clearly inefficient computational overhead.

[0004] Therefore, there is an urgent need to provide a new multimodal secure retrieval technology solution to reduce unnecessary computational overhead during the query and retrieval process. Summary of the Invention

[0005] To address the existing technical problems, this application provides a multimodal data query method and a multimodal data query system.

[0006] Firstly, this application provides a multimodal data query method, including: Divide the multimodal primitive object into multiple minimal controlled sub-objects; Generate a joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each smallest controlled sub-object; Establish at least one prefix secure routing bucket; the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix secure routing bucket satisfies a preset proximity condition; Based on the semantic vectors corresponding to the smallest controlled sub-objects, a restricted semantic graph is constructed with each smallest controlled sub-object as a node; Obtain the query request, and generate a user capability signature based on the query text, query boundaries, and user permission context in the query request; Identify the prefix security route bucket that matches the user's capability signature as the target route bucket; Starting from the entry node in the target routing bucket, traverse the nodes that satisfy the validity check conditions on the restricted semantic graph and output the query results; the entry node refers to the representative node used to start the graph traversal.

[0007] In one embodiment, the multimodal original object includes at least two of images, videos, audio, or documents, and the smallest controlled sub-object includes at least two of image regions, video clips, audio clips, or document clips; wherein, the image region includes at least one of a target bounding box, an instance region, or a layout region in an image; the video clip includes at least one of a shot clip, a time-sliding window clip, or an event clip in a video; the audio clip includes at least one of a speaker clip, a speech activity segment, or an anomalous acoustic segment in audio; and the document clip includes at least one of a layout block, a table cell, a paragraph segment, or a character span in a document.

[0008] In one embodiment, the dimensions of the unified boundary vectors corresponding to different minimum controlled sub-objects are the same; each unified boundary vector corresponding to a minimum controlled sub-object includes boundary information slots corresponding to various sub-object types and valid marker slots; each boundary information slot is used to record the boundary information of the minimum controlled sub-object under the corresponding sub-object type; the element value in the valid marker slot is used to identify the boundary information slot that is valid for the minimum controlled sub-object. The permission tuple corresponding to each smallest controlled sub-object includes at least one of the following: the tenant or business domain corresponding to the smallest controlled sub-object, the role bitmap with access permissions, the sensitivity level, the effective time interval, or the allowed operation type; Version generation identifier is used to represent the version to which the smallest controlled sub-object belongs; the version generation identifier changes as the smallest controlled sub-object changes in access permissions, boundaries, or content.

[0009] In one embodiment, a corresponding joint signature is generated based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each least controlled sub-object, including: The unified boundary vector, permission tuple, and version generation identifier are encoded, and the resulting boundary encoding result, permission encoding result, and generation encoding result are jointly mapped to the unified signature space to obtain the joint projection vector. Generate a corresponding deterministic jitter term for each component in the joint projection vector; By combining the deterministic jitter term with the joint projection vector for signature mapping, the corresponding joint signature is obtained.

[0010] In one embodiment, establishing at least one prefix security routing bucket includes: For each smallest controlled sub-object, the first preset number of bits of the joint signature is used as the routing key; A corresponding prefix security route bucket is established based on each routing key; the prefix security route bucket includes at least one of the following: a set of entry nodes, a set of object identifiers in the bucket, a union of role bitmaps in the bucket, an upper bound of sensitivity level in the bucket, a summary of generational distribution in the bucket, or bucket statistics. The entry node set represents the set of representative nodes used for startup graph traversal in the prefix secure routing bucket; the bucket object identifier set represents the set of identifiers of the smallest controlled sub-objects in the prefix secure routing bucket; the bucket role bitmap union represents the union of the role bitmaps of each smallest controlled sub-object in the prefix secure routing bucket; the bucket generation distribution summary represents the number distribution, dominant generation, smallest generation, and largest generation of each version generation in the prefix secure routing bucket; and the bucket statistics include the total number of objects, update time distribution, and sensitivity level distribution.

[0011] In one embodiment, a restricted semantic graph is constructed using the semantic vectors corresponding to the smallest controlled sub-objects as nodes, including: Using each smallest controlled sub-object as a node, edges are constructed between nodes to obtain an initial restricted semantic graph; Calculate the semantic similarity between the semantic vectors of the two nodes to which each edge belongs in the initial restricted semantic graph; The weight of each edge is determined based on the semantic similarity, boundary continuity, signature compatibility, and generation interval between the two nodes to which each edge belongs in the initial restricted semantic graph. For each node in the initial restricted semantic graph, retain the edges whose weights are ranked first by a preset order to obtain the restricted semantic graph.

[0012] In one embodiment, a user capability signature is generated based on the query text, query boundaries, and user permission context in the query request, including: Based on the query text, query boundaries, and user permission context in the query request, the query version identifier, query boundary encoding result, and user permission encoding result are obtained. A user capability signature is generated by jointly projecting the query version identifier, query boundary encoding result, and user permission encoding result.

[0013] In one embodiment, starting from the entry node in the target routing bucket, the system traverses the restricted semantic graph to find nodes that satisfy the validity check conditions, and outputs the query results, including: Take the entry node in the target routing bucket as the current node to be expanded, and traverse the adjacent nodes of the node to be expanded on the restricted semantic graph as the nodes to be visited. Perform a validity check on the node to be accessed; The node that passes the validity check is taken as the new node to be expanded. The adjacent nodes of the node to be expanded are returned to perform traversal on the restricted semantic graph to continue the traversal and obtain a set of candidate sub-objects. The set of candidate sub-objects includes the smallest controlled sub-object of each candidate that passes the validity check. Output query results based on the set of candidate sub-objects.

[0014] In one embodiment, the query results are output based on the candidate sub-object set, including: For each candidate minimum controlled sub-object in the candidate sub-object set, a corresponding risk quantification value is generated based on the semantic similarity, boundary consistency quantification value, version consistency quantification value, and risk penalty item. Here, semantic similarity refers to the similarity between the semantic vector of the minimum controlled sub-object and the query semantic vector corresponding to the query text; boundary consistency quantification value represents the consistency between the object boundary of the minimum controlled sub-object and the query boundary; and version consistency quantification value represents the consistency between the version generation of the minimum controlled sub-object and the query version corresponding to the query text. Remove the smallest controlled sub-object with a risk quantification value lower than a preset threshold from the candidate sub-object set, and output the remaining smallest controlled sub-object as the query result.

[0015] Secondly, a multimodal data query system is provided, the system comprising: The segmentation module is used to segment the multimodal original object into multiple minimal controlled sub-objects; The routing bucket creation module is used to generate a corresponding joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each smallest controlled sub-object; establish at least one prefix security routing bucket; and ensure that the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix security routing bucket meets a preset proximity condition. The semantic graph construction module is used to construct a restricted semantic graph based on the semantic vectors corresponding to the smallest controlled sub-objects, with each smallest controlled sub-object as a node; The query module is used to obtain query requests, generate user capability signatures based on query text, query boundaries, and user permission context in the query request; determine the prefix security routing bucket that matches the user capability signature as the target routing bucket; starting from the entry node in the target routing bucket, traverse the nodes that meet the legality verification conditions on the restricted semantic graph, and output the query results; the entry node refers to the representative node used to start the graph traversal.

[0016] Thirdly, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the above-described multimodal data query method.

[0017] Fourthly, a computer device is provided, including a processor and a memory connected to the processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed by the processor, it implements the steps of the multimodal data query method described above.

[0018] Fifthly, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the multimodal data query method described above.

[0019] The multimodal data query method, system, computer program product, computer device, and computer-readable storage medium provided in the above embodiments divide the multimodal original object into multiple minimal controlled sub-objects; generate a corresponding joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each minimal controlled sub-object; establish at least one prefix secure routing bucket; ensure that the first preset number of bits of the joint signature of each minimal controlled sub-object corresponding to each prefix secure routing bucket meets a preset proximity condition; construct a restricted semantic graph with each minimal controlled sub-object as a node based on the semantic vector corresponding to each minimal controlled sub-object; obtain a query request, and generate a user capability signature based on the query text, query boundary, and user permission context in the query request; determine the prefix secure routing bucket that matches the user capability signature as the target routing bucket; traverse the nodes that meet the legality verification conditions on the restricted semantic graph starting from the entry node in the target routing bucket, and output the query results; the entry node refers to the representative node used to initiate graph traversal. That is, based on the constructed prefix security routing bucket, the search scope can be initially narrowed during the query phase. Furthermore, based on the restricted semantic graph, only nodes that have passed the legality verification can be expanded accurately and conveniently. In this way, the nodes traversed and queried (i.e. the smallest controlled sub-objects) are already objects that have permission to access, avoiding queries on unauthorized objects and subsequent filtering processing, thereby reducing unnecessary consumption of computing resources. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a multimodal data query method in one embodiment; Figure 2 This is a flowchart illustrating the steps involved in constructing a restricted semantic graph in one embodiment. Figure 3 This is a simplified flowchart of a multimodal data query method in one embodiment; Figure 4 This is a structural block diagram of a multimodal data query system in one embodiment; Figure 5 This is a structural block diagram of a computer device in one embodiment. Detailed Implementation

[0021] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be considered as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0022] In the following description, the phrase "some embodiments" refers to a subset of all possible embodiments. It should be noted that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0023] The traditional "search first, filter later" approach not only incurs unnecessary computational overhead but also increases the risk of sensitive information leakage. This is because during the candidate retrieval phase, unauthorized objects (those without access privileges) have already entered the candidate pool, cache, intermediate sorting structures, and even the logging process, creating an intermediate exposure window. Even if unauthorized objects are ultimately filtered out, the system may still have performed feature reading, distance calculation, graph access, or intermediate state persistence on them, further increasing the risk of sensitive information leakage.

[0024] Based on this, this application proposes a multimodal data query method. This method combines prefix secure routing buckets with restricted semantic graph traversal to suppress unauthorized objects from entering the candidate pool, thus narrowing the intermediate exposure window from the source and reducing unnecessary computational overhead. The multimodal data query method will be described in more detail below.

[0025] like Figure 1 As shown, in some embodiments, a multimodal data query method is provided, which can be executed by a computer device, and specifically includes the following steps: S11 divides the multimodal primitive object into multiple minimal controlled sub-objects.

[0026] A multimodal raw object is the original multimodal data available for querying and retrieval. In some embodiments, a multimodal raw object includes at least two of the following: images, videos, audio, or documents.

[0027] The smallest controlled sub-object refers to the smallest data unit that is no longer further subdivided and is subject to security controls. For example, the smallest controlled sub-object includes at least one of an image region, a video clip, an audio clip, or a document clip.

[0028] The image region includes at least one of the following: a bounding box, an instance region, or a layout region in an image; the video segment includes at least one of the following: a shot segment, a time-sliding window segment, or an event segment in a video; the audio segment includes at least one of the following: a speaker segment, a speech activity segment, or an anomalous acoustic segment in audio; and the document segment includes at least one of the following: a layout block, a table cell, a paragraph segment, a heading block, a footnote block, or a character span in a document. It should be understood that traditional access control schemes are mostly built on document-level, record-level, or field-level objects, and rarely support finer-grained, smallest security units such as local regions in images, time segments in videos, speech segments in audio, and paragraph blocks, table cells, and character spans in documents.

[0029] Specifically, during the offline database building phase, the system first accesses multimodal raw objects such as images, videos, audio, and documents, and calls the corresponding segmenter according to the object type to obtain multiple minimal controlled sub-objects.

[0030] For image objects, object detectors, instance segmenters, or layout analyzers can be used to identify bounding boxes, region blocks, or structural blocks in the image, and each identified image region can be considered as a minimal controlled sub-object. For video objects, shot segmentation can be performed first, and then each shot can be segmented according to fixed-length sliding windows, event boundaries, or keyframe changes, with each segmented video segment considered as a minimal controlled sub-object. For audio objects, speech activity detection, speaker separation, or voiceprint clustering can be performed first, and then multiple audio segments can be segmented according to the speaker or a fixed time window as minimal controlled sub-objects. For document objects, layout analysis, paragraph parsing, table structure recognition, or OCR (Optical Character Recognition) localization can be performed, with layout blocks, paragraph segments, table cells, heading blocks, footnote blocks, or character spans considered as minimal controlled sub-objects.

[0031] To facilitate unified processing, each smallest controlled sub-object of different modalities is assigned a unique object identifier. (That is, any smallest controlled sub-object i or the i-th smallest controlled sub-object corresponds to a unique...) ), and retain the mapping relationship with the source object.

[0032] S12, generate the corresponding joint signature based on the unified boundary vector, permission tuple and version generation identifier corresponding to each smallest controlled sub-object.

[0033] In some embodiments, a unified boundary vector is used to characterize the boundary features of the smallest controlled sub-object. The unified boundary vector is equivalent to a unified structured vector capable of simultaneously expressing temporal boundaries, spatial boundaries, hierarchical positions, and text offset positions. It should be understood that different modalities of data are expressed in inconsistent ways regarding temporal, spatial, and offset boundaries, which increases the difficulty of implementing unified secure retrieval / query. To address this issue, this application defines a unified boundary vector to uniformly express the boundary information of different modal objects. Thus, the unified boundary vectors corresponding to different smallest controlled sub-objects have the same dimension; each unified boundary vector corresponding to a smallest controlled sub-object includes boundary information slots corresponding to various sub-object types, as well as valid marker slots; each boundary information slot records the boundary information of the smallest controlled sub-object under the corresponding sub-object type; the element value in the valid marker slot is used to identify the boundary information slot that is valid for the smallest controlled sub-object.

[0034] For example, the uniform boundary vector of the i-th (i is any positive integer) smallest controlled sub-object as follows: ; in, and These indicate the start and end positions of time (applicable to video and audio clips, and can be used to mark the start and end positions of time for video or audio clips). Indicates spatial location and regional scale (used to mark image areas and document page blocks or page areas). Indicates the level depth (used to indicate the document hierarchy or the nesting depth of objects in the document). and Indicates text or structured offset positions (e.g., text offset positions or character span positions within a table). These are all boundary information slots.

[0035] This represents a null mask, equivalent to a valid marker slot, used to identify which dimensions in the uniform boundary vector are valid for the current object. For example, for a video clip, the smallest controlled sub-object, the time start and end dimensions in the uniform boundary vector... and Effective, and text offset dimension and Generally ineffective; for the smallest controlled sub-object of the document's character span, the text offset dimension and Effective, spatial width and height dimensions and Whether it is valid depends on whether there is a layout positioning result. The mask position corresponding to the valid dimension can be recorded as 0, and the mask position corresponding to the invalid dimension can be recorded as 1, so that the invalid dimension can be skipped in subsequent comparison and encoding.

[0036] In some embodiments, each least controlled sub-object corresponds to or is associated with a permission tuple, which includes at least one of the following: the tenant or business domain identifier corresponding to the least controlled sub-object, a role bitmap with access permissions, a sensitivity level, a valid time interval, or a permitted operation type. For example, the permission tuple associated with the i-th least controlled sub-object... as follows: in, This represents the tenant identifier or business domain identifier corresponding to the i-th smallest controlled sub-object. This is a bitmap of roles that have access permissions to the i-th smallest controlled sub-object. The sensitivity level (also known as the sensitivity grade) is the sensitivity level of the i-th smallest controlled sub-object. and The effective time interval corresponding to the i-th smallest controlled sub-object. This specifies the type of operation allowed to be performed on the i-th smallest controlled sub-object. For example, the role bitmap can support composite permission factors such as department, position, project, and security level.

[0037] In some embodiments, a version generation identifier is used to characterize the version to which the smallest controlled sub-object belongs; the version generation identifier changes as the smallest controlled sub-object's access permissions, boundaries, or content change. It is important to emphasize that the "version generation" in this application embodiment refers to the evolutionary sequence formed when the permissions, boundaries, or content of the same smallest controlled sub-object change, rather than a general version number of the entire original document. For example, a paragraph fragment may correspond to version generation 3 when it is first added to the database; if its access permissions are changed from "visible to project team" to "visible to legal department," then the version generation changes to a new version generation 4.

[0038] Furthermore, a corresponding joint signature can be generated based on the semantic vector, unified boundary vector, permission tuple, and version generation identifier corresponding to each smallest controlled sub-object.

[0039] Specifically, a joint projection vector can be generated based on the semantic vector, unified boundary vector, permission tuple, and version generation identifier corresponding to each smallest controlled sub-object; a corresponding deterministic jitter term can be generated for each component in the joint projection vector; and a signature mapping can be performed on the joint projection vector in combination with the deterministic jitter term to obtain the corresponding joint signature.

[0040] In some embodiments, the unified boundary vector, the permission tuple, and the version generation identifier are encoded separately to obtain boundary encoding results, permission encoding results, and generation encoding results. Then, the boundary encoding results, permission encoding results, and generation encoding results are jointly mapped to the unified signature space to obtain a joint projection vector.

[0041] In other embodiments, the unified boundary vector and the permission tuple can be encoded separately to obtain the boundary encoding result and the permission encoding result. Then, the boundary encoding result, the permission encoding result, and the version generation identifier are jointly mapped to the unified signature space to obtain the joint projection vector.

[0042] In some examples, the quantization step size is determined based on the upper and lower bounds of the element values ​​of all minimal controlled sub-objects in each dimension of the unified boundary vector. Then, normalization is performed on each dimension of the unified boundary vector based on the quantization step size to obtain the boundary encoding result. For example, boundary quantization can be performed using the following formula: in, This represents the original element value of the uniform boundary vector corresponding to the i-th smallest controlled sub-object in the n-th dimension. This represents the quantized result of the element values ​​of the i-th smallest controlled sub-object in the n-th dimension. and Let these represent the upper and lower bounds obtained by statistically analyzing the elements of all minimal controlled sub-objects in the nth dimension, respectively. Let n be the quantization step size of the nth dimension.

[0043] In some examples, the boundary coding result can be obtained by performing discrete quantization on the normalization result of each dimension in the unified boundary vector by combining the quantization levels of each dimension.

[0044] For example, the formula for normalizing each dimension in the uniform boundary vector is as follows: in, This represents the normalized result (i.e., the normalized element value) of the unified boundary vector corresponding to the i-th smallest controlled sub-object in the n-th dimension. For smoothing terms. For example, the upper bound... and the lower realm The maximum and minimum values ​​can be used, or robust statistical values ​​after quantile truncation can be used; there is no restriction on which one is used.

[0045] The formula for performing discrete quantization on the normalization result is as follows: in, For the first The quantization series of the dimension, This represents the normalization result for the nth dimension. The result after discretization. It should be understood that discretizing all dimensions yields the boundary coding vector (i.e., the boundary coding result), denoted as... .

[0046] In some examples, the encoding process for permission tuples includes mapping the discrete fields of each dimension in the permission tuple (including tenant or business domain, role bitmap, sensitivity level, effective time interval or operation type) to the embedding space, normalizing them, mapping them to the numerical encoding space, and then concatenating them to obtain the permission encoding result.

[0047] For example, the specific formula is as follows: ; in, These are the element values ​​(i.e., discrete fields) under each dimension of the permission tuple. For their specific meanings, please refer to the previous text, which will not be repeated here. These represent the encoding mapping functions corresponding to tenant, role, sensitivity level, time interval, and operation type, respectively, used to map discrete fields to the embedding space, and after normalization, to the numerical encoding space; Concat() represents concatenation or merging. Represents the permission tuple of the i-th smallest controlled sub-object. The permission encoding result obtained after encoding.

[0048] In some examples, the joint projection vector is calculated as follows: ; in, This represents the joint projection vector corresponding to the i-th smallest controlled sub-object. This represents the boundary encoding result corresponding to the i-th smallest controlled sub-object. The boundary encoding mapping result obtained by mapping, ψ(p_i) represents the permission encoding result obtained after encoding the permission tuple p_i corresponding to the i-th smallest controlled sub-object. This represents the generation version identifier corresponding to the i-th smallest controlled sub-object. The generational coding results obtained after coding mapping , and This is the projection matrix (e.g., a fixed random matrix, a learned projection matrix, or a grouped projection matrix). This is the bias vector.

[0049] In some embodiments, to reduce frequent signature flips caused by slight perturbations near the quantization boundary, a stable band sign mapping function is introduced: Where δ is the stable band threshold. A deterministic jitter term is further introduced: in, For hash mapping functions; This is a preset jitter amplitude coefficient used to reduce frequent bucket migrations caused by boundary perturbations or permission fine-tuning. The object identifier representing the i-th smallest controlled sub-object; It is the generation version identifier of the i-th smallest controlled sub-object; the value of k ranges from 1 to m, where m represents the joint projection vector. The dimension; Represents the joint projection vector The kth component The corresponding deterministic jitter term.

[0050] Finally, based on the stable band sign mapping function, and combining the joint projection vector and the deterministic jitter terms corresponding to each component, a joint signature is generated, as shown in the following formula: ; in, () represents the stable band sign mapping function; joint projection vector Each component, combined with its corresponding deterministic jitter term, can serve as a stable band-signed mapping function. The input of () is used as the output of the stable signed mapping function as the joint signature corresponding to the i-th smallest controlled sub-object. One of the components. For example, This indicates that the joint projection vector will be used. The first component With the corresponding deterministic jitter term Combine, as Input of (); This indicates that the joint projection vector will be used. The second component With the corresponding deterministic jitter term Combine, as Input of (); Joint projection vector The m-th component (i.e., the last component) and its corresponding deterministic jitter term Combine, as The input is in parentheses (). In this way, the complete joint signature corresponding to the i-th smallest controlled sub-object can be obtained. This joint signature takes into account three types of information: boundary, permission, and generation, and can serve as a secure routing identifier for objects. Furthermore, the stable-band joint signature algorithm introduces a stable-band threshold and a deterministic jitter term on top of the joint projection to reduce frequent signature flipping caused by boundary perturbations, permission fine-tuning, and floating-point noise, thereby improving the stability of the secure routing bucket.

[0051] S13, establish at least one prefix secure routing bucket; the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix secure routing bucket satisfies the preset proximity condition.

[0052] Specifically, the first preset number of digits of the joint signature of each smallest controlled sub-object is taken as the routing key; a corresponding prefix secure routing bucket is established based on each routing key.

[0053] For example, the joint signature of each minimal controlled sub-object Take before The bit, used as the routing key, has the following formula: ;in, That is, the routing key corresponding to the i-th smallest controlled sub-object, where L is any positive integer.

[0054] It should be understood that the first L bits of the joint signature of multiple minimal controlled sub-objects may be consistent, therefore, one routing key may correspond to multiple minimal controlled sub-objects. Thus, the routing key... Prefix-secure routing buckets can be established. A prefix-secure routing bucket refers to a routing bucket formed by multiple smallest controlled sub-objects whose combined signatures have the same prefix (e.g., the first L bits are the same). It is equivalent to a routing index structure built with the combined signature prefix as the key. In other words, when the first L bits of the combined signatures of multiple smallest controlled sub-objects are the same or meet the set prefix nearest neighbor matching conditions, they can all fall into the same prefix-secure routing bucket. Different prefix-secure routing buckets can be distributed and stored in multiple index shards.

[0055] The prefix secure routing bucket includes information corresponding to multiple minimal controlled sub-objects with consistent prefixes in the joint signature. For example, the prefix secure routing bucket includes at least one of the following: a set of entry nodes, a set of object identifiers within the bucket, a union of role bitmaps within the bucket, an upper bound of sensitivity levels within the bucket, a summary of generational distribution within the bucket, or bucket statistics.

[0056] The entry node set represents the set of representative nodes (i.e., entry nodes) in the prefix safe routing bucket used to initiate restricted semantic graph traversal processing. Entry nodes can be several central nodes within the bucket, highly connected nodes, or the most recently updated node. A node is the smallest controlled sub-object.

[0057] The bucket-internal object identifier set represents the set of identifiers for all minimal controlled sub-objects within the prefix-secure routing bucket. For example, if the first L bits of the joint signature of 10 minimal controlled sub-objects are identical, these first L bits form the routing key, and the prefix-secure routing bucket corresponding to this routing key includes the identifiers of these 10 minimal controlled sub-objects. It should be understood that the set of entry nodes is typically a proper subset of the nodes corresponding to the bucket-internal object identifier set.

[0058] The union of role bitmaps within a bucket represents the union of the role bitmaps of the smallest controlled sub-objects in the bucket for prefix security routing. It should be understood that each smallest controlled sub-object has a corresponding role bitmap, and the union of the role bitmaps of all the smallest controlled sub-objects in the bucket can be obtained (for example, by performing a bitwise OR operation on the role bitmaps of each smallest controlled sub-object, the union can be obtained).

[0059] The bucket generation distribution summary is used to represent at least one of the following: the distribution of the number of version generations, the dominant generation, the minimum generation, or the maximum generation in the prefix security routing bucket. It should be understood that each minimum controlled sub-object has a corresponding version generation, and the dominant generation refers to the version generation with the largest number in the bucket (i.e., the number of corresponding minimum controlled sub-objects).

[0060] Bucket statistics include at least one of the following: total number of objects, average edge degree, update time distribution, or sensitivity level distribution. The total number of objects refers to the total number of the smallest controlled sub-objects within the bucket. Average edge degree characterizes the average density of relationships between the smallest controlled sub-objects within the bucket. A high average edge degree indicates complex and tightly connected relationships between the smallest controlled sub-objects; a low average edge degree indicates relatively independent and sparsely connected relationships between the smallest controlled sub-objects. Update time distribution or sensitivity level distribution refers to the distribution of update times or sensitivity levels corresponding to the smallest controlled sub-objects within the bucket, respectively.

[0061] S14. Based on the semantic vectors corresponding to the smallest controlled sub-objects, construct a restricted semantic graph with each smallest controlled sub-object as a node.

[0062] Among them, a restricted semantic graph refers to a semantic graph structure in which nodes and edges are subject to permission compatibility constraints. The weight of each edge in a restricted semantic graph is determined based on the semantic similarity, boundary continuity, signature compatibility, and generation interval between the two nodes to which it belongs. This ensures that the nearest neighbor relationships between objects not only reflect content relevance but also authorization compatibility and version proximity.

[0063] For example, a restricted semantic graph can be a nearest neighbor graph, a hierarchical navigation graph, or a sparse graph structure.

[0064] In some embodiments, the semantic vector corresponding to the smallest controlled sub-object is generated by an encoder corresponding to the modality to which the smallest controlled sub-object belongs.

[0065] For example, the initial semantic vector generated by the encoder can be length-normalized (e.g., L2 normalization) to obtain the final semantic vector corresponding to the smallest controlled sub-object. For example, L2 normalization can be performed using the following formula: in, It is the initial semantic vector. It is the final semantic vector after normalization.

[0066] For example, image clips, video clips, audio clips, and text clips can be encoded using a visual encoder, a temporal visual encoder, an acoustic encoder, and a text encoder, respectively, or they can be encoded using a unified multimodal encoder.

[0067] In some embodiments, such as Figure 2 As shown, step S14 (referred to as the construction step of the restricted semantic graph) includes S141 to S144: S141, using each smallest controlled sub-object as a node, and constructing edges between nodes, to obtain the initial restricted semantic graph.

[0068] S142, calculate the semantic similarity between the semantic vectors of the two nodes to which each edge belongs in the initial restricted semantic graph.

[0069] S143. Based on the semantic similarity, boundary continuity, signature compatibility and generation interval between the two nodes to which each edge belongs in the initial restricted semantic graph, determine the weight of each edge.

[0070] Specifically, for any node i (i.e., the i-th smallest controlled sub-object or the smallest controlled sub-object i) and node j (i.e., the j-th smallest controlled sub-object or the smallest controlled sub-object j) in the initial restricted semantic graph, the weight of the edge formed by these two nodes is calculated using the following formula: in, The semantic vector representing node i semantic vector of node j Semantic similarity between them; Represents the uniform boundary vector of node i The unified boundary vector of node j Boundary continuity between them; Represents the joint signature of node i The signature compatibility between the joint signature of node j and node j; This is the generation interval, used to characterize the version generation identifier of node i. Version generation identifier of node j The difference interval between them; α, γ These are the preset weighting coefficients.

[0071] In some examples, different algorithms can be used to calculate boundary continuity for different modal types. Specifically, for two nodes belonging to an image region, the intersection-union ratio or center distance can be used to calculate their boundary continuity; for two nodes belonging to a video or audio segment, the temporal overlap rate or temporal adjacency can be used to calculate their boundary continuity; for two nodes belonging to a text span, the offset adjacency, paragraph-level adjacency, or character coverage can be used to calculate their boundary continuity.

[0072] S144. For each node in the initial restricted semantic graph, retain the edges whose weights are in the first preset order to obtain the restricted semantic graph.

[0073] For example, each node only retains the top-weighted nodes. Edges are used to control the sparsity and traversal complexity of the restricted semantic graph. This results in the restricted semantic graph. V is the set of nodes, and E is the set of edges.

[0074] It should be understood that steps S11 to S14 can be understood as preparatory processing before the online query stage, such as the processing of the offline database building stage.

[0075] S15, obtain the query request, and generate a user capability signature based on the query text, query boundaries, and user permission context in the query request.

[0076] Specifically, a query request can be obtained during the online query phase. Then, based on the query text, query boundaries, and user permission context in the query request, a query version identifier, query boundary encoding result, and user permission encoding result are obtained. A joint projection is then performed based on the query version identifier, query boundary encoding result, and user permission encoding result to generate a user capability signature. It should be understood that the user capability signature can be generated using a projection matrix that is the same as or equivalent to the one used on the object side (i.e., the projection matrix used when generating the joint signature). .

[0077] For example, a user permission context refers to content related to or used to determine access permissions, including at least one of user identity or user role. User identity is used to uniquely identify the specific subject initiating the query, such as employee ID, account ID, or agent instance ID; user role is used to represent the set of permission categories granted to the subject, such as legal role, audit role, project manager role, or ordinary employee role.

[0078] In some examples, the authorization validity period can also be obtained based on the query request, and the user capability signature can be generated in combination with the authorization validity period.

[0079] S16, determine the prefix security route bucket that matches the user's capability signature as the target route bucket; starting from the entry node in the target route bucket, traverse the nodes that meet the legality verification conditions on the restricted semantic graph and output the query results; the entry node refers to the representative node used for starting graph traversal.

[0080] Specifically, the user capability signature is matched with the routing key corresponding to each prefix security routing bucket built during the offline database construction phase, and the matched prefix security routing bucket is recorded as the target routing bucket.

[0081] For example, a user capability signature can be selected. Calculate the distance (e.g., Hamming distance) between the first L bits and the routing keys corresponding to each prefix security route bucket pre-established during the offline database construction phase, and then compare this distance with a preset bucket prefix matching threshold. If the comparison is less than or equal to the prefix matching threshold of that bucket, If the route matches the routing key, then the prefix security routing bucket corresponding to the routing key can be recorded as the target routing bucket.

[0082] For example, filtering and user capability signatures The formula for matching the target route bucket is as follows: in, The threshold for bucket prefix matching. This means selecting a user capability signature. The preceding L position; Used to calculate user capability signature Hamming distance between the first L bits and any routing key K; This represents a set of target route buckets, including at least one target route bucket, where each target route bucket's route key is associated with a user capability signature. Hamming distance ≤ bucket prefix matching threshold .

[0083] It should be understood that only entry nodes located in the target routing bucket will enter the subsequent graph traversal process, thereby reducing the probability of unauthorized objects entering the candidate pool. Specifically, for each target routing bucket, the entry node in the target routing bucket can be used as the current node to be expanded. For example, an entry node can be selected from the set of entry nodes in the target routing bucket as the current node to be expanded (i.e., the starting point of the graph traversal). The adjacent nodes of the node to be expanded are traversed on the restricted semantic graph as nodes to be visited; then, the legality of the nodes to be visited is checked; nodes that pass the legality check are used as new nodes to be expanded (i.e., nodes to be expanded are nodes that have passed the legality check and are ready to visit their adjacent nodes and continue to expand). The adjacent nodes of the node to be expanded that have been traversed on the restricted semantic graph are returned to continue the traversal, resulting in a set of candidate sub-objects; the set of candidate sub-objects includes the smallest controlled sub-object of each candidate that has passed the legality check.

[0084] In some embodiments, the validity verification formula is as follows: in, It is a validity check function; It is an indicator function that returns 1 if the condition is true and 0 if it is false; This can be denoted as condition 1, representing the user's capability signature. Joint signature with the i-th smallest controlled sub-object The difference (or Hamming distance) between them does not exceed the threshold θ. If it does not exceed the threshold, the value is 1; if it exceeds the threshold, the value is 0. This can be denoted as condition 2, representing the user role bitmap being queried. Role bitmap of the i-th smallest controlled sub-object The values ​​between them have an intersection; if there is an intersection, the value is 1, and if there is no intersection, the value is 0. This can be denoted as condition 3, representing the current time. If it is located within the valid time interval corresponding to the i-th smallest controlled sub-object, the value is 1; otherwise, the value is 0. This represents the state bit in the generation cancellation graph. This can be denoted as condition 4, meaning The version generation identifier of the smallest controlled sub-object. If it is a new version (i.e., the state of the node in the generation cancellation graph is a new node), the value is 1; otherwise, the value is 0.

[0085] It should be understood that only when Upon successful validation, access to or expansion of the node to be accessed is permitted only if the validation passes. Subsequently, nodes that have passed validation can be designated as new nodes to be expanded. The process continues by traversing the restricted semantic graph, checking the validity of each of these neighboring nodes as new nodes to be accessed. This iterative process yields the smallest controlled sub-objects that have passed validation, forming a candidate sub-object set. Finally, query results can be output based on this candidate sub-object set.

[0086] In some embodiments, when a node to be expanded corresponds to multiple nodes to be visited that have passed the validity check, the priority of each node to be visited can be calculated based on semantic similarity and edge weights. Nodes with higher semantic relevance and larger edge weights (i.e., edge weights) that have passed the validity check can then be selected as new nodes to be expanded. This allows for faster acquisition of high-quality candidate results and reduces unnecessary traversal overhead.

[0087] For example, for any node to be accessed Its priority It can be defined as: in, From the parent node to the current node (i.e., the current node to be visited) The edge weight (i.e., the weight of the edge between the parent node and the current node). Query semantic vector representing the query text semantic vector of the current node Semantic similarity between them; and These are two pre-set weighting coefficients. It should be understood that this strategy allows graph traversal to simultaneously comply with semantic relevance and authorization compatibility constraints.

[0088] The above scheme uses the smallest controlled sub-object as the basic index unit. It models the spatial boundaries, temporal boundaries, hierarchical positions, and text offsets of different modalities using a unified boundary vector. Boundary encoding, permission encoding, and generation encoding (i.e., the mapping encoding of generation version identifiers) are jointly mapped to a stable joint signature. A prefix security routing bucket is established based on the joint signature, and a restricted semantic graph is constructed by combining semantic similarity, boundary continuity, signature compatibility, and generation interval. During the query phase, a user capability signature is first generated based on the user query (e.g., query version and query boundary) and permission context. Then, the search scope is narrowed down using the prefix routing bucket based on the user capability signature, and only nodes that meet the legality verification conditions are expanded on the restricted semantic graph. Through this mechanism, security control can be embedded in the candidate generation and graph traversal process, allowing permission control to be moved forward before candidate generation, rather than remaining at the result filtering stage. This reduces unnecessary computational overhead for unauthorized objects.

[0089] Furthermore, by suppressing unauthorized objects from entering the candidate pool during the candidate generation stage, the intermediate exposure window is narrowed from the source, thus reducing the risk of sensitive information leakage.

[0090] In some embodiments, when the permission tuple, boundary information, or content of the least controlled sub-object changes, using overwrite updates or index reconstruction to maintain permission changes would result in high update costs, significant service instability, and difficulty in version rollback, making it difficult to meet the real-time and stability requirements of the AI ​​foundation. This embodiment does not overwrite old nodes in place; instead, it employs a generational undo bitmap and append-write update mechanism. The generational undo bitmap refers to a bitmap structure that records the effective state of an object according to its generational identifier and object identifier.

[0091] Perform the following on old nodes (i.e., old generation nodes): That is, the object identifier of the i-th smallest controlled sub-object. Corresponding old node In the diagram, the state is set to the undo state, indicating the old node. It was revoked; at the same time, a new generation identifier was adopted. Generate a new node and set its state to satisfy the following conditions: Then, recalculate the new nodes. The boundary encoding results, permission encoding results, and joint signatures are obtained, and the new node is appended to the prefix security routing bucket and the restricted semantic graph.

[0092] For example, when the proportion of old nodes exceeds a threshold At that time, the background triggers compression and reconstruction: ;in, Indicates the number of old nodes. This indicates the total number of nodes. Then, graph reconnection, bucket reconstruction, and historical node archiving are performed. This mechanism ensures that revoked objects are quickly skipped during the query phase, while avoiding large-scale reconstruction under high-frequency updates. This enables low-cost incremental maintenance in a dynamic authorization environment while preserving traceable version information.

[0093] In some embodiments, for each candidate minimum controlled sub-object in the candidate sub-object set, a corresponding risk quantification value is generated based on the corresponding semantic similarity, boundary consistency quantification value, version consistency quantification value, and risk penalty item. Semantic similarity refers to the similarity between the semantic vector of the minimum controlled sub-object and the query semantic vector corresponding to the query text; boundary consistency quantification value represents the consistency between the object boundary of the minimum controlled sub-object and the query boundary; and version consistency quantification value represents the consistency between the version generation of the minimum controlled sub-object and the query version corresponding to the query text. Minimum controlled sub-objects with risk quantification values ​​lower than a preset threshold are removed from the candidate sub-object set, and the remaining minimum controlled sub-objects are output as query results.

[0094] For example, for the set of candidate sub-objects that pass the legality check, a risk constraint reordering is further applied. Here, the risk quantification value corresponding to the smallest controlled sub-object i in any candidate sub-object set is... It can be calculated using the following formula: ; Where q represents the query. Represents semantic similarity—that is, the semantic vector of the smallest controlled sub-object i. Query semantic vector corresponding to query text The similarity between them; This represents the boundary consistency quantization value; This represents the quantified value of version consistency. It is the risk penalty term corresponding to the smallest controlled sub-object i. to Preset weights.

[0095] For example, risk penalty items It can be calculated using the following formula: ; in, This indicates the sensitivity level of the smallest controlled sub-object i. This represents the potential exposure proportion of the smallest controlled sub-object i. This represents the cross-generation backtracking cost of the smallest controlled sub-object i. This represents the residual leakage risk after de-identification corresponding to the smallest controlled sub-object i. to This refers to the risk item weight. Based on the risk penalty item, highly sensitive objects can be prevented from being prioritized for output when boundary or authorization conditions are insufficient, thereby improving the retrieval security in sensitive business environments.

[0096] In some examples, sensitivity level Potential exposure ratio can be obtained from manual confidentiality labeling, rule mapping, or classification model output. The cost can be calculated based on the percentage of unauthorized areas, the percentage of unauthorized duration, or the percentage of visible characters; cross-generational backtracking cost. The risk of residual leakage after desensitization can be estimated from the backtracking chain length, the number of supplementary reads, or the additional decoding overhead; It can be estimated from the re-identification probability, reversible recovery risk, or residual readability. Risk term weights. to It can be obtained through historical sample tuning, expert rule setting, or target optimization based on the validation set. For example, it can be dynamically configured according to tenant, business line, industry regulatory level, or scenario risk preference. For example, in financial or medical scenarios, the weight of risk items (i.e., penalty weight) on sensitivity level and residual leakage risk after desensitization can be increased.

[0097] It should be understood that after sorting the smallest controlled sub-objects of each candidate in the candidate sub-object set based on the risk quantification value (i.e., after risk constraint reordering), the smallest controlled sub-object with a preset preset position or the smallest controlled sub-object with a risk quantification value exceeding a preset quantification threshold can be selected as the final hit smallest controlled sub-object. Subsequently, query results can be generated and output based on these final hit smallest controlled sub-objects.

[0098] In some embodiments, for the smallest controlled sub-object that is ultimately hit, different output strategies are executed based on its authorization template: ; The local desensitization rendering can include at least one method such as image region blurring, video clip masking, audio clip muting, text character replacement, field masking, or table cell hiding; the summary replacement output can be generated by the visible summary or secure summary generation module of the original object. Through the authorization template control in the output stage, a hierarchical presentation of "fully visible, partially visible, summary visible, and completely invisible" can be achieved, improving the controllability of the results. For example, the authorization template can also include at least one output method such as differential summarization, watermarking, structured field replacement, or secure fragment splicing, without limitation.

[0099] Figure 3 This is a simplified flowchart of a multimodal data query method in one embodiment. From Figure 3 It is clear that the entire process can be divided into an offline database building stage and an online query stage.

[0100] During the offline database construction phase, preparatory work such as data partitioning and index building is carried out. Specifically, after the multimodal original objects are accessed, the smallest controlled sub-objects are partitioned; then, object encoding is performed, such as generating semantic vectors, unified boundary vectors, permission tuples, and generation identifiers; furthermore, a stable band joint signature (i.e., joint signature) is generated, and a secure index is built based on the stable band joint signature and semantic vector, that is, a prefix secure routing bucket and a restricted semantic graph are built.

[0101] Once the online query phase begins, a retrieval can be performed based on the constructed prefix security routing buckets and restricted semantic graph. Specifically, a query request is received, and a user capability signature is generated. Prefix security routing is then performed based on the user capability signature and the prefix security routing bucket to perform pre-access filtering and select matching target routing buckets. Based on the entry nodes in the target routing buckets, a restricted semantic graph traversal is initiated based on legality constraints to select a set of candidate sub-objects. Subsequently, the candidate sub-object set is reordered according to risk constraints (i.e., security assessment sorting). Based on the sorting results, the query results are output according to the authorization template to achieve result output control and adapt to the security requirements of different authorization levels and different business scenarios.

[0102] Overall, this method enables multimodal fine-grained secure retrieval for AI-based systems. It can suppress unauthorized objects from entering the candidate pool (i.e., the candidate sub-object set) before candidate generation, and also takes into account low-cost index maintenance in dynamic authorization environments. It has good engineering practicality and industrial application value.

[0103] It should be understood that the methods in this application embodiment can be applied to any multimodal data query scenario, such as AI foundation scenarios like enterprise knowledge bases, RAG systems, audio and video auditing, intelligent agent collaboration, and industrial intelligent analysis. The following three scenarios illustrate this further.

[0104] Scenario 1: Enterprise Knowledge Base Retrieval Enterprises integrate regulations, R&D documents, meeting recordings, surveillance videos, and on-site images into an AI platform (i.e., the multimodal data query system in this embodiment). During the offline database construction phase, the system segments regulations by paragraphs, table units, and character spans; recordings by speaker segments; videos by shot and event segments; and images by target boxes and region blocks to form minimal controlled sub-objects, thereby constructing a restricted semantic graph. Furthermore, a joint signature is generated for each minimal controlled sub-object to build a prefix-secure routing bucket based on the joint signature. Subsequently, during the online query phase, when employees from different departments ask questions through the intelligent question-and-answer system, the system generates user capability signatures based on their department, position, and project permissions. Based on these user capability signatures, searches are limited to permitted prefix-secure routing buckets, and legitimate nodes are expanded within the restricted semantic graph, thus preventing unauthorized project materials from entering the candidate pool.

[0105] Scenario 2: Audio and Video Auditing Platform In the auditing platform, auditors are only allowed to view the original content of sensitive video clips, while ordinary analysts are only allowed to view partially masked results. During the offline database construction phase, the system segments the video by shot and event, and establishes a unified boundary vector, permission tuple, and generational encoding for each event segment (i.e., the smallest controlled sub-object), thereby generating a joint signature to construct a prefix-secure routing bucket, and generating a restricted semantic graph based on each event segment. During the online query phase, when different users query the same event, they will be routed to different prefix-secure routing buckets due to their different capability signatures, thus obtaining different sets of candidate event segments. For example, complete segments, partially masked segments, or summary results can be output.

[0106] Scenario 3: Dynamic Authorization Change Scenario When a project document transitions from the confidential phase to the public phase, the system does not rewrite the old nodes. Instead, it sets the old generation nodes to the revocation state in the revocation bitmap, re-encodes them with the new generation identifier, and appends them to the new nodes. Subsequent queries only access the valid nodes of the new generation, thus completing the permission switch at a lower cost.

[0107] like Figure 4 As shown, in some embodiments, a multimodal data query system is provided, the system comprising: The segmentation module 401 is used to segment the multimodal original object into multiple minimal controlled sub-objects; The routing bucket creation module 402 is used to generate a corresponding joint signature based on the unified boundary vector, permission tuple and version generation identifier corresponding to each smallest controlled sub-object; establish at least one prefix security routing bucket; and ensure that the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix security routing bucket meets a preset proximity condition. Semantic graph construction module 403 is used to construct a restricted semantic graph with each of the smallest controlled sub-objects as nodes, based on the semantic vectors corresponding to the smallest controlled sub-objects. The query module 404 is used to obtain the query request, generate a user capability signature based on the query text, query boundary and user permission context in the query request; determine the prefix security route bucket that matches the user capability signature as the target route bucket; starting from the entry node in the target route bucket, traverse the nodes that meet the legality verification conditions on the restricted semantic graph and output the query results; the entry node refers to the representative node used to start the graph traversal.

[0108] It should be understood that, unless otherwise expressly stated herein, there is no strict order restriction on the execution of the various steps in the processes involved in the above embodiments. Moreover, these steps or stages are not necessarily to be completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the steps or stages of other steps.

[0109] In another aspect, this application also provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the various processes in the multimodal data query method described in the above embodiments and achieves the same technical effect. To avoid repetition, further details are omitted. The computer-readable storage medium may include, for example, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] In another aspect, this application also provides a computer device, which includes a processor and a memory connected to the processor. The memory stores a computer program that can be executed by the processor. When the computer program is executed by the processor, it implements the various processes in the multimodal data query method and achieves the same technical effect, which will not be elaborated here. Exemplarily, the structural diagram of the computer device can be as follows: Figure 5 As shown.

[0111] In another aspect, this application also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the various processes in the multimodal data query method and can achieve the same technical effect. To avoid repetition, it will not be described again.

[0112] It should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0113] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of the present invention. The above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A multimodal data query method, characterized in that, The method includes: Divide the multimodal primitive object into multiple minimal controlled sub-objects; Generate a corresponding joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each of the minimum controlled sub-objects; Establish at least one prefix secure routing bucket; the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix secure routing bucket satisfies a preset proximity condition; Based on the semantic vectors corresponding to the smallest controlled sub-objects, a restricted semantic graph is constructed with each smallest controlled sub-object as a node; Obtain a query request, and generate a user capability signature based on the query text, query boundaries, and user permission context in the query request; Identify the prefix security routing bucket that matches the user capability signature as the target routing bucket; The entry node in the target routing bucket is taken as the current node to be expanded, and the adjacent nodes of the node to be expanded are traversed on the restricted semantic graph as nodes to be visited; the entry node refers to the representative node used for graph traversal. Perform a validity check on the node to be visited, and take the node that passes the validity check as the new node to be expanded. Return to the execution of traversing the adjacent nodes of the node to be expanded on the restricted semantic graph to continue traversing, and obtain a set of candidate sub-objects. The set of candidate sub-objects includes the smallest controlled sub-object of each candidate that has passed the validity check. Output query results based on the set of candidate sub-objects.

2. The method according to claim 1, characterized in that, The multimodal original object includes at least two of the following: image, video, audio, or document. The smallest controlled sub-object includes at least two of the following: image region, video clip, audio clip, or document clip. The image region includes at least one of the following: a target bounding box, an instance region, or a layout region in an image. The video clip includes at least one of the following: a shot clip, a time-sliding window clip, or an event clip in a video. The audio clip includes at least one of the following: a speaker clip, a speech activity segment, or an anomalous acoustic segment in an audio file. The document clip includes at least one of the following: a page block, a table cell, a paragraph segment, or a character span in a document.

3. The method according to claim 1, characterized in that, The dimensions of the unified boundary vectors corresponding to different minimum controlled sub-objects are the same; each minimum controlled sub-object's unified boundary vector includes boundary information slots corresponding to various sub-object types and valid marker slots; each boundary information slot is used to record the boundary information of the minimum controlled sub-object under the corresponding sub-object type; the element value in the valid marker slot is used to identify the valid boundary information slot for the minimum controlled sub-object. The permission tuple corresponding to each smallest controlled sub-object includes at least one of the following: the tenant or business domain corresponding to the smallest controlled sub-object, the role bitmap with access permissions, the sensitivity level, the effective time interval, or the allowed operation type; The version generation identifier is used to represent the version to which the smallest controlled sub-object belongs; the version generation identifier changes as the smallest controlled sub-object changes in access permissions, boundaries, or content.

4. The method according to claim 1, characterized in that, The step of generating a corresponding joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each of the smallest controlled sub-objects includes: The unified boundary vector, permission tuple, and version generation identifier are encoded respectively. The obtained boundary encoding result, permission encoding result, and generation encoding result are jointly mapped to the unified signature space to obtain the joint projection vector. For each component in the joint projection vector, a corresponding deterministic jitter term is generated; By combining the deterministic jitter term with the joint projection vector to perform signature mapping, the corresponding joint signature is obtained.

5. The method according to claim 1, characterized in that, The establishment of at least one prefix security routing bucket includes: For each smallest controlled sub-object, the first preset number of bits of the joint signature is used as the routing key; A corresponding prefix security routing bucket is established based on each routing key; the prefix security routing bucket includes at least one of the following: a set of entry nodes, a set of object identifiers in the bucket, a union of role bitmaps in the bucket, an upper bound of sensitivity level in the bucket, a summary of generational distribution in the bucket, or bucket statistics. Wherein, the set of entry nodes represents the set of representative nodes used for startup graph traversal in the prefix security routing bucket; the set of object identifiers within the bucket represents the set of identifiers of the smallest controlled sub-objects in the prefix security routing bucket; the union of role bitmaps within the bucket represents the union of the role bitmaps of each smallest controlled sub-object in the prefix security routing bucket; the summary of generation distribution within the bucket represents the distribution of the number of generations, dominant generation, smallest generation, and largest generation of each version in the prefix security routing bucket; and the bucket statistics include the total number of objects, the distribution of update time, and the distribution of sensitivity levels.

6. The method according to claim 1, characterized in that, The construction of a restricted semantic graph based on the semantic vectors corresponding to the smallest controlled sub-objects, with each smallest controlled sub-object as a node, includes: Using each smallest controlled sub-object as a node, edges are constructed between nodes to obtain an initial restricted semantic graph; Calculate the semantic similarity between the semantic vectors of the two nodes to which each edge belongs in the initial restricted semantic graph; The weight of each edge is determined based on the semantic similarity, boundary continuity, signature compatibility, and generation interval between the two nodes to which each edge belongs in the initial restricted semantic graph. For each node in the initial restricted semantic graph, the edges with weights in the first preset order corresponding to the node are retained to obtain the restricted semantic graph.

7. The method according to claim 1, characterized in that, The step of generating a user capability signature based on the query text, query boundaries, and user permission context in the query request includes: Based on the query text, query boundaries, and user permission context in the query request, the query version identifier, query boundary encoding result, and user permission encoding result are obtained. A user capability signature is generated by performing a joint projection based on the query version identifier, query boundary encoding result, and user permission encoding result.

8. The method according to claim 1, characterized in that, The step of outputting query results based on the candidate sub-object set includes: For each candidate minimum controlled sub-object in the candidate sub-object set, a corresponding risk quantification value is generated based on the corresponding semantic similarity, boundary consistency quantification value, version consistency quantification value, and risk penalty item. Semantic similarity refers to the similarity between the semantic vector of the minimum controlled sub-object and the query semantic vector corresponding to the query text. The boundary consistency quantification value represents the consistency between the object boundary of the minimum controlled sub-object and the query boundary. The version consistency quantification value represents the consistency between the version generation of the minimum controlled sub-object and the query version corresponding to the query text. Remove the smallest controlled sub-object with a risk quantification value lower than a preset threshold from the candidate sub-object set, and output the remaining smallest controlled sub-object as the query result.

9. A multimodal data query system, characterized in that, The system includes: The segmentation module is used to segment the multimodal original object into multiple minimal controlled sub-objects; The routing bucket creation module is used to generate a corresponding joint signature based on the unified boundary vector, permission tuple, and version generation identifier corresponding to each of the smallest controlled sub-objects; establish at least one prefix security routing bucket; and ensure that the first preset number of bits of the joint signature of each smallest controlled sub-object corresponding to each prefix security routing bucket meets a preset proximity condition. The semantic graph construction module is used to construct a restricted semantic graph based on the semantic vectors corresponding to the smallest controlled sub-objects, with each smallest controlled sub-object as a node; The query module is used to obtain query requests, generate user capability signatures based on query text, query boundaries, and user permission context in the query request; determine the prefix security routing bucket that matches the user capability signature as the target routing bucket; take the entry node in the target routing bucket as the current node to be expanded, and traverse the adjacent nodes of the node to be expanded on the restricted semantic graph as nodes to be accessed; the entry node refers to the representative node used to initiate graph traversal; perform legality verification on the nodes to be accessed, and take the nodes that pass the legality verification as new nodes to be expanded, return to execute the traversal of the adjacent nodes of the node to be expanded on the restricted semantic graph to continue traversal, and obtain a set of candidate sub-objects; the set of candidate sub-objects includes the smallest controlled sub-object of each candidate that has passed the legality verification; and output query results based on the set of candidate sub-objects.

Images