Retrieval method, device and equipment and readable storage medium
By converting multimodal query conditions into semantic feature vectors and spatiotemporal vectors, and then calculating relevance scores after fusion, the problem of integrating semantic and spatiotemporal information in multimodal retrieval is solved, achieving efficient and low-latency retrieval results.
Patent Information
- Application Number
- CN202511755858.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-26
- Publication Date
- 2026-02-27
AI Technical Summary
Existing multimodal retrieval methods struggle to effectively integrate semantic and spatiotemporal information, resulting in high system complexity and increased query latency, failing to meet the low-latency, high-precision retrieval requirements for large-scale heterogeneous spatiotemporal multimodal data.
Multimodal query conditions are converted into semantic feature vectors, unit circle vectors, and unit sphere vectors, which are then fused to form a multimodal query vector. The relevance score with the unified encoding vector in the database is calculated, and the search results are directly output, avoiding additional spatiotemporal filtering steps.
It achieves effective integration of semantic and spatiotemporal information in multimodal retrieval, reduces system complexity, improves retrieval efficiency, and meets the requirements of low latency and high accuracy retrieval.
Smart Images
Figure CN121579478A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of computer application, in particular to a retrieval method, device and equipment and readable storage medium. BACKGROUND
[0002] With the exponential growth of multi-modal data such as images, texts, videos and audios, the development of multi-modal joint decision technology combined with large language models, the demand for multi-modal retrieval systems in scenarios such as security monitoring (such as text query of space-time monitoring clips), intelligent medical treatment (such as joint retrieval of images and electronic medical records), emergency management (such as disaster image and meteorological data association) is increasingly urgent.
[0003] The current multi-modal retrieval method has obvious defects: one type only focuses on cross-modal semantic alignment, ignoring temporal and geographical location context; another type integrates temporal and spatial constraints, but requires additional filtering mechanisms or multi-index design, resulting in increased system complexity, increased query delay, and difficulty in supporting multi-attribute fusion and efficient dynamic data update, which cannot meet the low-delay and high-precision retrieval requirements of large-scale heterogeneous spatio-temporal multi-modal data.
[0004] To sum up, how to effectively unify the retrieval scheme of semantic and spatio-temporal information and other problems are technical problems that need to be solved by technical personnel in the field at present. SUMMARY
[0005] The purpose of the present application is to provide a retrieval method, device, equipment and readable storage medium, which unifies semantic and spatio-temporal information to achieve multi-modal retrieval with consideration of efficiency and scalability.
[0006] To solve the above technical problems, the present application provides the following technical solutions:
[0007] A retrieval method comprises:
[0008] Obtaining a multi-modal query condition and parsing the multi-modal query condition to obtain input time, geographical location and sample;
[0009] Converting the sample into a semantic feature vector, converting the time into a unit circle vector, and converting the geographical location into a unit sphere vector;
[0010] Fusing the semantic feature vector, the unit circle vector and the unit sphere vector to obtain a multi-modal query vector;
[0011] Calculating the relevance score of the multi-modal query vector and the unified encoding vector in the database, and outputting the retrieval result meeting the multi-modal query condition according to the relevance score.
[0012] Preferably, converting the time into a unit circle vector comprises:
[0013] converting the time into a time stamp in seconds ;
[0014] obtaining a time range of a retrieval scene, and determining a time scale parameter using the time range ;
[0015] using a time mapping formula , mapping the time stamp into the unit circle vector .
[0016] Preferably, the geographic location is converted into a unit sphere vector, comprising:
[0017] obtaining a geographic range of a retrieval scene, and determining a geographic scale parameter using the geographic range ;
[0018] using a geographic mapping formula mapping the geographic location into the unit sphere vector .
[0019] Preferably, the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector, comprising:
[0020] normalizing the semantic feature vector, the unit circle vector and the unit sphere vector;
[0021] obtaining weights corresponding to the input sample, time and geographic location respectively;
[0022] element-level multiplying the normalized semantic feature vector, unit circle vector and unit sphere vector respectively with the corresponding weights;
[0023] sequentially splicing the multiplied results in a preset order to obtain the multi-modal query vector.
[0024] Preferably, a relevance score of the multi-modal query vector and a uniform encoding vector in a database is calculated, comprising:
[0025] calculating the inner product of the multi-modal query vector and each uniform encoding vector in the database one by one;
[0026] determining the inner product as the relevance score.
[0027] Preferably, the database is established, comprising:
[0028] determining a data timeliness parameter and a data update granularity parameter The data timeliness parameter is an integer multiple L of a data update granularity parameter.
[0029] determining that the condition is met corresponding phase step ;
[0030] constructing L disjoint phase buckets; the phase buckets B k corresponding time interval wherein T0 is an initial time node; each phase bucket is implemented to store in a continuous identifier interval and locate data in association with an external list including a bucket start / end offset and a record ID list.
[0031] Preferably, further comprising:
[0032] obtaining a new data item to be inserted into the database calculating a phase value of the new data item ;
[0033] determining a time-updated phase offset ; the phase offset is updated once every time, n is a time unit count, and the adjusted phase is obtained by the formula ;
[0034] if , the new data item is assigned to the corresponding phase bucket according to , otherwise, the new data item is determined to be expired data and discarded;
[0035] when the phase offset is updated, data in the original phase bucket exceeds timeliness and is marked as deleted and emptied of the corresponding external list, so as to skip data in the original phase bucket during retrieval.
[0036] A retrieval device, comprising:
[0037] a query condition determination module configured to obtain a multi-modal query condition and parse the multi-modal query condition to obtain an input sample, time, and geographic location;
[0038] a vector conversion module configured to convert the sample into a semantic feature vector, convert the time into a unit circle vector, and convert the geographic location into a unit sphere vector;
[0039] a vector fusion module configured to fuse the semantic feature vector, the unit circle vector, and the unit sphere vector to obtain a multi-modal query vector;
[0040] The multi-dimensional retrieval module is configured to calculate a relevance score of the multi-modal query vector and the uniformly encoded vector in the database, and output a retrieval result meeting the multi-modal query condition according to the relevance score.
[0041] An electronic device comprises:
[0042] A memory is configured to store a computer program.
[0043] A processor is configured to implement the steps of the retrieval method when executing the computer program.
[0044] A readable storage medium has a computer program stored thereon, and the computer program is configured to implement the steps of the retrieval method when executed by a processor.
[0045] According to the method provided in the embodiments of the present application, a multi-modal query condition is obtained, and time, a geographical position and a sample input by the multi-modal query condition are parsed. The sample is converted into a semantic feature vector, the time is converted into a unit circle vector, and the geographical position is converted into a unit sphere vector. The semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector. A relevance score of the multi-modal query vector and a uniformly encoded vector in a database is calculated, and a retrieval result meeting the multi-modal query condition is output according to the relevance score.
[0046] In the present application, in order to integrate uniform semantic and spatio-temporal information, after obtaining the query condition, the time, the geographical position and the sample input can be first determined. Then, the sample can be converted into a semantic feature vector, the time can be converted into a unit circle vector, and the geographical position can be converted into a unit sphere vector. After the time is converted into the unit circle vector, the cosine similarity of any two time stamps only depends on the time difference, so that the time correlation can be accurately represented in the subsequent retrieval process. After the geographical position is converted into the unit sphere vector, the cosine similarity of any two geographical positions corresponds to the great circle distance of the geographical positions after the scale adjustment, so that the spatial correlation can be accurately described. Subsequently, the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector. The relevance score of the multi-modal query vector and a uniformly encoded vector in a database is calculated, and a retrieval result meeting the multi-modal query condition is output according to the relevance score. That is, the multi-modal query vector can be directly used for subsequent approximate nearest neighbor retrieval without an additional spatio-temporal filtering step, which can effectively reduce the system complexity. Moreover, the multi-modal query vector fuses the unit circle vector corresponding to the time, the unit sphere vector corresponding to the geographical position and the semantic feature vector corresponding to the sample, so that the semantic and spatio-temporal information can be effectively integrated.
[0047] That is, the present application has the technical effect of integrating uniform semantic and spatio-temporal information, achieving multi-modal retrieval with consideration of efficiency and expansibility.
[0048] Correspondingly, the embodiments of the present application further provide a retrieval device, equipment and readable storage medium corresponding to the above retrieval method, which have the above technical effects, and details are not repeated here. BRIEF DESCRIPTION OF DRAWINGS
[0049] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.
[0050] Figure 1 The flowchart of the retrieval method in the embodiments of the present application is shown in the figure.
[0051] Figure 2 The specific implementation schematic diagram of the retrieval method in the embodiments of the present application is shown in the figure.
[0052] Figure 3 The structure schematic diagram of the retrieval device in the embodiments of the present application is shown in the figure.
[0053] Figure 4 The structure schematic diagram of the electronic equipment in the embodiments of the present application is shown in the figure.
[0054] Figure 5 The specific structure schematic diagram of the electronic equipment in the embodiments of the present application is shown in the figure. DETAILED DESCRIPTION
[0055] In order to make the person skilled in the art better understand the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.
[0056] Please refer to Figure 1 , Figure 1 The flowchart of the retrieval method in the embodiments of the present application is shown in the figure, which includes the following steps:
[0057] S101, obtain the multi-modal query condition, and parse the multi-modal query condition to obtain the input time, geographic position and sample.
[0058] In the embodiments of the present application, the multi-modal query condition can be obtained through an interactive interface, such as an improved query page. The input time, geographic location and sample are obtained by analyzing the multi-modal query condition. The sample can be one or more of image, video, text, audio and other forms of information. The time can be the query associated time, i.e., the query time; and the geographic location is the query associated address, i.e., the query address.
[0059] For subsequent processing, the multi-modal data item corresponding to the multi-modal query condition can be defined, and the data item (i.e., the sample) is represented as , wherein is a multi-modal semantic feature vector (such as a 1024-dimensional vector encoded by a CLIP model for an image, a 768-dimensional vector encoded by an LLM for text), is a time stamp in seconds, is the latitude (radian), is the longitude (radian).
[0060] S102, convert the sample into a semantic feature vector, convert the time into a unit circle vector, and convert the geographic location into a unit sphere vector.
[0061] For different specific contents of the sample, it can be converted accordingly. For example, if it is an image, it is converted into an image vector, if it is a text, it is converted into a text vector, if it is a video, it is converted into a video vector, and if it is an audio, it is converted into an audio vector. For how to convert the sample into the corresponding vector, reference can be made to related processing techniques, which will not be described here. The vector obtained after converting the sample is the semantic feature vector.
[0062] For time, in the present embodiment, it is converted into a vector representation in the unit circle coordinate system, i.e., a unit circle vector. This is convenient for subsequent time association retrieval. After converting the time into a unit circle vector, the cosine similarity of any two time stamps only depends on the time difference, so that the time association can be accurately represented in the subsequent retrieval process.
[0063] For geographic location, it is converted into a vector representation in the unit sphere coordinate system, i.e., a unit sphere vector. After converting the geographic location into a unit sphere vector, the cosine similarity of any two geographic locations corresponds to the great circle distance after scale adjustment, which can accurately depict the spatial association.
[0064] In a specific embodiment of the present application, the time is converted into a unit circle vector, including:
[0065] The time is converted into a time stamp in seconds ;
[0066] The time range of the retrieval scene is obtained, and the time scale parameter is determined using the time range ;
[0067] Utilize time mapping formula , map timestamp to unit circle vector .
[0068] That is, in the present embodiment, for time dimension encoding, a time scale parameter may be introduced, which can be adjusted according to the time range requirement of the actual retrieval scene, for example, when data within 1 year needs to be focused on, set , map timestamp to unit circle vector, and the mapping formula is . In this way, the cosine similarity of any two timestamps only depends on the time difference , that is , which ensures accurate representation of time relevance.
[0069] In a specific embodiment of the present application, the geographic position is converted into a unit sphere vector, comprising:
[0070] Obtain the geographic range of the retrieval scene, and determine the geographic scale parameter using the geographic range;
[0071] Map the geographic position to the unit sphere vector using the geographic mapping formula .
[0072] That is, in the present embodiment, for spatial dimension encoding, a spatial scale parameter is introduced, and if city-level retrieval (such as a range of 100 km radius) is focused on, can be set to improve the local spatial resolution, and map the geographic coordinates to the unit sphere vector using the mapping formula , and the cosine similarity of two geographic positions corresponds to the great circle distance of the two points after scale adjustment, that is ( is the angular distance of the two points after rotation), which realizes accurate depiction of spatial relevance.
[0073] S103, fuse the semantic feature vector, the unit circle vector and the unit sphere vector to obtain a multi-modal query vector.
[0074] When fusing the semantic feature vector, the unit circle vector and the unit sphere vector, these vectors can be directly spliced to obtain a multi-modal query vector.
[0075] In practical applications, the input samples may be of various types. Therefore, when concatenating the samples, they can be concatenated in a fixed order. For missing samples of a certain type, zero padding can be used to replace them for subsequent retrieval.
[0076] Among them, semantic feature vectors Time-coded vector (i.e., unit circle vector), spatial encoding vector (i.e., the unit sphere vector z) are concatenated and then normalized using methods such as L2 normalization to obtain a uniformly encoded multimodal query vector. This vector can be directly used for subsequent approximate nearest neighbor retrieval without the need for additional spatiotemporal filtering steps, effectively reducing system complexity.
[0077] The core of L2 normalization is a standardization operation that scales the L2 norm (Euclidean distance) of a vector to 1. It is also called "Euclidean normalization". Essentially, it transforms the original vector into a unit vector (magnitude = 1), retaining only its direction information and eliminating the influence of length differences.
[0078] In one specific embodiment of this application, a multimodal query vector is obtained by fusing semantic feature vectors, unit circle vectors, and unit sphere vectors, including:
[0079] Normalize the semantic feature vector, unit circle vector, and unit sphere vector;
[0080] Obtain the weights corresponding to the input sample, time, and geographical location;
[0081] The normalized semantic feature vector, unit circle vector, and unit sphere vector are multiplied element-wise with their respective weights.
[0082] The results of multiplication are sequentially concatenated according to a preset order to obtain a multimodal query vector.
[0083] That is, in this embodiment, users can also specify the weight distribution for each search requirement. Specifically, after normalizing the semantic feature vector, unit circle vector, and unit sphere vector, the weights corresponding to the input sample, time, and geographical location can be obtained. Then, the normalized semantic feature vector, unit circle vector, and unit sphere vector are multiplied element-wise with their corresponding weights. Finally, the results of the multiplications are concatenated in a preset order to obtain the multimodal query vector.
[0084] For example: It can receive user input of multimodal query conditions and weight configurations, where the multimodal query conditions include image queries. Text query Time range query , geographical position query , the user sets the weight of each modality according to the retrieval intention (for example, when focusing on image content retrieval, set , and the sum of the remaining weights is 0.4), all query vectors are first subjected to L2 normalization processing.
[0085] Then, the modality query vectors are weighted and spliced, specifically: the and , and , (rotated and encoded vectors) and , (rotated and encoded vectors) and are respectively multiplied element by element, and then the weighted modality vectors are spliced in the order of “image-text-time-space” to obtain a unified query vector The splicing process does not require the introduction of an additional learning projection layer, maintains parameter independence, and reduces model complexity.
[0086] S104, calculate the relevance score of the multi-modal query vector and the unified encoding vector in the database, and output the retrieval result meeting the multi-modal query condition according to the relevance score.
[0087] In the present application, please refer to Figure 2 , for the data content stored in the database, the unified encoding vector can be encoded and stored according to the encoding mode of the multi-modal query vector.
[0088] In this way, when retrieving, the relevance score of the multi-modal query vector and the unified encoding vector in the database can be directly calculated, and finally based on the relevance score, the retrieval result meeting the multi-modal query condition can be output.
[0089] In one specific embodiment of the present application, calculating the relevance score of the multi-modal query vector and the unified encoding vector in the database comprises:
[0090] Calculate the inner product of the multi-modal query vector and each unified encoding vector in the database one by one;
[0091] Determine the inner product as the relevance score.
[0092] For example, the unified query vector can be input into the HNSW-based ANN index to perform a one-time approximate nearest neighbor search, and the inner product of and the unified encoding vector in the database is calculated as the relevance score, and the formula is for the i-th modality sub-vector of the data item Top-K results are returned according to the scores.
[0093] By applying the method provided in the embodiments of the present application, the multi-modal query condition is obtained, and the input time, geographical position and sample are parsed from the multi-modal query condition; the sample is converted into a semantic feature vector, the time is converted into a unit circle vector, and the geographical position is converted into a unit sphere vector; the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector; the relevance score of the multi-modal query vector and the uniform encoding vector in the database is calculated, and the retrieval result meeting the multi-modal query condition is output according to the relevance score.
[0094] In the present application, in order to balance the unified integration of semantics and spatio-temporal information, after obtaining the query condition, the input time, geographical position and sample can be first determined. Then, the sample can be converted into a semantic feature vector, the time is converted into a unit circle vector, and the geographical position is converted into a unit sphere vector. After the time is converted into a unit circle vector, the cosine similarity of any two time stamps only depends on the time difference, so that the accurate representation of the time correlation can be ensured in the subsequent retrieval process. After the geographical position is converted into a unit sphere vector, the cosine similarity of any two geographical positions corresponds to the great circle distance of the adjusted scale, which can realize the accurate description of the spatial correlation. Subsequently, the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector. The relevance score of the multi-modal query vector and the uniform encoding vector in the database is calculated, and the retrieval result meeting the multi-modal query condition is output according to the relevance score. That is, the multi-modal query vector can be directly used for subsequent approximate nearest neighbor retrieval without additional spatio-temporal filtering steps, which can effectively reduce the system complexity. And the multi-modal query vector fuses the unit circle vector corresponding to the time, the unit sphere vector corresponding to the geographical position and the semantic feature vector corresponding to the sample, which can effectively unify the integration of semantics and spatio-temporal information.
[0095] That is, the present application has the technical effect of balancing the efficiency and expansibility of multi-modal retrieval by unified integration of semantics and spatio-temporal information.
[0096] It should be noted that based on the above embodiments, the present application also provides corresponding improvement schemes. The steps involved in the preferred / improved embodiments can be mutually referenced, and the corresponding beneficial effects can also be mutually referenced. In the preferred / improved embodiments of the present application, they will not be described one by one.
[0097] In a specific embodiment of the present application, a database is established, comprising:
[0098] determining a data timeliness parameter and data update granularity parameters The data timeliness parameter is an integer multiple L of the data update granularity parameter.
[0099] Determined to meet The corresponding phase step size ;
[0100] Construct L disjoint phase dot matrix; phase dot matrix B k Corresponding time interval T0 is the initial time node; each phase bucket is stored using a continuous identifier range and is used in conjunction with an external list for locating data. The external list includes the start / end offset of the bucket and a list of record IDs.
[0101] First determine the length of the sliding window. The settings are based on the data timeliness requirements of actual applications. For example, in emergency management scenarios, data within the last 6 months needs to be retained. Seconds, and a base unit for time. Months (i.e., (seconds), make (here) ), and calculate the phase step size. ,make sure This allows effective data to occupy the phase space. A semi-circular region.
[0102] Divide the database into L disjoint phase buckets. Each phase bucket Corresponding time interval (T0 is the initial time node). Each bucket is stored using a continuous range of internal identifiers within an ANN index, and is paired with an external list containing the bucket's start / end offsets and a list of record IDs for easy data location.
[0103] In one specific embodiment of this application, it further includes:
[0104] Retrieve new data items to be inserted into the database Calculate the phase value of the new data item. ;
[0105] Determine the phase offset updated over time Phase shift every time Updated once per time, , where n is the time unit for counting, determined by the formula The adjusted phase is obtained;
[0106] like Then according to Assign new data items to the corresponding phase buckets , otherwise, determine the new data item as expired data and discard it;
[0107] At phase shift update, the original phase bucket data exceeds time limit, and is marked as deleted and emptied of the corresponding external list, so as to skip the data in the original phase bucket when retrieving.
[0108] When a new data item is inserted into the database, its phase value is calculated at the same time, the phase shift is introduced, which is updated over time (every time unit, n is the count of time units), and the adjusted phase is obtained by the formula If , the data item is assigned to the corresponding phase bucket , otherwise, it is determined as expired data and discarded.
[0109] When the phase shift is updated, the data in the original phase bucket exceeds the time limit, and is marked as deleted in the ANN index (so as to skip when retrieving) and emptied of the corresponding external list, without the need for global reconstruction of the existing index, and only through logical phase adjustment to achieve data update, avoiding the delay peak caused by traditional global recoding.
[0110] Corresponding to the above method embodiment, the present embodiment also provides a retrieval device, and the retrieval device described below can be mutually corresponding and referred to with the retrieval method described above.
[0111] Referring to Figure 3 , the device includes the following modules:
[0112] The query condition determination module 101 is used to obtain a multi-modal query condition, and parse the multi-modal query condition to obtain an input sample, time, and geographical location;
[0113] The vector conversion module 102 is used to convert the sample into a semantic feature vector, convert the time into a unit circle vector, and convert the geographical location into a unit sphere vector;
[0114] The vector fusion module 103 is used to fuse the semantic feature vector, the unit circle vector, and the unit sphere vector to obtain a multi-modal query vector;
[0115] The multi-dimensional retrieval module 104 is used to calculate the relevance score of the multi-modal query vector and the uniform encoding vector in the database, and output the retrieval result meeting the multi-modal query condition according to the relevance score.
[0116] The device provided by the embodiment of the application is applied to obtain a multi-modal query condition, and time, a geographical position and a sample input are obtained by analyzing the multi-modal query condition; the sample is converted into a semantic feature vector, the time is converted into a unit circle vector, and the geographical position is converted into a unit sphere vector; the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector; a correlation score of the multi-modal query vector and a uniform coding vector in a database is calculated, and a search result meeting the multi-modal query condition is output according to the correlation score.
[0117] In the application, in order to integrate uniform semantic and space-time information, after obtaining the query condition, the time, the geographical position and the sample input can be determined first. Then, the sample can be converted into a semantic feature vector, the time can be converted into a unit circle vector, and the geographical position can be converted into a unit sphere vector. After the time is converted into the unit circle vector, the cosine similarity of any two time stamps only depends on the time difference, so that the time correlation can be accurately represented in the subsequent search process. After the geographical position is converted into the unit sphere vector, the cosine similarity of any two geographical positions corresponds to the great circle distance of the geographical positions after the scale adjustment, so that the space correlation can be accurately described. Then, the semantic feature vector, the unit circle vector and the unit sphere vector are fused to obtain a multi-modal query vector. The correlation score of the multi-modal query vector and the uniform coding vector in the database is calculated, and the search result meeting the multi-modal query condition is output according to the correlation score. That is, the multi-modal query vector can be directly used for subsequent approximate nearest neighbor search without an additional time-space filtering step, which can effectively reduce the system complexity. The multi-modal query vector fuses the unit circle vector corresponding to the time, the unit sphere vector corresponding to the geographical position and the semantic feature vector corresponding to the sample, and can effectively integrate uniform semantic and space-time information.
[0118] That is, the application has the technical effect of integrating uniform semantic and space-time information, and achieving multi-modal search considering efficiency and expansibility.
[0119] In one specific embodiment of the application, the vector conversion module is specifically configured to convert the time into a time stamp in seconds ;
[0120] The time range of the search scene is obtained, and the time scale parameter is determined by using the time range ;
[0121] The time stamp is mapped into the unit circle vector by using a time mapping formula . .
[0122] In one specific embodiment of the application, the vector conversion module is specifically configured to convert the geographical position into a unit sphere vector, comprising:
[0123] acquiring a geographical range of a search scenario, and determining a geographical scale parameter using the geographical range ;
[0124] using a geographical mapping formula mapping a geographical position into a unit sphere vector .
[0125] In an embodiment of the present application, the vector fusion module is specifically configured to normalize the semantic feature vector, the unit circle vector and the unit sphere vector;
[0126] acquiring weights corresponding to the input sample, time and geographical position respectively;
[0127] multiplying the normalized semantic feature vector, the unit circle vector and the unit sphere vector respectively with the corresponding weights at an element level;
[0128] sequentially splicing the multiplied results in a preset order to obtain a multi-modal query vector.
[0129] In an embodiment of the present application, the multi-dimensional retrieval module is specifically configured to calculate the inner product of the multi-modal query vector and each uniform encoding vector in the database one by one;
[0130] determining the inner product as a relevance score.
[0131] In an embodiment of the present application, the database establishment module is configured to
[0132] determine a data timeliness parameter and a data update granularity parameter ; the data timeliness parameter is an integer multiple L of the data update granularity parameter;
[0133] determine a corresponding phase step under the condition that ;
[0134] construct L disjoint phase buckets; a phase bucket B k corresponds to a time interval , wherein T0 is an initial time node; each phase bucket is stored in a continuous identifier interval and is positioned with an external list, the external list including a bucket start / end offset and a record ID list.
[0135] In an embodiment of the present application, the data update module is configured to acquire a new data item to be inserted into the database, and calculate a phase value of the new data item;
[0136] determining the phase offset updated over time ; the phase offset is updated every time unit, n is the count of time units, and the adjusted phase is obtained by formula
[0137] If , the new data item is assigned to the corresponding phase bucket according to , otherwise, the new data item is determined as expired data and discarded.
[0138] When the phase offset is updated, the data in the original phase bucket is out of time limit, and is marked as deleted and the corresponding external list is emptied, so that the data in the original phase bucket is skipped during retrieval.
[0139] Corresponding to the above method embodiments, the embodiments of the present application also provide an electronic device, and the electronic device described below can be mutually corresponding with reference to the retrieval method described above.
[0140] Referring to Figure 4 , the electronic device includes:
[0141] a memory 332 for storing a computer program;
[0142] a processor 322 for executing the computer program to implement the steps of the retrieval method of the above method embodiments.
[0143] Specifically, please refer to Figure 5 , Figure 5 The specific structural diagram of an electronic device provided by the present embodiment can have great differences due to different configurations or performances, and can include one or more than one processor (central processing unit, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be temporary storage or persistent storage. The program stored in the memory 332 can include one or more modules (not shown in the figure), and each module can include a series of instruction operations in the data processing device. Further, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the electronic device 301.
[0144] The electronic device 301 can also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.
[0145] The steps in the retrieval method described above can be implemented by the structure of the electronic device.
[0146] Corresponding to the above method embodiments, the embodiments of the present application also provide a readable storage medium. The readable storage medium described below can be correspondingly referred to the retrieval method described above.
[0147] A readable storage medium, the readable storage medium has a computer program stored thereon, the computer program is executed by a processor to implement the steps of the retrieval method of the above method embodiments.
[0148] The readable storage medium can be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various readable storage media that can store program codes.
[0149] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.
[0150] Those skilled in the art can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of each example are described in general in the above description. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.
[0151] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, software modules executed by a processor, or a combination of the two. The software modules can be placed in random access memory (RAM), memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.
[0152] Finally, it should be noted that, in this document, relationships such as first and second, and the like, are intended to distinguish one entity or action from another, and do not necessarily require or imply any actual such relationship or order between entities or actions. Moreover, the terms including, including, comprising, and variations thereof, are intended to encompass non-exclusive inclusions, such that a process, method, article, or apparatus that includes a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.
[0153] The principles and implementation modes of the present application are described herein by applying specific examples, and the above description of the examples is only for the purpose of helping to understand the method and its core idea of the present application; meanwhile, for those skilled in the art, the specific implementation modes and application ranges will be changed according to the idea of the present application, and in summary, the content of the specification should not be understood as a limitation of the present application.
Claims
1. A retrieval method characterized by, The method comprises the following steps: acquiring a multi-modal query condition and parsing the multi-modal query condition to obtain an input time, a geographic location and a sample; converting the sample into a semantic feature vector, converting the time into a unit circle vector, and converting the geographic location into a unit sphere vector; fusing the semantic feature vector, the unit circle vector and the unit sphere vector to obtain a multi-modal query vector; calculating a relevance score of the multi-modal query vector and a uniformly encoded vector in a database, and outputting a retrieval result meeting the multi-modal query condition according to the relevance score.
2. The method of claim 1, wherein, The step of converting the time into a unit circle vector comprises the following steps: converting the time into a time stamp in seconds ; Acquiring a time range of a search scenario, and determining a time scale parameter using the time range ; Utilizing a time mapping formula mapping the timestamp to the unit circle vector .
3. The method of claim 1, wherein, The step of converting the geographic location into a unit sphere vector comprises the following steps: Acquiring a geographic extent of a search scenario and determining a geographic scale parameter using the geographic extent ; Utilizing a geographic mapping formula mapping the geographic location to the unit sphere vector .
4. The method of claim 1, wherein, The step of fusing the semantic feature vector, the unit circle vector and the unit sphere vector to obtain a multi-modal query vector comprises the following steps: performing normalization processing on the semantic feature vector, the unit circle vector and the unit sphere vector; acquiring weights corresponding to the input sample, time and geographic location respectively; performing element-level multiplication on the normalized semantic feature vector, unit circle vector and unit sphere vector respectively and the corresponding weights; sequentially splicing the multiplication results in a preset order to obtain the multi-modal query vector.
5. The method of claim 1, wherein, The step of calculating a relevance score of the multi-modal query vector and a uniformly encoded vector in a database comprises the following steps: calculating the inner product of the multi-modal query vector and each uniformly encoded vector in the database one by one; determining the inner product as the relevance score.
6. The method according to any one of claims 1 to 5, characterized in that, The step of establishing the database comprises the following steps: Determining a data timeliness parameter and a data update granularity parameter ; the data timeliness parameter is an integer multiple L of the data update granularity parameter determining that the condition is met the corresponding phase step ; Constructing L disjoint phase buckets; the phase buckets B k Corresponding time interval Wherein T0 is an initial time node; each phase bucket is implemented with a continuous identifier interval for storage, and is located with external lists including start / end offset of the bucket and a record ID list.
7. The method of claim 6, wherein, The method further comprises the following steps: acquiring a new data item to be inserted into the database , calculating a phase value of the new data item ; Determining a phase offset updated over time ; the phase offset is updated once every time unit, n is the count of time units, the adjusted phase is obtained by the formula φn+1= φn+ Δφ If then assigning the new data item to a corresponding phase bucket according to otherwise determining the new data item as expired data and discarding; At the phase offset Upon update, the original phase bucket Internal data beyond Stale, marked as deleted and emptied of the corresponding external manifest so that upon retrieval, data in the original phase bucket is skipped.
8. A retrieval device characterized by, The method comprises the following steps: a query condition determination module configured to acquire a multi-modal query condition and parse the multi-modal query condition to obtain an input sample, time and geographic location; a vector conversion module configured to convert the sample into a semantic feature vector, convert the time into a unit circle vector, and convert the geographic location into a unit sphere vector; a vector fusion module configured to fuse the semantic feature vector, the unit circle vector and the unit sphere vector to obtain a multi-modal query vector; a multi-dimensional retrieval module configured to calculate a relevance score of the multi-modal query vector and a uniformly encoded vector in a database, and output a retrieval result meeting the multi-modal query condition according to the relevance score.
9. An electronic device, comprising: The method comprises the following steps: a memory configured to store a computer program; a processor configured to implement the steps of the retrieval method according to any one of claims 1 to 7 when executing the computer program.
10. A readable storage medium, characterized by, The computer program is stored on the readable storage medium and is executed by the processor to implement the steps of the retrieval method according to any one of claims 1 to 7.