Multi-modal time sequence mixed query system and method

Abstract

The invention discloses a multi-modal time sequence hybrid query system and method. The hybrid query method comprises the steps of collecting multi-modal time sequence data, performing normalization, denoising and feature extraction on a numerical time sequence, performing cleaning, conversion and classification feature extraction on text data, and performing enhancement and feature extraction on picture data; the statistical features of the numerical time series are converted into vectors, similarities of different modes are calculated, comprehensive similarity measurement is constructed, and a neighbor graph index and a multi-layer index structure are constructed based on the measurement; the method comprises the following steps: analyzing a mixed query statement input by a user, extracting a multi-modal keyword and a natural language description, and converting the natural language description into a multi-modal feature vector; according to the method, a comprehensive similarity measurement and multi-layer index structure is constructed, so that related data can be quickly positioned and matched in the query process, the problems of low multi-modal search efficiency and poor accuracy are solved, the combination of a time sequence and multi-modal query is realized, and the query efficiency is improved.

Description

Technical Field

[0001] This application relates to the technical field of data processing, and specifically relates to a multi-modal time series hybrid query system and method. Background Art

[0002] With the development of digitalization, data in various fields has shown explosive growth. Time series data widely exists in scenarios such as transaction records in the financial field, patient vital sign monitoring in the medical field, equipment operation parameter records in the industrial field, and vehicle flow statistics in the transportation field. Moreover, the data often exhibits multi-modal characteristics, covering various types such as text, pictures, and numerical values. For example, in the financial scenario, there are both numerical time series of customer transaction amounts, as well as transaction note texts and customer ID photos. With the continuous improvement of the demand for data value mining in various industries, users' complex query requirements that include time series conditions and multi-modal constraints are increasing day by day. The realization of these requirements depends on efficient data processing technologies.

[0003] After retrieval, Patent No. CN119149561A discloses a method and system for a multi-modal data query language with a hybrid structure, and the solution is as follows: create a data schema for a multi-modal data set, and obtain the nested relationship between different types of data elements in the data schema; select a key data structure or supplement it with an algorithm as the first-layer data structure of the entire data schema, and use different types of data elements as elements or sub-structures under the first-layer data structure or the first-layer data structure; establish data element objects according to the nested relationship formed by the single-layer or multi-layer data structure and the constructed query language; complete the dynamic autonomous construction of the multi-modal data set query language according to the data element objects, defined data schema representation rules, defined attribute representation rules, defined value transformation rules, and defined ordinary expression filtering conditions and complex filtering conditions for nested type data elements; obtain the query statement input by the user, and obtain the query result according to the query statement and the constructed multi-modal data set query language.

[0004] Patent No. CN117195144A discloses a multi-modal time series outlier detection method and device that combines AAE and GAN, and the solution is as follows: obtain the time series data to be detected in multiple modalities and divide it into a training set and a test set; input the obtained training set into an outlier detection model for training to obtain a trained outlier detection model; input the obtained test set into the trained outlier detection model to obtain the outlier scores corresponding to the time series data to be detected in multiple modalities; determine whether the outlier scores are greater than a discrimination threshold. If so, the time series data to be detected in the test set is outlier data. If not, the time series data to be detected in the test set is normal data.

[0005] In summary, existing multi-modal data query technologies rely on preset data patterns and rules when constructing query languages. Facing complex and variable data types and relationships, they are difficult to flexibly adjust, resulting in low query efficiency. At the same time, this query method based on fixed nested relationships cannot effectively handle complex query logics that mix multi-modal and time series, affecting the accuracy of query results.

[0006] In addition, existing multi-modal time series outlier detection technologies only focus on outlier detection and are not combined with query functions, unable to meet the need for multi-modal association query of outlier data in actual scenarios. In addition, this technology does not sufficiently mine the potential connections of multi-modal data, resulting in limited accuracy and reliability of outlier detection. Summary of the Invention

[0007] In this embodiment, a multi-modal time series hybrid query system and method are provided to solve the problem in related technologies that cannot efficiently process the hybrid query of multi-modal time series data.

[0008] In the first aspect, an embodiment of the present invention provides a multi-modal time series hybrid query system, and the multi-modal time series hybrid query system includes: A data collection and preprocessing module, configured to collect multi-modal time series data, perform normalization, denoising, and feature extraction on numerical time series, perform cleaning, conversion, and classification feature extraction on text data, and perform enhancement and feature extraction on picture data; An index construction module, configured to convert the statistical features of numerical time series into vectors, calculate the similarity of different modalities, construct a comprehensive similarity metric, and construct a nearest neighbor graph index and a multi-layer index structure based on this metric; A query parsing and optimization module, configured to parse the hybrid query statement input by the user, extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions, convert the natural language descriptions into multi-modal feature vectors, and perform screening in combination with time series conditions; A result processing and return module, configured to sort the filtered data items according to the comprehensive similarity, calculate statistical information, and display the results in a visual form.

[0009] In an optional embodiment, the data collection and preprocessing module is specifically configured to: Perform normalization processing on numerical time series and perform feature extraction on the denoised time series; Clean the text data, remove special characters and stop words, and convert the cleaned text into a vector representation; Perform image enhancement processing on picture data and perform image feature extraction using a convolutional neural network.

[0010] In an alternative embodiment, the index construction module is specifically configured to: Calculate text-image similarity, text-time series similarity, and image-time series similarity, and calculate a comprehensive similarity metric based on weighted average; Construct a nearest neighbor graph index, where each data item is connected to its k most similar data items, and the weight of the edge is the comprehensive similarity metric; Construct a multi-layer index structure, divide time slices according to time granularity, and construct sub-indexes within each time slice.

[0011] In an alternative embodiment, the query parsing and optimization module is specifically configured to: Extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions from the user query statement; Use a pre-trained model to convert the natural language description into a multi-modal feature vector, and calculate its similarity with the feature vectors of different modal data; Combine the time series conditions and the multi-layer index structure, first perform a quick screening, and then perform an accurate screening.

[0012] In an alternative embodiment, the result processing and return module is specifically configured to: Sort the filtered data items according to the comprehensive similarity, and select the top-ranked data items as the query results; Calculate the statistical information of the query results, and organize it into a visual form for display.

[0013] Compared with the prior art, the beneficial effects of the multi-modal time series hybrid query system of the present invention are as follows: The comprehensive similarity metric and multi-layer index structure constructed by the present invention enable the query process to quickly locate and match relevant data, solve the problems of low multi-modal search efficiency and poor accuracy, realize the combination of time series and multi-modal queries, and improve the query efficiency. Using a pre-trained model to optimize query parsing and execution can convert natural language into multi-modal feature vectors, and perform accurate screening in combination with time series conditions, which can efficiently meet the diverse and complex query requirements in practical applications. Fusing and processing multi-modal time series data can more accurately capture data features, improve the accuracy and comprehensiveness of data processing, and provide data support for subsequent analysis.

[0014] In a second aspect, an embodiment of the present invention provides a multi-modal time series hybrid query method, including the following steps: Collect multi-modal time series data, normalize, denoise, and extract features from numerical time series, clean, transform, and extract classification features from text data, and enhance and extract features from image data; Convert the statistical features of the numerical time series into vectors, calculate the similarity of different modalities, construct a comprehensive similarity measure, and build a neighbor graph index and a multi-level index structure based on this measure; Parse the mixed query statement input by the user, extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions, convert the natural language descriptions into multi-modal feature vectors, and perform filtering in combination with the time series conditions; Sort the filtered data items according to the comprehensive similarity, calculate the statistical information, and display the results in a visual form.

[0015] In an optional embodiment, the steps of constructing the comprehensive similarity measure include: Calculate the text-image similarity, text-time series similarity, and image-time series similarity; Calculate the comprehensive similarity measure based on weighted average, and the weights are determined according to experiments and domain knowledge.

[0016] In an optional embodiment, the steps of constructing the multi-level index structure include: Divide the time slices according to the time granularity, and construct sub-indexes within each time slice; During query, first quickly locate the corresponding time slice index according to the time range, and then perform exact similarity matching within the time slice.

[0017] In an optional embodiment, the steps of parsing the user query statement include: Extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions; Use the pre-trained model to convert the natural language description into a multi-modal feature vector, and calculate its similarity with the feature vectors of different modality data.

[0018] In an optional embodiment, the steps of result processing and return include: Sort the filtered data items according to the comprehensive similarity, and select the top-ranked data items as the query results; Calculate the statistical information of the query results, and organize and display it in a visual form.

[0019] In a third aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a bus, wherein the processor, the communication interface, and the memory communicate with each other through the bus, and the processor can call the logical instructions in the memory to execute the steps of the method provided in the second aspect.

[0020] Fourthly, an embodiment of the present invention provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the multi-modal time series hybrid query method described in the second aspect are implemented.

[0021] Compared with the prior art, the beneficial effects of the multi-modal time series hybrid query method, electronic device and storage medium of the present invention are the same as those of the multi-modal time series hybrid query system described in the first aspect, so they will not be elaborated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following briefly introduces the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 It is a flowchart of the multi-modal time series hybrid query method in the embodiment of the present invention; Figure 2 It is a schematic diagram of the multi-modal time series hybrid query method in the embodiment of the present invention; Figure 3 It is a structural block diagram of the multi-modal time series hybrid query system in the embodiment of the present invention; Figure 4 It is a structural block diagram of the electronic device in the embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0024] In order to more clearly understand the purpose, technical solution and advantages of the present application, the present application will be described and illustrated below with reference to the drawings and embodiments.

[0025] Unless otherwise defined, technical terms or scientific terms involved in this application shall have the general meanings understood by those with ordinary skills in the technical field to which this application belongs. In this application, words such as "a", "an", "one kind", "the", "these", etc. do not indicate a limitation in quantity, and they can be singular or plural. The terms "including", "containing", "having" and any variants thereof involved in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product or device containing a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent in these processes, methods, products or devices. The terms "connected", "linked", "coupled", etc. involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application means two or more. "And / or" describes the association relationship of associated objects and indicates that three relationships can exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, and B exists alone. Usually, the character " / " indicates that the objects associated before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific order for the objects.

[0026] Time series data widely exists in many fields and often has the characteristics of multimodality. In actual application scenarios, users often need to perform complex queries that include time series conditions and multimodal constraints. However, existing data query technologies are difficult to efficiently process such complex hybrid query requirements. Traditional multimodal search methods have problems in efficiency and accuracy when dealing with multimodal data and do not involve the processing of time series data. And methods for processing time series data usually focus on sequence analysis and are difficult to effectively combine with the query of multimodal data. Therefore, it is of great practical significance to develop a technology that can effectively process multimodal and time series hybrid queries.

[0027] In an embodiment of the present invention, a multimodal time series hybrid query method is provided. Figure 1 It is a flowchart of the multimodal time series hybrid query method of the present invention. Figure 2 It is a schematic diagram of the multimodal time series hybrid query method of the present invention. As Figure 1 and Figure 2 shown, the multimodal time series hybrid query method includes the following steps: S100. Collect multi-modal time series data, normalize, denoise, and extract features from numerical time series, clean, transform, and extract classification features from text data, and enhance and extract features from image data; Specifically, according to the query needs, collect multi-modal time series data, including text modality, image modality, and numerical time series modality. Mark the data of different modalities at the time point as . To unify the data scale, perform normalization processing on the numerical time series. Denoise the numerical time series, and extract features from the denoised time series. Clean the collected text data, removing irrelevant information such as special characters and stop words. Use a pre-trained word vector model to convert the cleaned text into a vector representation, and based on the trained text classification model, extract classification features from the text. Apply image enhancement technology to process the image data, and use a convolutional neural network to extract image features.

[0028] Exemplarily, as shown in combination with Figure 2 , taking the financial scenario as an example, during the process of carrying out financial business, it is necessary to collect information related to customers and transactions. In terms of the text modality, collect personal information filled in by customers during registration, transaction remarks, communication records with customer service, etc.; in terms of the image modality, obtain customer ID card photos, bank card photos, transaction scene screenshots, etc.; in terms of the numerical time series modality, record data such as the amount, time, change in account balance, and transaction frequency of each transaction. The transaction amount at time can be represented by , the transaction remarks text at this moment can be represented by , and the ID card photo image data at this moment can be represented by .

[0029] S200. Convert the statistical features of the numerical time series into vectors, calculate the similarity of different modalities, construct a comprehensive similarity measure, and based on this measure, construct a nearest neighbor graph index and a multi-level index structure; Specifically, the steps for constructing the comprehensive similarity measure include: Calculate text-image similarity, text-time series similarity, and image-time series similarity; Calculate the comprehensive similarity measure based on weighted average, and the weights are determined according to experiments and domain knowledge.

[0030] Convert the statistical features of the numerical time series within a specific time window into vector form, and use cosine similarity to calculate text-image similarity, text-time series similarity, and image-time series similarity respectively. Define the comprehensive similarity measure For the weighted average of these similarities, it should be noted that the weights are determined based on experiments and domain knowledge.

[0031] and respectively represent the norms of vectors and The feature vectors of images are extracted using a convolutional neural network , and the image-time series similarity is calculated using cosine similarity, and the calculation formula is . The text-image similarity is calculated using cosine similarity, and the calculation formula is . Define the comprehensive similarity metric as the weighted average of the above three similarities, that is , where , and are the weights of text-image similarity, text-time series similarity, and image-time series similarity respectively, and satisfy . These weights are determined based on experiments and domain knowledge. For example, in the scenario of multimodal data analysis in the medical field, it is assumed that text data (such as medical record descriptions) is more important for judging data similarity, while image data (such as medical images) and numerical time series data (such as vital sign data) are relatively less important. Then the weights can be set as , , . In the multimodal data analysis in the e-commerce field, if image data (such as product pictures) plays a key role in similarity judgment, text data (such as product descriptions) is secondary, and time series data (such as sales data) is relatively less important, then the weights can be set as , , .

[0032] Centered on each data item in the data item set , according to the comprehensive similarity metric , find the most similar data items in the set as neighbors to construct a neighbor graph index . is the node set (corresponding to data items), is the edge set, and the edge indicates that and are in a neighbor relationship, and the weight of the edge is . The

[0033] The steps of constructing the multi - layer index structure include: Divide time slices according to time granularity, and construct sub - indexes within each time slice; During query, first quickly locate the corresponding time - slice index according to the time range, and then perform precise similarity matching within the time slice.

[0034] Specifically, to speed up the query speed, a multi - layer index structure is constructed. The data items are divided into different time slices according to time order, and sub - indexes are constructed within each time slice according to comprehensive similarity. For example, divide time slices according to time granularity such as days, weeks, months, etc. During query, first quickly locate the corresponding time - slice index according to the time range, and then perform precise similarity matching within the time slice.

[0035] S300. Parse the mixed query statement input by the user, extract multi - modal keywords, natural - language descriptions, time range, and trend conditions, convert the natural - language description into a multi - modal feature vector, and perform screening in combination with the time - series conditions; Specifically, the steps of parsing the user's query statement include: Extract multi - modal keywords, natural - language descriptions, time range, and trend conditions; Use a pre - trained model to convert the natural - language description into a multi - modal feature vector, and calculate its similarity with the feature vectors of different - modality data.

[0036] When the query user input contains a mixed query statement with multi - modal and time - series conditions, the system parses the query condition set , extracts the multi - modal keyword set , natural - language description , as well as the time range and trend conditions .

[0037] Use the pre - trained model to convert the natural - language description into a multi - modal feature vector , calculate the similarity with the feature vectors of different - modality data and determine the comprehensive similarity. Screen out the data items whose comprehensive similarity is greater than the threshold, within the time range , and meet the trend conditions at the same time. Use the multi - layer structure of the fusion index, first perform quick screening in the sub - indexes related to the time slice, and then perform further precise screening according to the comprehensive similarity.

[0038] During query, start from the starting node of the index and search along the edges according to the query conditions. Suppose the currently searched node is , and its set of neighbor nodes is . For each neighbor node , determine whether it meets the query conditions. If it meets, continue to search from ; if it does not meet, skip this node.

[0039] Exemplarily, taking the financial scenario as an example, the keyword set extracted from the hybrid query statement may include "cash out", "blurred ID photo", etc., and the natural language description may be "find customers with abnormal transaction amounts and mentions of cash out in transaction remarks in the past month", and the trend condition may be "the transaction amount has risen sharply in the short term". During the query, according to these conditions, layer-by-layer screening is performed in the multi-layer fusion index structure, and finally further screening is performed according to the calculated comprehensive similarity.

[0040] S400. Sort the screened data items according to the comprehensive similarity, calculate the statistical information, and display the results in a visual form.

[0041] Specifically, the steps of result processing and return include: Sort the screened data items according to the comprehensive similarity, and select the data items with the top rankings as the query results; Calculate the statistical information of the query results and organize them into a visual form for display.

[0042] Sort the screened data items according to the comprehensive similarity, and select the data items with the top rankings as the query results. Based on the query results, different modal data can be comprehensively calculated again to obtain the required statistical information and organize it into a visual form for display to the query user.

[0043] Taking the financial scenario as an example, according to the query results, the fraud risk scores of these data items can be calculated, using a logistic regression model , where is a comprehensive eigenvalue that integrates multi-modal and time series features (such as the degree of abnormality of transaction amounts, the degree of text keyword matching, the abnormal score of image features, and the time series trend features, etc.), is the corresponding weight, is the bias term. Finally, the query results are sorted out and formatted, and presented to the risk control personnel in a visual way, such as drawing a line chart of the number of transactions that meet the conditions changing over time, marking transactions with different risk levels with different colors, displaying the text information, pictures and corresponding risk scores of relevant transactions, etc., to help the risk control personnel intuitively understand the query results and make accurate anti-fraud decisions.

[0044] An embodiment of the present invention also provides a multi-modal time series hybrid query system, which is used to implement the above method embodiment, and the description that has been given will not be repeated here. The following terms "module", "unit", "sub-unit", etc. can be a combination of software and / or hardware that can achieve a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware or a combination of software and hardware is also possible and contemplated.

[0045] As Figure 3 shown, Figure 3 is a structural block diagram of the multi-modal time series hybrid query system in the present invention. The system includes: A data collection and preprocessing module 101, which is used to collect multi-modal time series data, perform normalization, denoising and feature extraction on numerical time series, perform cleaning, conversion and classification feature extraction on text data, and perform enhancement and feature extraction on picture data; Specifically, the data collection and preprocessing module 101 is used for: Performing normalization processing on numerical time series and performing feature extraction on the denoised time series; Cleaning text data, removing special characters and stop words, and converting the cleaned text into a vector representation; Performing image enhancement processing on picture data and performing image feature extraction using a convolutional neural network.

[0046] An index construction module 102, which is used to convert the statistical features of numerical time series into vectors, calculate the similarity of different modalities, construct a comprehensive similarity metric, and construct a nearest neighbor graph index and a multi-layer index structure based on this metric; Specifically, the index construction module 102 is used for: Calculating text-image similarity, text-time series similarity and image-time series similarity, and calculating a comprehensive similarity metric based on weighted average; Constructing a nearest neighbor graph index, where each data item is connected to its k most similar data items, and the weight of the edge is the comprehensive similarity metric; Constructing a multi-layer index structure, dividing time slices according to time granularity, and constructing sub-indexes within each time slice.

[0047] A query parsing and optimization module 103, which is used to parse the mixed query statement input by the user, extract multi-modal keywords, natural language descriptions, time ranges and trend conditions, convert the natural language description into a multi-modal feature vector, and perform screening in combination with time series conditions; Specifically, the query parsing and optimization module 103 is used for: Extracting multi-modal keywords, natural language descriptions, time ranges and trend conditions in the user query statement; Convert the natural language description into a multi-modal feature vector using a pre-trained model, and calculate its similarity with the feature vectors of different modal data; Combine the time series conditions and the multi-level index structure, first perform a quick screening, and then perform an accurate screening.

[0048] The result processing and return module 104 is used to sort the selected data items according to the comprehensive similarity, calculate statistical information, and display the results in a visual form.

[0049] Specifically, the result processing and return module 104 is used for: Sort the selected data items according to the comprehensive similarity, and select the top-ranked data items as the query results; Calculate the statistical information of the query results and organize them into a visual form for display.

[0050] Figure 4 It is a structural block diagram of the electronic device provided by the embodiment of the present invention. As Figure 4 shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630, and a communication bus 640. Among them, the processor 610, the communication interface 620, and the memory 630 complete mutual communication through the communication bus 640. The processor 610 can call the logical instructions in the memory 630 to execute the following methods: Collect multi-modal time series data, normalize, denoise, and extract features from numerical time series, clean, transform, and extract classification features from text data, and enhance and extract features from image data; Convert the statistical features of the numerical time series into vectors, calculate the similarities of different modalities, construct a comprehensive similarity metric, and build a nearest neighbor graph index and a multi-level index structure based on this metric; Parse the mixed query statement input by the user, extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions, convert the natural language descriptions into multi-modal feature vectors, and perform screening in combination with time series conditions; Sort the selected data items according to the comprehensive similarity, calculate statistical information, and display the results in a visual form.

[0051] In addition, when the logical instructions in the above-mentioned memory 630 are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: various media such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs that can store program codes.

[0052] The embodiments of the present invention also provide a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is configured to execute the methods provided in the above-mentioned various embodiments, for example, including: Collect multi-modal time series data, normalize, denoise, and extract features from numerical time series, clean, transform, and extract classification features from text data, and enhance and extract features from image data; Convert the statistical features of numerical time series into vectors, calculate the similarity of different modalities, construct a comprehensive similarity metric, and build a nearest neighbor graph index and a multi-layer index structure based on this metric; Parse the mixed query statement input by the user, extract multi-modal keywords, natural language descriptions, time ranges, and trend conditions, convert the natural language description into a multi-modal feature vector, and perform screening in combination with time series conditions; Sort the filtered data items according to the comprehensive similarity, calculate statistical information, and display the results in a visual form.

[0053] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical discs, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments or some parts of the embodiments.

[0054] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multimodal time series hybrid query system, characterized in that: The multimodal time series hybrid query system comprises: The data collection and preprocessing module is used to collect multimodal time series data, and normalize, denoise and extract features for numerical time series, clean, convert and extract classification features for text data, and enhance and extract features for image data; The index building module is used to convert the statistical features of numerical time series into vectors, calculate the similarity of different modes, build a comprehensive similarity metric, and build a neighbor graph index and a multi-layer index structure based on the metric; The query parsing and optimization module is used to parse the mixed query statements input by users, extract multimodal keywords, natural language descriptions, time ranges and trend conditions, convert natural language descriptions into multimodal feature vectors, and perform screening in combination with time series conditions; The result processing and return module is used to sort the filtered data items according to comprehensive similarity, calculate statistical information, and display the results in a visual form.

2. The multimodal time series hybrid query system according to claim 1, characterized in that: The data collection and preprocessing module is specifically used for: Normalize the numerical time series and extract features from the denoised time series; Clean the text data, remove special characters and stop words, and convert the cleaned text into a vector representation; The image data is enhanced and the convolutional neural network is used to extract image features.

3. The multimodal time series hybrid query system according to claim 1, characterized in that: The index building module is specifically used for: Compute text-image similarity, text-time series similarity, and image-time series similarity, and calculate a comprehensive similarity measure based on weighted average; Construct a neighbor graph index, where each data item is connected to its k most similar data items, and the edge weight is a comprehensive similarity measure; Build a multi-layer index structure, divide time slices according to time granularity, and build a sub-index in each time slice.

4. The multimodal time series hybrid query system according to claim 1, characterized in that: The query parsing and optimization module is specifically used for: Extract multimodal keywords, natural language descriptions, time ranges, and trend conditions from user query statements; Use the pre-trained model to convert the natural language description into a multimodal feature vector and calculate its similarity with the feature vectors of different modal data; Combine time series conditions and multi-layer index structure to perform quick screening first and then precise screening.

5. The multimodal time series hybrid query system according to claim 1, characterized in that: The result processing and returning module is specifically used for: Sort the filtered data items by comprehensive similarity, and select the top-ranked data items as the query results; Calculates statistics of query results and organizes them into visual displays.

6. A multimodal time series hybrid query method, characterized in that: The following steps are involved: Collect multimodal time series data, normalize, denoise and extract features for numerical time series, clean, convert and extract classification features for text data, and enhance and extract features for image data; Convert the statistical features of numerical time series into vectors, calculate the similarity of different modes, construct a comprehensive similarity metric, and build a neighbor graph index and multi-layer index structure based on the metric; Parse the mixed query statements input by users, extract multimodal keywords, natural language descriptions, time ranges and trend conditions, convert natural language descriptions into multimodal feature vectors, and filter them in combination with time series conditions; The filtered data items are sorted by comprehensive similarity, statistical information is calculated, and the results are displayed in a visual form.

7. The multimodal time series hybrid query method according to claim 6, characterized in that: The step of constructing a comprehensive similarity measure comprises: Compute text-image similarity, text-time series similarity, and image-time series similarity; The comprehensive similarity measure is calculated based on weighted average, and the weights are determined based on experiments and domain knowledge.

8. The multimodal time series hybrid query method according to claim 6, characterized in that: The steps of constructing a multi-layer index structure include: Divide time slices by time granularity and build sub-indexes in each time slice; When querying, first quickly locate the corresponding time slice index according to the time range, and then perform accurate similarity matching within the time slice.

9. The multimodal time series hybrid query method according to claim 6, characterized in that: The step of parsing the user query statement comprises: Extract multimodal keywords, natural language descriptions, time ranges, and trend conditions; The pre-trained model is used to convert the natural language description into a multimodal feature vector, and its similarity with the feature vectors of different modal data is calculated.

10. The multimodal time series hybrid query method according to claim 6, characterized in that: The steps of processing and returning the result include: Sort the filtered data items by comprehensive similarity, and select the top-ranked data items as the query results; Calculates statistics of query results and organizes them into visual displays.

Images