A large model-based multi-modal data intelligent chart query method and system

Abstract

This invention discloses a method and system for intelligent chart querying based on a large model of multimodal data. The method includes acquiring fuzzy query information input by the user and real-time monitoring data of a machine tool; performing intent recognition and entity extraction on the fuzzy query information, parsing it into structured query conditions; establishing a multimodal data association library based on a large model, inputting the real-time monitoring data of the machine tool into the multimodal data association library, and storing it according to preset association requirements; inputting the structured query conditions into the established multimodal data association library; and generating interactive charts based on the output information of the multimodal data association library, embedding key conclusion summaries. This application utilizes a large model to understand user intent and dynamically recommend related charts, breaking through the traditional fixed menu navigation mode and realizing intelligent interaction through joint analysis of data and charts.

Description

Technical Field

[0001] This invention relates to the field of industrial intelligence and CNC machining technology, and in particular to a method and system for intelligent chart querying of multimodal data based on a large model. Background Technology

[0002] Traditional data analysis for rolling mills relies on professionals manually querying charts (such as vibration spectra and Campbell's diagrams) using specialized software, which presents the following pain points: Complex operation: Requires familiarity with technical terminology and tool interfaces, making it difficult for non-technical personnel to operate independently; Response delay: Manual data filtering is time-consuming and cannot respond to dynamic operating conditions in real time; Knowledge silos: Data and charts are scattered across different modules, lacking a unified interaction interface. While existing systems support chart generation, they lack natural language-driven intelligent interaction capabilities, failing to achieve end-to-end automation of the "intent-data-chart" process. Summary of the Invention

[0003] Therefore, the purpose of this invention is to provide a multimodal data intelligent chart query method and system based on a large model. It adopts the method of understanding user intent through a large model, dynamically recommending related charts, breaking through the traditional fixed menu navigation mode, and realizing intelligent interaction of data and chart joint analysis.

[0004] To achieve the above objectives, this invention provides a multimodal data intelligent chart query method based on a large model, comprising the following steps: S1. Obtain fuzzy query information input by the user and real-time monitoring data of the machine tool; S2. Perform intent recognition and entity extraction on the fuzzy query information, and parse it into structured query conditions; S3. Establish a multimodal data association library based on a large model, input the real-time monitoring data of the machine tool into the multimodal data association library, and store it according to the preset association requirements; S4. Input the structured query conditions into the established multimodal data association database, generate interactive charts based on the output information of the multimodal data association database, and embed key conclusion summaries.

[0005] Furthermore, in S2, the fuzzy query information is subjected to intent recognition and entity extraction, and parsed into structured query conditions; including: A multi-head attention mechanism is used to perform intent recognition and entity extraction on fuzzy query information input by users; The multi-head attention mechanism includes a shared coding layer, an intent recognition branch, and an entity extraction branch; The input is represented as user query text, which is then segmented and converted into a word embedding sequence. Perform single-head attention calculation and multi-head concatenation and linear transformation on the input word embedding sequence; The loss function is calculated using an intent recognition branch and an entity extraction branch, respectively. The total loss is calculated, and the correlation between the intent recognition and entity recognition tasks is jointly trained to achieve high-precision structured query condition parsing; thus forming SQL query statements.

[0006] Furthermore, it also includes the development of a self-learning optimization mechanism: recording high-frequency user queries and user feedback, continuously optimizing the domain knowledge base and chart generation rules of the large model, and improving query accuracy.

[0007] Furthermore, the multimodal data association library includes machine tool CNC system data, sensor time domain graph data, sensor frequency domain graph data, workpiece quality data, and expert experience, which are associated and mapped according to data association requirements; The machine tool CNC system data includes: position data, speed data, tool data, load data, and pressure data; The sensor time-domain data includes: vibration sensor data and temperature sensor data; The sensor frequency domain data includes: a spectrum diagram and a Campbell diagram formed after frequency domain conversion of the vibration sensor data; The workpiece quality data includes: dimensional accuracy data, surface roughness data, quality assessment results, and quality index data; The expert experience includes: equipment maintenance experts' experience in troubleshooting and repair, and process optimization experts' experience data.

[0008] Furthermore, the multimodal data association library also includes hierarchical storage of all data and the establishment of an association mapping table for the hierarchically stored data; After hierarchical storage, the multimodal data association library includes: The structured data layer is used to store relational data, including: basic machine tool information, process parameters, and production data; The time-series data layer is used to store real-time data from the sensors and frequency domain data after Fourier transform in a time-series manner. The unstructured data layer is used to store expert experience, including troubleshooting rules and process optimization strategies formed based on expert experience.

[0009] Optionally, the association mapping according to data association requirements includes: Establish a direct relationship table, which includes: machine tool operating status table, sensor data table, and workpiece quality table; Establish an indirect data table, which includes a maintenance record table and a process parameter optimization table; Establish foreign key relationships between various data tables using machine tool ID and timestamp; Define indirect influencing factors and dynamically calculate the correlation strength using a statistical model.

[0010] Furthermore, establishing foreign key relationships between various data tables using machine tool IDs and timestamps includes: By using the machine tool ID and timestamp, the load data is associated with the vibration sensor data to generate a time domain graph or frequency domain graph corresponding to the load data; Based on a predefined rule base using expert experience, establish the association between query conditions and corresponding type charts, and establish the association between chart threshold data and fault causes based on monitoring thresholds; Train a supervised model, inputting maintenance records, process parameters, and load data into the supervised model, and outputting the weights of indirect impacts on workpiece quality. Define the nodes and edges of the fault phenomena and causes, establish causal relationships, construct a Bayesian network, and verify the causal strength through historical data.

[0011] This invention also provides a multimodal data intelligent chart query system based on a large model, comprising the steps of implementing the above-mentioned multimodal data intelligent chart query method based on a large model, including: The data acquisition module is used to acquire real-time monitoring data of the machine tool; The user input module is used to obtain fuzzy query information input by the user; The information extraction module is used to perform intent recognition and entity extraction on the fuzzy query information and parse it into structured query conditions. The database creation module stores machine tool CNC system data, sensor time domain data, sensor frequency domain data, workpiece quality data, and expert experience based on the large model, according to the preset correlation requirements. The chart interaction module takes structured query conditions as input into the established multimodal data association library, generates interactive charts based on the output information of the multimodal data association library, and embeds key conclusion summaries.

[0012] The present invention also provides an electronic device, comprising: Memory, which stores computer program instructions; The processor, when the computer program instructions are executed by the processor, implements the steps of the above-described intelligent chart query method for multimodal data based on a large model.

[0013] The present invention also provides a computer-readable storage medium for storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described intelligent chart query method for multimodal data based on a large model.

[0014] The method and system for intelligent chart querying of multimodal data based on large models disclosed in this application adopt the approach of understanding user intent through large models and dynamically recommending related charts, breaking through the traditional fixed menu navigation mode and realizing intelligent interaction of joint data and chart analysis.

[0015] This application adjusts the level of detail in the charts based on user needs and current operating conditions. By recording high-frequency user queries and feedback, it continuously optimizes the domain knowledge base and chart generation rules of the large model to improve query accuracy.

[0016] This application reduces chart query time from minutes to seconds, and lowers operator training costs by 50%; through multi-chart correlation analysis, the accuracy of identifying the root cause of anomalies is increased to over 90%; and implicit experience is transformed into explicit chart rules, supporting new employees to get started quickly. Attached Figure Description

[0017] Figure 1 This is a flowchart illustrating the intelligent chart query method for multimodal data based on a large model according to the present invention.

[0018] Figure 2 This is a structural block diagram of the multimodal data intelligent chart query system based on a large model provided by the present invention; Figure 3 This is a schematic diagram of the chart interaction output after a user asks a question in Embodiment 1 of the present invention.

[0019] Figure 4 This is a schematic diagram of the chart interaction output after a user asks a question in Embodiment 2 of the present invention.

[0020] Figure 5 This is a schematic diagram of the workpiece monitoring chart interaction corresponding to the first abnormal data in Embodiment 2 of the present invention.

[0021] Figure 6 This is a summary diagram of the possible causes and solutions for abnormal data in Embodiment 2 of the present invention. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0023] like Figure 1 As shown in the figure, one embodiment of the present invention provides a method for intelligent chart querying of multimodal data based on a large model, which includes the following steps: S1. Obtain the fuzzy query information input by the user and the real-time monitoring data of the machine tool; it should be noted that the real-time monitoring data of the machine tool in this application is stored in real time. Therefore, in subsequent data analysis, historical data is output and analyzed based on the time range of the user's query. The fuzzy query information input by the user can be classified according to intent, mainly including: data query, fault analysis, and optimization prediction. The data query class outputs data based on the entity queried by the user (e.g., "trend chart of abnormal fluctuations in rolling pressure in the last two hours"), performs intent recognition and entity extraction, and parses it into structured query conditions: time range, monitoring items, and chart type.

[0024] The fault analysis category retrieves the corresponding time-domain graph and frequency spectrum graph based on the user's query, extracts abnormal components from the graph, such as "the surface roughness of the workpieces processed in the past two days has increased, is there an abnormal cutting load on the machine tool?" Extracting structured query conditions: time range, surface roughness, abnormal cutting load; retrieving the time-domain graph and frequency-domain graph of the sensor during that period, extracting abnormal signals in the time-domain graph or frequency-domain graph such as increased or decreased amplitude, appearance of high-frequency components, etc., for analysis, and providing adjustment suggestions.

[0025] The optimization prediction function retrieves the corresponding time-domain graph and spectrum graph based on the user's query, extracting abnormal components from the graphs. For example, when the wear of guide rails and lead screws is severe, can the wear trend be predicted by analyzing machine tool data? The structured query conditions are extracted as guide rail, lead screw, wear, machine tool data, and predicted wear trend. Based on the load data fluctuations monitored by sensors, reflecting changes in frictional resistance, and the high-frequency components of vibration data in the frequency domain graph, the relationship between abnormal changes in frictional resistance and vibration and the degree of wear is analyzed. Based on historical and current data, future wear trends are predicted.

[0026] S2. Perform intent recognition and entity extraction on the fuzzy query information, and parse it into structured query conditions; further, perform intent recognition and entity extraction on the fuzzy query information, and parse it into structured query conditions; including: A multi-head attention mechanism is used to perform intent recognition and entity extraction on fuzzy query information input by users; The multi-head attention mechanism includes a shared coding layer, an intent recognition branch, and an entity extraction branch; The structured query conditions are generated according to the preset template, and then the SQL query statement is generated.

[0027] Furthermore, it also includes the development of a self-learning optimization mechanism: recording high-frequency user queries and user feedback, continuously optimizing the domain knowledge base and chart generation rules of the large model, and improving query accuracy.

[0028] This application employs a deep learning model based on a multi-head attention mechanism. It constructs an intent recognition branch and an entity extraction branch on top of a shared encoding layer, achieving structured parsing of fuzzy user query information through joint training. The model framework consists of three parts: a shared encoding layer, a multi-head attention layer, and a dual-task branch.

[0029] The shared encoding layer provides consistent input features for all attention heads, but each attention head projects the shared features from a different perspective through independent linear transformations (query, key, and value weight matrices).

[0030] The multi-head attention layer extracts semantic features from different dimensions through a multi-head attention mechanism.

[0031] The dual-task branch includes intent recognition (classification task) and entity extraction (sequence labeling task).

[0032] The input is represented as user query text, which is then segmented and converted into a word embedding sequence. ; Where n is the sequence length and d is the embedding dimension. Multi-head attention splits the embedding into h independent attention heads, each focusing on different semantic features.

[0033] Generate query (Q), key (K), and value (V) through linear transformation. ; in, .

[0034] Single-head attention computation .

[0035] Multi-head splicing and linear transformation: , in, This is the output weight matrix.

[0036] The intent recognition branch implemented through the dual-task branch is as follows: Embedded representation of input CLS Token , Classification layer , Loss function: cross-entropy loss .

[0037] The entity extraction branch implemented with dual task branches is as follows: Input all token embeddings , Sequence labeling layer , Loss function: CRF negative log-likelihood .

[0038] Joint training and loss balancing: The total loss is a weighted sum of the two tasks, and stability is trained by dynamically balancing the weights. The initial values ​​a=b=1 are dynamically adjusted during training based on task difficulty. In the shared encoding layer, the two branches are trained independently, while the classification / labeling layer parameters are shared. After the model outputs the intent category and entity label, it is mapped to a structured SQL statement according to a preset template. A multi-head attention mechanism enhances the model's ability to capture complex semantics, and a joint training strategy balances the correlation between intent and entity tasks, ultimately achieving high-precision structured query condition parsing.

[0039] S3. Establish a multimodal data association library based on a large model, input the real-time monitoring data of the machine tool into the multimodal data association library, and store it according to the preset association requirements; Furthermore, the multimodal data association library includes machine tool CNC system data, sensor time domain graph data, sensor frequency domain graph data, workpiece quality data, and expert experience, which are associated and mapped according to data association requirements; The machine tool CNC system data includes: position data, speed data, tool data, load data, and pressure data; The sensor time-domain data includes: vibration sensor data and temperature sensor data; The sensor frequency domain data includes: a spectrum diagram and a Campbell diagram formed after frequency domain conversion of the vibration sensor data; The workpiece quality data includes: dimensional accuracy data, surface roughness data, quality assessment results, and quality index data; The expert experience includes: equipment maintenance experts' experience in troubleshooting and repair, and process optimization experts' experience data.

[0040] Furthermore, the multimodal data association library also includes hierarchical storage of all data and the establishment of an association mapping table for the hierarchically stored data; After hierarchical storage, the multimodal data association library includes: The structured data layer is used to store relational data, including: basic machine tool information, process parameters, and production data; The time-series data layer is used to store real-time data from the sensors and frequency domain data after Fourier transform in a time-series manner. The unstructured data layer is used to store expert experience, including troubleshooting rules and process optimization strategies formed based on expert experience.

[0041] Optionally, the association mapping according to data association requirements includes: Establish a direct relationship table, which includes: machine tool operating status table, sensor data table, and workpiece quality table; Establish an indirect data table, which includes a maintenance record table and a process parameter optimization table; Establish foreign key relationships between various data tables using machine tool ID and timestamp; Define an indirect influence factor and dynamically calculate the association strength using a statistical model. The indirect influence factor is used to measure the indirect association between parameters that are not directly related, arising from mediating factors or historical data patterns.

[0042] Specifically, the parameter correlations measured include indirect causal correlations and statistical dependency correlations. Indirect causal correlations include, for example, the relationship between maintenance records (such as lubrication frequency) and workpiece quality. Maintenance records themselves do not directly affect quality, but indirectly affect workpiece surface roughness or dimensional accuracy by influencing machine tool stability (such as reducing vibration or wear). Another example is the correlation between process parameter optimization (such as cutting speed adjustment) and failure rate; parameter optimization may indirectly reduce the probability of abnormal vibration or temperature rise by reducing tool load fluctuations. Statistical dependency correlations include, for example, the correlation between load data and sensor frequency domain characteristics (such as high-frequency components in a spectrum). Load changes may indirectly lead to frequency domain data anomalies by inducing machine tool resonance or mechanical stress. Finally, there is the correlation between historical process parameters and expert experience rules; statistical models are used to analyze the implicit relationship between historical parameter settings and expert-recommended optimization strategies, dynamically adjusting the correlation strength.

[0043] Association strength is used to characterize the degree of indirect influence of parameters in the indirect association mapping table on the target output object (workpiece quality indicators). It is quantified by the absolute value of the weights. The specific implementation method of dynamically calculating association strength through a statistical model is as follows: the historical data sequence of each parameter in the indirect association mapping table is used as the input feature vector of the statistical model, and the measured value of the target output object within the corresponding time window is used as the model output label. Regression analysis (such as linear regression, support vector regression) or probabilistic inference methods are used to solve the weight coefficient of each input feature on the target output. This weight coefficient is the quantitative characterization value of association strength. At the same time, a dynamic weight update mechanism is established. The model is retrained according to the new data at a preset period, so that the association strength is continuously and adaptively updated with the evolution of working conditions, rather than being statically fixed. Meanwhile, the association strength value is directly updated and stored in the indirect association mapping table as the core basis for data association mapping: the system dynamically constructs a data index based on the value of this field. When the association strength exceeds a preset threshold, a strong association path is established to ensure that when the user performs a fuzzy query, the system can prioritize and quickly retrieve and jointly analyze highly correlated data across tables based on this mapping relationship, achieving a second-level response from intent to chart.

[0044] Furthermore, establishing foreign key relationships between various data tables using machine tool IDs and timestamps includes: By using the machine tool ID and timestamp, the load data is associated with the vibration sensor data to generate a time domain graph or frequency domain graph corresponding to the load data; Based on a predefined rule base using expert experience, establish the association between query conditions and corresponding type charts, and establish the association between chart threshold data and fault causes based on monitoring thresholds; The supervised model is trained by inputting maintenance records, process parameters, and load data, and outputting indirect influence weights on workpiece quality. These indirect influence weights quantify the contribution of different factors to workpiece quality, serving as a basis for fault attribution ranking. Specifically, the system uses a linear regression model as the basic example of the supervised model to quantify indirect influence factors. For example, when measuring workpiece surface roughness, the following statistical model formula is established: Workpiece surface roughness = β0 + β1 × lubrication frequency + β2 × cutting speed + ϵ. Where β1 and β2 are the weights of indirect influence factors, and ϵ is the error term. The system collects machining datasets from several months of historical data and uses the least squares method to iteratively solve for the weight values. The larger the absolute value of the calculated weight, the stronger the correlation. For example, if β1 = 0.85 is obtained, it means that the influence weight of lubrication frequency on workpiece surface roughness is as high as 85%, that is, the influence of surface lubrication frequency on roughness is dominant. The system also establishes a dynamic weight update mechanism, triggering model retraining monthly based on newly added processing data to adaptively update weight parameters. This ensures that the quantification of indirect correlations is not statically fixed but continuously evolves with the changing operating conditions. It should be understood that although this embodiment uses linear regression as an example, in other operating conditions with stronger nonlinear characteristics, the supervised model can be replaced with support vector regression or a deep neural network, as long as it can output the indirect influence weights.

[0045] Establish causal relationships, construct Bayesian networks, and verify the strength of causality using historical data. By defining nodes and edges, expert experience, such as "instability in the hydraulic system may lead to workpiece quality problems," is transformed into a computable probabilistic relationship model, making tacit knowledge explicit.

[0046] The system defines nodes and edges for fault phenomena and their causes, establishes causal relationships, constructs a Bayesian network, and verifies the causal strength using historical data. The causal strength specifically represents the posterior probability of a fault cause occurring given an observed phenomenon. This probability value is stored as an inference result in an expert experience rule base and directly input into the interactive chart module. While generating interactive charts, the system automatically calls upon this probability value to embed a summary of key conclusions. For example: "An abnormal vibration was detected. Based on the causal model, the probability of hydraulic system instability is 92%, and it is recommended to prioritize checking the hydraulic circuit." This achieves end-to-end automation from data association to intelligent decision-making recommendations. The Bayesian network is the core defense line for transforming implicit experience into a computable probabilistic model. For common fault scenarios, such as "hydraulic system instability leading to workpiece quality problems," the system constructs a Bayesian network as follows: First, define the fault cause node C (e.g., hydraulic system instability) and the fault phenomenon node E (e.g., pressure deviation and surface ripples). Second, initialize the conditional probability through expert experience; for example, set P(E|C) = 0.92, indicating that the probability of pressure deviation and surface ripples occurring under hydraulic system instability is 92%. Subsequently, update the probability distribution based on historical fault data and calculate the posterior probability using the Bayesian formula: P(C|E) = P(E|C) × P(C) / P(E). When new abnormal data is input, the system dynamically updates the occurrence probability of the cause node C according to this formula, achieving causal inference by deducing the root cause from the phenomenon.

[0047] To verify whether the causal model constructed by the Bayesian network is effective and not overfitting, the system further verifies the causal strength using historical data, specifically employing KL divergence for constraint verification. The system calculates the KL divergence between the predicted distribution and the historical true distribution. If the threshold condition of DKL(P_hist‖P_pred) < 0.05 is met, the causal model is deemed effective. A KL divergence less than 0.05 means that the probability distribution of the failure causes predicted by the model almost coincides with the historical actual distribution, proving the statistical reliability of the causal inference. This embodiment, through the aforementioned supervised model and the deep computational system of the Bayesian network, enables the precise quantification and dynamic inference of implicit associations that were originally impossible to obtain through direct table lookup (such as the 85% indirect influence of lubrication frequency on roughness and the 92% causal probability of hydraulic instability on anomalies). This is the key mechanism for achieving a root cause analysis accuracy of over 90%.

[0048] S4. Input the structured query conditions into the established multimodal data association database, generate interactive charts based on the output information of the multimodal data association database, and embed key conclusion summaries.

[0049] This invention also provides a multimodal data intelligent chart query system based on a large model, comprising the steps of implementing the above-mentioned multimodal data intelligent chart query method based on a large model, including: The data acquisition module is used to acquire real-time monitoring data of the machine tool; The user input module is used to obtain fuzzy query information input by the user; The information extraction module is used to perform intent recognition and entity extraction on the fuzzy query information and parse it into structured query conditions. The database creation module stores machine tool CNC system data, sensor time domain data, sensor frequency domain data, workpiece quality data, and expert experience based on the large model, according to the preset correlation requirements. The chart interaction module takes structured query conditions as input into the established multimodal data association library, generates interactive charts based on the output information of the multimodal data association library, and embeds key conclusion summaries.

[0050] The present invention also provides an electronic device, comprising: Memory, which stores computer program instructions; The processor, when the computer program instructions are executed by the processor, implements the steps of the above-described intelligent chart query method for multimodal data based on a large model.

[0051] The present invention also provides a computer-readable storage medium for storing instructions that, when executed on a computer, cause the computer to perform the steps of the above-described intelligent chart query method for multimodal data based on a large model. Example 1

[0052] This application categorizes the data intent in the fuzzy query information input by the user. If it belongs to the data query category, such as a query for production data, when the entity in the user's query contains data for a specific time period (production / monitoring), such as: production data from the day before yesterday / yesterday / today: like Figure 3 As shown, the production data report has the following statistical dimensions: 1 day + 1 machine tool + 1 part number. That is, it counts the number of parts a produced on machine tool A on that day; the production quantity is counted according to the total number of processing operations on that day; the start time is recorded according to the start time of the first workpiece of that part produced on that machine tool on that day; the end time is recorded according to the start time of the last workpiece of that part produced on that machine tool on that day; the list is arranged in ascending order of the start time of a particular machine tool.

[0053] Chart data expectations: If there is no production data for a given day / time, it should be displayed that there was no production, for example, "No machine tool production data on February 6, 2025". A user-friendly prompt should be given when data output exceeds the limits. Example 2

[0054] This application also includes the ability to directly query production curves. For example, the query information is: the production process curve of '** part' processed on '** machine tool (machine tool ID)' the day before yesterday / yesterday / today at ** step / (or all steps).

[0055] The interactive chart display results are as follows Figure 4 As shown, the production process curve for machining workpiece b on machine tool B at step 1 (or all steps) on February 28, 2025 is as follows: (Note: Production process curves, problems, and solutions are displayed for each abnormal part.) The production process curves for parts experiencing processing abnormalities include: [Example 1] Figure 5 The production process curve for the 170th workpiece is shown. Figure 5 The rightmost axis represents the amount of vibration.

[0056] Finally, the following conclusions are drawn based on the analysis of potential problems and solutions: Duplication of monitored items for the same machine tool, part number, and machining operation number on the same day is required. Possible causes and solutions are summarized as follows: Figure 6 As shown. Example 3

[0057] This application can also be categorized according to user roles. For example, the production department mainly focuses on production volume data, while the process department mainly focuses on workpiece quality data. When the process department asks for abnormal data from the day before yesterday / yesterday / today... Prototype Notes: Questions containing key information will output the corresponding content. The large language model needs to understand times such as "the day before yesterday," "yesterday," and "today," and can also understand replacing specific time parameters, such as abnormal data for February 6, 2025. Abnormal Data Report: Data statistics dimensions are 1 day + 1 machine tool + 1 part number + 1 monitoring item. Specifically, it counts data where each monitoring item exceeds the upper or lower limit for part a produced on machine tool A on that day. Alarm times are recorded according to the first alarm time generated by a specific monitoring item exceeding the tolerance for that part number on that machine tool on that day. The list order is in ascending order of alarm times for the same part on the same machine tool.

[0058] Chart data expectations: If there is no abnormal data for a given day / time, it should be indicated that there is no abnormality. For example, "There is no abnormal data for machine tools on February 6, 2025." A user-friendly prompt should be given when data output exceeds the limits.

[0059] Question: Generate a Campbell's plot of the order of the 'part name' processed on 'Machine Tool ID' at 'process step' for the past two weeks. Specifically, generate the Campbell's plot of the order of machining 'b' on Machine Tool B at process step 1 from February 15, 2025 to February 28, 2025. Analysis revealed that the 111th processed workpiece exhibited excessive amplitude at 08:00.40.000 on February 16, 2025. Further analysis revealed an out-of-tolerance issue at order 33. Example 4

[0060] When the user inputs a fuzzy query for fault analysis, such as: "Recently we have found that the surface roughness of the machined workpieces has increased. Is there an abnormality in the cutting load of the machine tool?" Large model application answer: By analyzing the cutting load data in the CNC system of the machine tool, we found that the cutting load has fluctuated greatly recently, especially in the high-speed cutting stage.

[0061] By combining the time-domain plot of the sensor data, it was observed that the cutting force changes with time in an unstable trend.

[0062] The frequency domain plot shows an increase in high-frequency components in the cutting force, which may indicate accelerated tool wear or improper cutting parameter settings.

[0063] It is recommended to check the tool wear and optimize the cutting parameters to reduce the surface roughness of the workpiece.

[0064] For example, if a machine tool vibrates abnormally during processing, does it affect the quality of the workpiece? Large model application answer: By analyzing the feed force data in the CNC system of the machine tool, we found that the feed force fluctuated significantly during vibration.

[0065] The time-domain plot of the sensor data shows that vibration caused a significant increase in amplitude.

[0066] The frequency domain plot further revealed the frequency components of the vibration, which were close to the machine tool's natural frequency, possibly triggering resonance.

[0067] This vibration has a negative impact on the quality of the workpiece, such as surface ripples and decreased dimensional accuracy.

[0068] It is recommended to adjust the machine tool parameters to avoid operating near the resonant frequency and to enhance the stability of the machine tool. Example 5

[0069] When the user's fuzzy query is related to optimization and prediction, such as: Question: We noticed that after a long period of operation, the guide rails and lead screws of a certain machine tool showed significant wear. Is it possible to predict wear trends and develop a preventative maintenance plan by analyzing machine tool data? Large model application answer: By analyzing the load and pressure data, as well as the time-domain and frequency-domain graphs of sensor data in the CNC system of a machine tool, we can monitor the wear of the guide rails and lead screws.

[0070] Fluctuations in load data can reflect changes in frictional resistance of the guide rail and lead screw during motion.

[0071] The time-domain plot of the sensor data can show the vibration of the guide rail and lead screw, while the frequency-domain plot can reveal the frequency components of the vibration, which are closely related to the wear of the guide rail and lead screw.

[0072] It is recommended to establish a wear prediction model for guide rails and lead screws, and predict future wear trends based on historical and current data.

[0073] Develop preventative maintenance plans based on the forecast results, such as regular lubrication, adjusting the clearance of guide rails and lead screws, and replacing severely worn parts.

[0074] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A method for intelligent chart querying of multimodal data based on a large model, characterized in that, Includes the following steps: Acquire fuzzy query information input by the user and real-time monitoring data of the machine tool; The fuzzy query information is subjected to intent recognition and entity extraction, and parsed into structured query conditions; Establish a multimodal data association library based on a large model, input the real-time monitoring data of the machine tool into the multimodal data association library, and store it according to the preset association requirements; The structured query conditions are input into the established multimodal data association database. Based on the output information of the multimodal data association database, interactive charts are generated and key conclusion summaries are embedded.

2. The intelligent chart query method for multimodal data based on a large model according to claim 1, characterized in that, In S2, the fuzzy query information is subjected to intent recognition and entity extraction, and parsed into structured query conditions; including: A multi-head attention mechanism is used to perform intent recognition and entity extraction on fuzzy query information input by users; The multi-head attention mechanism includes a shared coding layer, an intent recognition branch, and an entity extraction branch; The input is represented as user query text, which is then segmented and converted into a word embedding sequence. Perform single-head attention calculation and multi-head concatenation and linear transformation on the input word embedding sequence; The loss function is calculated using an intent recognition branch and an entity extraction branch, respectively. The total loss is calculated, and the correlation between the intent recognition and entity recognition tasks is jointly trained to achieve high-precision structured query condition parsing; thus forming SQL query statements.

3. The intelligent chart query method for multimodal data based on a large model according to claim 2, characterized in that, It also includes developing a self-learning optimization mechanism: recording high-frequency user queries and user feedback, continuously optimizing the domain knowledge base and chart generation rules of the large model, and improving query accuracy.

4. The intelligent chart query method for multimodal data based on a large model according to claim 1, characterized in that, The multimodal data association library includes machine tool CNC system data, sensor time domain graph data, sensor frequency domain graph data, workpiece quality data, and expert experience, which are associated and mapped according to data association requirements; The machine tool CNC system data includes: position data, speed data, tool data, load data, and pressure data; The sensor time-domain data includes: vibration sensor data and temperature sensor data; The sensor frequency domain data includes: a spectrum diagram and a Campbell diagram formed after frequency domain conversion of the vibration sensor data; The workpiece quality data includes: dimensional accuracy data, surface roughness data, quality assessment results, and quality index data; The expert experience includes: equipment maintenance experts' experience in troubleshooting and repair, and process optimization experts' experience data.

5. The intelligent chart query method for multimodal data based on a large model according to claim 3, characterized in that, The multimodal data association library also includes hierarchical storage of all data and the establishment of an association mapping table for the hierarchically stored data; After hierarchical storage, the multimodal data association library includes: The structured data layer is used to store relational data, including: basic machine tool information, process parameters, and production data; The time-series data layer is used to store real-time data from the sensors and frequency domain data after Fourier transform in a time-series manner. The unstructured data layer is used to store expert experience, including troubleshooting rules and process optimization strategies formed based on expert experience.

6. The intelligent chart query method for multimodal data based on a large model according to claim 3 or 4, characterized in that, The process of performing association mapping according to data association requirements includes: Establish direct association tables, which include: machine tool operating status table, sensor data table, and workpiece quality table; Foreign key relationships are established between directly related tables using machine tool ID and timestamp to enable direct data linkage; Establish an indirect association mapping table, which includes a maintenance record table and a process parameter optimization table; The association strength between indirect association mapping tables is calculated using weighted association rules, including defining indirect influence factors and dynamically calculating association strength using statistical models.

7. The intelligent chart query method for multimodal data based on a large model according to claim 6, characterized in that, The method of establishing foreign key relationships between directly related tables using machine tool ID and timestamp includes: By using the machine tool ID and timestamp, the load data is associated with the vibration sensor data to generate a time domain graph or frequency domain graph corresponding to the load data; Based on a predefined rule base using expert experience, establish the association between query conditions and corresponding type charts, and establish the association between chart threshold data and fault causes based on monitoring thresholds.

8. The intelligent chart query method for multimodal data based on a large model according to claim 6, characterized in that, The calculation of the association strength between indirect association mapping tables using weighted association rules includes: Train a supervised model, inputting maintenance records, process parameters, and load data into the supervised model, and outputting the weights of indirect impacts on workpiece quality. Define the nodes and edges of the fault phenomena and causes, establish causal relationships, construct a Bayesian network, and verify the causal strength through historical data.

9. A multimodal data intelligent chart query system based on a large model, used to implement the steps of the multimodal data intelligent chart query method based on a large model as described in any one of claims 1-8, characterized in that, include: The data acquisition module is used to acquire real-time monitoring data of the machine tool; The user input module is used to obtain fuzzy query information input by the user; The information extraction module is used to perform intent recognition and entity extraction on the fuzzy query information and parse it into structured query conditions. The database creation module stores machine tool CNC system data, sensor time domain data, sensor frequency domain data, workpiece quality data, and expert experience based on the large model, according to the preset correlation requirements. The chart interaction module takes structured query conditions as input into the established multimodal data association library, generates interactive charts based on the output information of the multimodal data association library, and embeds key conclusion summaries.

10. An electronic device, characterized in that, include: Memory, which stores computer program instructions; A processor, when the computer program instructions are executed by the processor, implements the steps of the intelligent chart query method for multimodal data based on a large model as described in any one of claims 1 to 8.

Images