Flink and large model collaborative data identification task completion method and device, equipment and medium

Abstract

The invention discloses an Flink and large model collaborative data identification task completion method, device and equipment and a medium, and relates to the technical field of large data processing, and the method comprises the steps: carrying out the preprocessing operation of multi-modal data in a real-time data flow based on Flink, and distributing the preprocessed data to a preset time window according to a preset business demand; loading a preset large model sub-graph from a model registration center, and performing semantic recognition on the preprocessed data based on the preset large model sub-graph; carrying out confidence coefficient verification on the first recognition result, and carrying out semantic recognition on the preprocessed data by utilizing a preset lightweight model after the verification is not passed, so as to fuse the second recognition result and the first recognition result to obtain a target recognition result; and carrying out concept drift detection on the target model through the Flink and the target identification result, and updating the target model based on the Flink after judging that the concept drift occurs so as to execute a data identification task.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a method, device, equipment, and medium for completing data recognition tasks in collaboration with Flink and a large model. Background Art

[0002] As a stream processing framework, Flink (a framework for unified stream and batch processing) offers high efficiency and reliability in real-time data processing. It can handle large-scale, real-time data streams, ensuring low latency and high throughput. However, its shortcomings are also evident when it comes to complex pattern recognition tasks. For example, in real-time identification of financial transaction fraud, it can only perform simple data matching and statistical analysis based on preset rules. However, it lacks the ability to deeply understand complex fraud patterns and struggles to uncover the underlying semantic information in the data, resulting in low recognition accuracy and efficiency. In industrial equipment fault prediction scenarios, it can only perform simple time-series analysis of device sensor data, failing to understand the complex correlations and potential failure modes within the data, making accurate fault prediction difficult. In other words, Flink suffers from data processing limitations due to its insufficient semantic understanding capabilities, leaving room for improvement in both accuracy and efficiency.

[0003] In summary, how to solve the problem of low recognition accuracy and efficiency of data recognition tasks caused by Flink's insufficient semantic understanding ability is a technical problem that needs to be solved urgently. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, device, equipment, and medium for completing data recognition tasks in collaboration with Flink and a large model, which can solve the problem of low recognition accuracy and efficiency of data recognition tasks caused by Flink's insufficient semantic understanding capabilities. The specific solution is as follows:

[0005] First, this application provides a method for completing data recognition tasks in collaboration between Flink and a large model, including:

[0006] Based on Flink, preprocess multimodal data in real-time data streams to allocate the preprocessed data to corresponding preset time windows according to preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is the real-time data stream to be identified for the target data identification task;

[0007] Analyzing the preprocessed data in the preset time window to load a corresponding preset large model subgraph from a model registration center based on the obtained analysis results, and performing semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; the preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model whose model scale meets the preset large-scale determination condition;

[0008] Performing a confidence check on the first recognition result, and if the check fails, performing semantic recognition on the preprocessed data using a preset lightweight model, and fusing the obtained second recognition result and the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; the preset lightweight model is a model whose parameters meet a preset lightweight determination condition;

[0009] Concept drift detection is performed on the target model using the Flink and the target recognition result. After determining that concept drift has occurred in the target model, the target model is updated based on the Flink so as to perform the data recognition task using the updated target model; the target model includes the preset large model and the preset lightweight model.

[0010] Optionally, the preprocessing operation based on Flink on the multimodal data in the real-time data stream to allocate the obtained preprocessed data to corresponding preset time windows according to preset business requirements includes:

[0011] Perform data cleaning operations on multimodal data in real-time data streams based on a preset Flink engine to process missing values ​​and outliers in the multimodal data and obtain preprocessed data.

[0012] Extracting data type features corresponding to the preprocessed data according to the data type of the preprocessed data and the target data identification task;

[0013] Calibrating the timestamps of the preprocessed data to unify the time base of the preprocessed data;

[0014] Through the time window mechanism of Flink, the pre-processed data with unified time bases and the corresponding data type characteristics are allocated to corresponding preset time windows according to preset business requirements, and the delayed data in the preset time windows are processed using the event time processing function of Flink to address the data disorder problem in the preset time windows.

[0015] Optionally, the Flink-based preset engine performs a data cleaning operation on the multimodal data in the real-time data stream to process missing values ​​and outliers in the multimodal data to obtain preprocessed data, including:

[0016] Based on Flink's preset engine, the multimodal data in the real-time data stream is formatted and standardized according to the preset data format standards to obtain standardized data;

[0017] Filling missing values ​​in the standardized data using a preset missing value filling technique, and determining a data ratio of target missing values ​​in the standardized data; the target missing values ​​are missing values ​​in the standardized data that cannot be filled using the preset missing value filling technique;

[0018] Comparing the data ratio with a preset data ratio threshold, and if the comparison result indicates that the data ratio is higher than the preset data ratio threshold, discarding the target missing value from the filled normalized data to obtain missing value processed data;

[0019] The outliers in the missing value processed data are determined by a preset outlier detection method, and the outliers are processed according to preset business rules to obtain pre-processed data.

[0020] Optionally, analyzing the preprocessed data in the preset time window to load a corresponding preset large model subgraph from a model registration center according to the obtained analysis result includes:

[0021] Analyzing the pre-processed data in the preset time window to obtain an analysis result including data content features;

[0022] Matching the data content features in the analysis results with preset subgraph determination rules to determine a corresponding preset large model subgraph;

[0023] Determining a target subgraph version with the best performance from among several subgraph versions of the preset large model subgraph based on historical operation performance data of the preset large model subgraph;

[0024] A preset subgraph loading strategy for this subgraph loading is determined based on current resource utilization, so as to load the preset large model subgraph whose subgraph version is the target subgraph version from the model registration center according to the preset subgraph loading strategy.

[0025] Optionally, the confidence check is performed on the first recognition result, and if the check fails, semantic recognition is performed on the preprocessed data using a preset lightweight model, and a preset fusion strategy is used to fuse the obtained second recognition result and the first recognition result to obtain a target recognition result for the target data recognition task, including:

[0026] Determining a confidence score corresponding to the first recognition result;

[0027] Comparing the confidence score with a preset confidence score threshold; if the confidence score is lower than the preset confidence score threshold, determining that the verification fails, and performing semantic recognition on the preprocessed data using a preset lightweight model to obtain a corresponding second recognition result;

[0028] Determining a corresponding preset fusion strategy according to the task type of the target data recognition task;

[0029] The preset fusion strategy is used to perform a fusion operation on the second recognition result and the first recognition result to obtain a target recognition result for the target data recognition task.

[0030] Optionally, performing concept drift detection on the target model using the Flink and the target recognition result includes:

[0031] Determine the confidence score corresponding to the target recognition result, and calculate the average confidence of the current preset large model based on the confidence score. If it is determined that the decrease in the average confidence compared to the historical average confidence satisfies a preset confidence decrease condition, it is determined that concept drift has occurred in the preset large model;

[0032] Alternatively, the checkpoint mechanism of Flink is utilized to determine the current data distribution corresponding to the current target recognition result based on a preset test frequency, and a KS test is performed on the current data distribution and the historical data distribution to determine the corresponding cumulative distribution function difference. When the cumulative distribution function difference is higher than a preset cumulative distribution function difference threshold, it is determined that concept drift has occurred in the target model.

[0033] Optionally, updating the target model based on Flink so as to perform a data recognition task using the updated target model includes:

[0034] Collect a preset number of new real-time data from the current real-time data stream of the Flink;

[0035] The target model is updated using the newly added real-time data and a preset model optimization algorithm to obtain corresponding updated model parameters;

[0036] Based on the broadcast function of Flink, the updated model parameters are synchronized to the pending tasks in the current task queue, so that the pending tasks are executed using the updated target model.

[0037] Secondly, this application provides a device for completing data recognition tasks in collaboration with Flink and a large model, including:

[0038] The data preprocessing module is used to preprocess the multimodal data in the real-time data stream based on Flink, so as to allocate the obtained preprocessed data to the corresponding preset time window according to the preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is the real-time data stream to be identified for the target data recognition task;

[0039] a first semantic recognition module, configured to analyze the preprocessed data in the preset time window, load a corresponding preset large model subgraph from a model registration center based on the obtained analysis result, and perform semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; the preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model whose model scale meets a preset large-scale determination condition;

[0040] a second semantic recognition module, configured to perform a confidence check on the first recognition result and, if the check fails, perform semantic recognition on the preprocessed data using a preset lightweight model, so as to fuse the obtained second recognition result with the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; the preset lightweight model is a model whose parameters satisfy a preset lightweight determination condition;

[0041] A model update module is configured to perform concept drift detection on a target model using Flink and the target recognition result, and, upon determining that concept drift has occurred in the target model, update the target model based on Flink so as to perform data recognition tasks using the updated target model; the target model includes the preset large model and the preset lightweight model.

[0042] In a third aspect, the present application provides an electronic device, comprising:

[0043] Memory, used to store computer programs;

[0044] A processor is used to execute the computer program to implement the aforementioned method for completing the data recognition task in collaboration with Flink and a large model.

[0045] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the aforementioned method for completing the data recognition task in collaboration between Flink and a large model is implemented.

[0046] In this application, a preprocessing operation is performed on the multimodal data in the real-time data stream based on Flink, so as to allocate the obtained preprocessed data to the corresponding preset time window according to the preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is a real-time data stream to be identified for the target data identification task; the preprocessed data in the preset time window is analyzed, and the corresponding preset large model subgraph is loaded from the model registration center according to the obtained analysis results, and the preprocessed data is semantically recognized based on the preset large model subgraph to obtain the corresponding first recognition result; the preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model A model whose scale meets the preset large-scale judgment conditions; performing confidence verification on the first recognition result, and if the verification fails, using a preset lightweight model to perform semantic recognition on the preprocessed data, so as to use a preset fusion strategy to fuse the obtained second recognition result and the first recognition result to obtain a target recognition result for the target data recognition task; the preset lightweight model is a model whose parameter quantity meets the preset lightweight judgment conditions; performing concept drift detection on the target model through the Flink and the target recognition result, and after determining that the target model has concept drift, updating the target model based on the Flink, so as to use the updated target model to perform the data recognition task; the target model includes the preset large model and the preset lightweight model. As can be seen from the above, this application uses Flink to preprocess real-time data streams containing multimodal data such as text, images, audio, and video, and allocates them to corresponding preset time windows according to preset business requirements. The preprocessed data within the preset time window is then analyzed to load the corresponding preset large model subgraph from the model registration center for semantic recognition based on the analysis results to obtain a first recognition result. The first recognition result is then confidence-checked. If it fails, the preset lightweight model is used for recognition again, and the second recognition result is fused with the first recognition result to obtain a target recognition result. Finally, Flink is used to perform concept drift detection on the target model including the preset large model and the preset lightweight model. Once concept drift is determined to have occurred, the target model is updated based on Flink to ensure that it can continue to accurately perform data recognition tasks.In this way, through the above process of this application, by introducing the preset large model, Flink is given deep semantic understanding capabilities, enabling it to handle complex semantic analysis tasks and improve the accuracy of data recognition; the preset large model is split into preset large model subgraphs according to the type of data processing function to ensure that the subgraphs can be flexibly combined and replaced; the preset large model subgraphs are used in combination with the preset lightweight model, taking into account the accuracy and efficiency of recognition, and further improving the reliability of the recognition results; the model is subjected to concept drift detection and update, so that the target model can adapt to the dynamic changes of data distribution and maintain good data recognition performance, thereby providing stable and accurate support for the target data recognition task, thereby solving the problem of low recognition accuracy and efficiency of data recognition tasks due to insufficient semantic understanding capabilities of Flink. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0048] Figure 1 This is a flowchart of a method for completing a data recognition task by collaborating with Flink and a large model, as disclosed in this application.

[0049] Figure 2 This is a schematic diagram of the structure of a device for completing data recognition tasks in collaboration with Flink and a large model disclosed in this application;

[0050] Figure 3 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0051] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0052] As a stream processing framework, Flink offers high efficiency and reliability in real-time data processing. It can handle large-scale, real-time data streams, ensuring low latency and high throughput. However, its shortcomings are more pronounced when it comes to complex pattern recognition tasks. For example, in real-time identification of financial transaction fraud, it can only perform simple data matching and statistical analysis based on pre-set rules. However, it lacks deep semantic understanding of complex fraud patterns and struggles to uncover the underlying semantic information in the data, resulting in low recognition accuracy and efficiency. In industrial equipment fault prediction scenarios, it can only perform simple time-series analysis of device sensor data, failing to understand the complex correlations and potential failure modes between the data, making accurate fault prediction difficult. In other words, Flink suffers from data processing deficiencies due to its insufficient semantic understanding capabilities, leaving room for improvement in both accuracy and efficiency.

[0053] In order to overcome the above technical problems, this application provides a method for completing data recognition tasks by collaborating with Flink and a large model, so as to solve the problems of low recognition accuracy and efficiency of data recognition tasks caused by Flink's insufficient semantic understanding ability.

[0054] See also Figure 1 As shown, an embodiment of the present invention discloses a method for completing a data recognition task by collaborating with Flink and a large model, including:

[0055] Step S11: Preprocess the multimodal data in the real-time data stream based on Flink to allocate the obtained preprocessed data to the corresponding preset time window according to the preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is the real-time data stream to be identified for the target data recognition task.

[0056] In this embodiment, Flink preprocesses multimodal data in a real-time data stream to obtain preprocessed data, which is then allocated to corresponding preset time windows based on preset business requirements. The multimodal data includes, but is not limited to, text data, image data, audio data, and video data; the real-time data stream is a real-time data stream to be identified for target data recognition tasks; and the target data recognition tasks include, but are not limited to, data recognition tasks in real-time data processing and analysis application scenarios in fields such as big data analysis, industrial Internet of Things, social media, and smart healthcare.

[0057] Specifically, a data cleaning operation is performed on multimodal data in a real-time data stream based on a preset engine of Flink to process missing values ​​and outliers in the multimodal data and obtain preprocessed data; data type features corresponding to the preprocessed data are extracted based on the data type of the preprocessed data and the target data identification task; the timestamps of the preprocessed data are calibrated to unify the time base of the preprocessed data; and the preprocessed data with a unified time base and the corresponding data type features are assigned to corresponding preset time windows based on preset business requirements through the time window mechanism of Flink, and delayed data in the preset time windows is processed using the event time processing function of Flink to address data disorder in the preset time windows. The preset engine can be Flink's CEP (Complex Event Processing) engine. That is, Flink's preset engine is used to clean the multimodal data in the real-time data stream to process missing values ​​and outliers in the multimodal data to obtain preprocessed data. Subsequently, according to the data type of the preprocessed data and the target data identification task, the data type features corresponding to the preprocessed data are extracted. For the text data, natural language processing technologies such as word segmentation, part-of-speech tagging, and named entity recognition are used to extract features such as keywords, phrases, and semantic vectors; for the image data, convolutional neural networks are used to extract visual features of the image, such as edges, textures, and shapes; for time series data such as audio data and video data, statistical features such as mean, variance, maximum, minimum, trend terms, and periodic terms are calculated, and their frequency domain features are extracted through methods such as Fourier transform and wavelet transform. After extracting the data type features, the timestamps of the pre-processed data are calibrated to unify the time base of the pre-processed data, ensuring the accuracy and consistency of the timestamps, and through the time window mechanism of Flink, the pre-processed data and the data type features after the time base is unified are allocated to the corresponding preset time windows according to the preset business requirements to achieve the alignment of multimodal data in the same window, and the event time processing function of Flink is utilized to process the delayed data in the preset time window through the watermark mechanism to deal with the data disorder problem and ensure the correct alignment of multimodal data in time. For example, when processing video streams and audio streams, the video frames and audio data at the same time point are aligned according to the timestamps of the video frames and audio samples to ensure that the subsequent joint analysis has time consistency. It can be understood that the preset time window of this embodiment is a time window set according to the preset business requirements, including but not limited to sliding windows, rolling windows, session windows, etc.

[0058] It should be pointed out that the processing flow of performing data cleaning operations on the multimodal data is as follows: based on Flink's preset engine, the multimodal data in the real-time data stream is format-standardized according to the preset data format standard to obtain standardized data; the missing values ​​in the standardized data are filled using the preset missing value filling technology, and the data ratio of the target missing values ​​in the standardized data is determined; the target missing values ​​are the missing values ​​in the standardized data that cannot be filled by the preset missing value filling technology; the data ratio is compared with the preset data ratio threshold, and if the comparison result obtained indicates that the data ratio is higher than the preset data ratio threshold, the target missing value is discarded from the filled standardized data to obtain missing value processed data; the outliers in the missing value processed data are determined by the preset outlier detection method, and the outliers are processed according to preset business rules to obtain preprocessed data. Among them, the preset missing value filling technology includes but is not limited to mean filling, median filling, model-based prediction filling and other technologies; the preset outlier detection method includes but is not limited to statistical methods such as Z-score method (i.e. Z score, a method for evaluating the distance between a sample point and the overall mean), IQR method (i.e. interquartile range, a statistical technique for identifying outliers), and machine learning algorithms such as isolation forest, One-Class SVM (i.e. single-class support vector machine, a supervised learning algorithm for anomaly detection). That is, through Flink's preset engine, data from different sources and formats in real-time data streams are converted into a unified format according to preset data format standards. For example, timestamps in different time formats are converted into a unified time format, and texts in different encodings are converted into UTF-8 encoding (a variable-length character encoding), etc., to obtain standardized data. Based on the characteristics of the data and business needs, the missing values ​​in the standardized data are filled using the preset missing value filling technology. At the same time, the data ratio of target missing values ​​that cannot be filled in the standardized data is determined. If the data ratio is higher than the preset data ratio threshold, the target missing values ​​are discarded to obtain missing value-processed data. Subsequently, outliers in the missing value-processed data are detected using a preset outlier detection method, and the outliers are processed according to preset business rules, such as correcting outliers, marking outliers, or deleting outliers, to obtain preprocessed data.

[0059] It should be noted that data recognition tasks for real-time data streams are typically implemented using the Flink framework, or large models such as language and vision models based on the Transformer architecture (a deep learning model used for natural language processing and other sequence-to-sequence tasks). However, while the Flink framework offers high efficiency and reliability in real-time data processing, capable of handling large-scale real-time data streams while ensuring low latency and high throughput, it lacks deep semantic understanding capabilities when faced with complex pattern recognition tasks, making it difficult to mine the underlying semantic information in the data. This results in low recognition accuracy and efficiency. While large models possess powerful semantic understanding and pattern recognition capabilities, their application in real-time stream data processing scenarios suffers from high operating costs, high latency, and poor adaptability to dynamic data. This results in low data recognition efficiency and makes it difficult to adapt to changing business needs. In order to resolve the contradiction between the real-time processing capability of the existing streaming computing framework and the high latency and high resource consumption of large models, this application proposes a data recognition task completion method that collaborates with Flink and large models. By building a deep collaboration mechanism between the Flink stream processing engine and the large model, real-time intelligent recognition in complex data scenarios is achieved, and Flink is combined with the large model to improve the throughput of real-time data processing, reduce latency, and improve the efficiency and accuracy of data recognition. In this way, this embodiment performs timestamp calibration on multimodal data in real-time data streams to unify the time base and ensure the accuracy and consistency of timestamps; utilizes Flink's event time processing function to process delayed data in the time window to handle data disorder problems and ensure the correct alignment of multimodal data in time; performs data cleaning on multimodal data in real-time data streams to ensure the accuracy and integrity of input data and provide high-quality data input for subsequent intelligent recognition; and achieves high-throughput, low-latency real-time intelligent recognition through the collaboration of the Flink stream processing framework and the large model lightweight service, thereby improving the efficiency and accuracy of data recognition.

[0060] Step S12: Analyze the preprocessed data in the preset time window to load the corresponding preset large model subgraph from the model registration center according to the obtained analysis results, and perform semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; the preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model whose model scale meets the preset large-scale judgment condition.

[0061] In this embodiment, the pre-processed data in the preset time window is analyzed to obtain corresponding analysis results, and the corresponding preset large model subgraph is loaded from the model registration center according to the analysis results, and the pre-processed data is semantically recognized using the preset large model subgraph to obtain a corresponding first recognition result. The preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model whose model scale meets the preset large-scale judgment conditions, which can be a language model and a visual model based on the Transformer architecture such as DeepSeek-MoE (a model based on a hybrid expert model architecture); the model registration center is used to maintain the metadata of all available subgraphs, including but not limited to the subgraph name, function description, input and output format, computing resource requirements, version information, etc.

[0062] It is understood that this embodiment splits the preset large model according to functional modules to obtain pluggable subgraphs of the preset large model, including but not limited to a text understanding subgraph, an image recognition subgraph, a time series prediction subgraph, and a multimodal fusion subgraph, which are stored in the model registration center. Each of the preset large model subgraphs focuses on a specific type of data processing and semantic understanding task, with independent input and output interfaces and computational logic. For example, the text understanding subgraph, based on the Transformer architecture, handles semantic understanding and generation tasks of text data; the image recognition subgraph, based on the CNN (convolutional neural network) architecture, handles feature extraction and object recognition tasks of image data. The subgraph splitting adheres to the principles of high cohesion and low coupling, ensuring that the subgraphs can be flexibly combined and replaced, flexibly adapting to different task requirements, and enhancing the adaptability and practicality of the large model in different fields and scenarios. It not only facilitates internal optimization and iteration of the subgraphs, but also allows for flexible combination and replacement of the subgraphs according to actual needs, giving the model high flexibility and scalability, reducing the complexity of model development and maintenance.

[0063] It should be noted that the processing flow for loading the preset large model subgraph is as follows: the pre-processed data in the preset time window is analyzed to obtain an analysis result containing data content features; the data content features in the analysis result are matched with the preset subgraph determination rules to determine the corresponding preset large model subgraph; based on the historical operation performance data of the preset large model subgraph, the target subgraph version with the best performance is determined from several subgraph versions of the preset large model subgraph; based on the current resource utilization, the preset subgraph loading strategy for this subgraph loading is determined, so as to load the preset large model subgraph with the subgraph version of the target subgraph version from the model registration center according to the preset subgraph loading strategy. The historical operation performance data includes but is not limited to inference latency, accuracy, resource consumption, etc.; the resource utilization includes but is not limited to GPU (graphics processing unit) memory occupancy, CPU (central processing unit) utilization, memory usage, etc. That is, within each of the preset time windows, the meta-information and content features of the pre-processed data, such as data type, data source, business scenario, priority, etc., are analyzed to extract data content features for model selection, and obtain analysis results containing the data content features. For example, for a window dominated by text data, features such as the data type "text", text length, and keyword distribution are extracted; for a window containing image data, features such as the data type "image", image resolution, and preliminary prediction results of the target category are extracted. The data content features in the analysis results are then matched with the preset subgraph determination rules to determine the corresponding preset large model subgraph. For example, when the data type is text, a text understanding subgraph is loaded, and when a time series anomaly feature is detected, the subgraph is automatically switched to the time series prediction subgraph to quickly determine the required subgraph. At the same time, the target subgraph version with the best performance is determined from several subgraph versions in combination with the historical operation performance data of the preset large model subgraph, so as to ensure that the system performance is optimized under the premise of meeting business needs, and the preset subgraph loading strategy of the subgraph is dynamically adjusted according to the current resource utilization, so as to load the preset large model subgraph with the subgraph version of the target subgraph version from the model registration center according to the preset subgraph loading strategy, thereby avoiding resource overload and ensuring the stable operation of the system. In this way, this embodiment splits the preset large model into multiple pluggable preset large model subgraphs according to the functional modules, and each has an independent input and output interface and calculation logic, ensuring that they can be flexibly combined and replaced with each other to flexibly adapt to different task requirements, enhance the adaptability and practicality of the large model, not only facilitate the internal optimization and iteration of the subgraph, but also can be flexibly combined and replaced according to actual needs, giving the model a high degree of flexibility and scalability, and reducing the complexity of model development and maintenance; dynamically selecting the preset large model subgraph according to the feature analysis results of the preset time window can reduce redundant calculations, improve model reasoning efficiency, and ensure that the system performance is optimized under the premise of meeting business needs.

[0064] Step S13: Perform a confidence check on the first recognition result, and if the check fails, perform semantic recognition on the preprocessed data using a preset lightweight model, and fuse the obtained second recognition result and the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; the preset lightweight model is a model whose parameter quantity meets the preset lightweight judgment condition.

[0065] In this embodiment, a confidence check is performed on the first recognition result. If the check fails, a preset lightweight model is used to perform semantic recognition on the preprocessed data to obtain a corresponding second recognition result. At the same time, a preset fusion strategy is used to fuse the second recognition result with the first recognition result to obtain a target recognition result for the target data recognition task. The preset lightweight model is a discriminant model whose parameters meet the preset lightweight judgment conditions, and can be XGBoost (i.e., extreme gradient boosting, an efficient machine learning algorithm), LightGBM (a distributed gradient boosting framework based on decision trees), etc.

[0066] Specifically, determine the confidence score corresponding to the first recognition result; compare the confidence score with a preset confidence score threshold; if the confidence score is lower than the preset confidence score threshold, determine that the verification has failed, and use a preset lightweight model to perform semantic recognition on the preprocessed data to obtain a corresponding second recognition result; determine a corresponding preset fusion strategy based on the task type of the target data recognition task; use the preset fusion strategy to fuse the second recognition result and the first recognition result to obtain a target recognition result for the target data recognition task. The confidence score includes but is not limited to Softmax (an activation function) probability value, logarithmic probability, etc.; the preset fusion strategy includes but is not limited to ensemble voting method, weighted average method, stacking method, etc. That is, determine the confidence score corresponding to the first recognition result, and compare the confidence score with the preset confidence score threshold. If it is lower than the preset confidence score threshold, for example, lower than 0.8, then it is determined that the verification fails, triggering secondary verification. The preset lightweight model has the characteristics of high computational efficiency and flexible deployment, and can quickly predict data in a short time. It is used to perform secondary recognition on the preprocessed data to obtain the corresponding second recognition result. Then, the corresponding preset fusion strategy is determined according to the task type of the target data recognition task, and the preset fusion strategy is used to fuse the second recognition result and the first recognition result to obtain the target recognition result for the target data recognition task. For example, in a text classification task, when the large model has a low classification confidence for a certain text, the XGBoost model is called for secondary classification, and then the results of the two models are fused through integrated voting such as majority voting and weighted voting to improve the classification accuracy.

[0067] It should be noted that, to improve recognition accuracy and efficiency, this embodiment can also construct a cascade decision-making mechanism to achieve hierarchical recognition and result fusion. Specifically, a three-layer processing pipeline is constructed. First, a lightweight rule engine, such as regular matching and simple statistical rules, is deployed in the first layer to quickly filter the real-time data stream and remove data that is obviously invalid or does not meet the conditions. For example, in financial risk control scenarios, invalid transaction records with a transaction amount of 0 are quickly filtered out through preset rules, reducing the amount of data to be processed later and improving processing efficiency. Subsequently, a large model is run in the second layer to complete fine-grained semantic recognition, performing deep semantic analysis and complex pattern recognition on the data filtered by the first layer. The large model uses its powerful semantic understanding capabilities to handle complex semantic reasoning tasks, such as sentiment analysis and intent recognition in text, object detection and semantic segmentation in images, etc. Finally, in the third layer, the recognition results are fed back in real time through Flink's State Backend (a component used to manage and store state) for online learning and updating of the model. At the same time, the confidence check and result fusion of the results of the second layer output are performed, and combined with the secondary verification of the lightweight model, the reliability of the recognition results is improved to achieve layer-by-layer filtering and refined processing of the data. In this way, after obtaining the first recognition result, this embodiment performs a confidence check on it, and if the check fails, it uses the preset lightweight model to perform re-recognition, which can ensure the accuracy of the final recognition result and improve the efficiency and accuracy of recognition; a cascade decision-making mechanism is constructed, which first quickly filters the real-time data stream to remove obviously invalid or non-compliant data, reduce the amount of data to be processed subsequently, improve processing efficiency, and achieve layered recognition and result fusion.

[0068] Step S14: Perform concept drift detection on the target model using Flink and the target recognition result. After determining that concept drift has occurred in the target model, update the target model based on Flink so as to perform the data recognition task using the updated target model; the target model includes the preset large model and the preset lightweight model.

[0069] In this embodiment, concept drift detection is performed on the target model using Flink and the target recognition results. After determining that concept drift has occurred, the target model is updated based on Flink so that the updated target model can be used to perform the data recognition task. The target model includes the preset large model and the preset lightweight model.

[0070] It should be pointed out that the processing flow for concept drift detection of the target model is as follows: determine the confidence score corresponding to the target recognition result, and calculate the average confidence of the current preset large model based on the confidence score. If it is determined that the decrease in the average confidence compared to the historical average confidence meets the preset confidence decrease condition, then it is determined that the preset large model has concept drift; or, using the checkpoint mechanism of Flink, determine the current data distribution corresponding to the current target recognition result based on the preset test frequency, and perform a KS test on the current data distribution and the historical data distribution to determine the corresponding cumulative distribution function difference, and after the cumulative distribution function difference is higher than the preset cumulative distribution function difference threshold, determine that the target model has concept drift. That is, this embodiment can use the confidence score to perform concept drift detection through the model confidence decrease indicator method, or it can use the KS test method (a non-parametric hypothesis testing method) to perform detection based on the data distribution. Specifically, this embodiment can determine the confidence score corresponding to the target recognition result, and monitor and count the average confidence of the current preset large model based on the confidence score. If it is determined that the decline in the average confidence compared with the historical average confidence meets the preset confidence decline condition, that is, it continues to decline and exceeds a certain amplitude, it indicates that the model may not be adapted to the current data distribution, and it is determined that the preset large model has undergone concept drift; this embodiment can also use the checkpoint mechanism of Flink to periodically determine the current data distribution corresponding to the current target recognition result based on a preset inspection frequency, and perform a KS test on the current data distribution and the historical data distribution to calculate the cumulative distribution function difference between the two. At the same time, when the cumulative distribution function difference is higher than the preset cumulative distribution function difference threshold, it is determined that the target model has undergone concept drift. It should be further pointed out that this embodiment can also improve the efficiency of concept drift detection by monitoring the accuracy of the model. Specifically, it can evaluate the accuracy of the target model on the validation set in real time. When it is detected that the accuracy has dropped significantly, it is judged that concept drift may have occurred. Then, the above-mentioned concept drift detection method can be used to further determine whether concept drift has occurred. If the accuracy is not detected to have dropped significantly, it is not necessary to execute the above-mentioned concept drift detection method to reduce system load and unnecessary detection operations.

[0071] It should be noted that after detecting concept drift, an online fine-tuning process is initiated to update the target model. The processing flow is as follows: a preset number of new real-time data are collected from the current real-time data stream of Flink; the target model is updated using the new real-time data and a preset model optimization algorithm to obtain the corresponding updated model parameters; based on the broadcast function of Flink, the updated model parameters are synchronized to the pending tasks in the current task queue so that the pending tasks can be executed using the updated target model. The preset model optimization algorithm can be a small batch gradient descent optimization algorithm. That is, a preset number of new real-time data are collected from the current real-time data stream of Flink as fine-tuning training data, and then the target model is updated on the training cluster using the new real-time data and the preset model optimization algorithm to avoid overfitting, and the corresponding updated model parameters are obtained. Finally, based on the broadcast function of Flink, the updated model parameters are synchronized to the pending tasks in the current task queue to achieve real-time update of model parameters, efficiently broadcast data to all parallel task instances, ensure that the latest model parameters are used for data processing, realize model hot update, and the update delay is less than 1 second, supporting business-unaware iteration, so as to use the updated target model to execute the pending tasks. In this way, this embodiment continuously performs concept drift detection on the target model and makes a preliminary judgment on concept drift by monitoring the accuracy of the model. If the preliminary judgment shows that concept drift will not occur, there is no need to perform other concept drift detection methods, which can reduce system load and unnecessary detection operations. After detecting that concept drift has occurred in the model, the target model is promptly updated and synchronized in real time through Flink's mechanisms and functions, supporting dynamic optimization and hot deployment of the model, ensuring that the system can adapt to changes in data distribution in a timely manner and supporting business-unaware iteration.

[0072] As can be seen from the above, the embodiment of the present application uses Flink to preprocess the real-time data stream containing multimodal data such as text, images, audio, video, etc., and allocates it to the corresponding preset time window according to preset business requirements. The preprocessed data in the preset time window is then analyzed to load the corresponding preset large model subgraph from the model registration center for semantic recognition based on the analysis results to obtain a first recognition result, and then the first recognition result is confidence checked. If it fails, the preset lightweight model is used to perform recognition again, and the second recognition result is fused with the first recognition result to obtain the target recognition result. Finally, Flink is used to detect concept drift on the target model including the preset large model and the preset lightweight model. Once concept drift is determined to occur, the target model is updated based on Flink to ensure that it can continue to accurately perform data recognition tasks.In this way, through the above process of the embodiment of the present application, on the one hand, the timestamp calibration is performed on the multimodal data in the real-time data stream to unify the time base and ensure the accuracy and consistency of the timestamp; on the other hand, the event time processing function of Flink is used to process the delayed data in the time window to deal with the data disorder problem and ensure the correct alignment of the multimodal data in time; on the one hand, the multimodal data in the real-time data stream is cleaned to ensure the accuracy and integrity of the input data and provide high-quality data input for subsequent intelligent recognition; on the one hand, the Flink stream processing framework is coordinated with the large model lightweight service to achieve high-throughput, low-latency real-time intelligent recognition, thereby improving the efficiency and accuracy of data recognition; on the one hand, the preset large model is split into multiple pluggable preset large model subgraphs according to the functional modules, and each has an independent input and output interface and calculation logic, ensuring that they can be flexibly combined and replaced with each other to flexibly adapt to different task requirements, thereby enhancing the adaptability and practicality of the large model, not only facilitating the optimization and iteration within the subgraph, but also flexibly combining and replacing according to actual needs, giving the model a high degree of flexibility and scalability, and reducing the complexity of model development and maintenance; on the one hand, the preset is dynamically selected based on the feature analysis results of the preset time window Large model subgraphs can reduce redundant computations, improve model inference efficiency, and ensure optimized system performance while meeting business needs. After obtaining the first recognition result, a confidence check is performed on it. If the check fails, a preset lightweight model is used for re-recognition, ensuring the accuracy of the final recognition result and improving recognition efficiency and accuracy. A cascade decision-making mechanism is constructed to quickly filter the real-time data stream to remove obviously invalid or non-compliant data, reducing the amount of data to be processed later, improving processing efficiency, and achieving layered recognition and result fusion. Concept drift detection is continuously performed on the target model, and a preliminary judgment on concept drift is made by monitoring the model's accuracy. If the preliminary judgment is that concept drift will not occur, no additional concept drift detection methods are required, reducing system load and unnecessary detection operations. Furthermore, after detecting concept drift in the model, the target model is promptly updated and synchronized in real time through Flink's mechanisms and functions. This supports dynamic optimization and hot deployment of the model, ensuring that the system can promptly adapt to changes in data distribution and enabling business-unaware iteration. This addresses the problem of low recognition accuracy and efficiency in data recognition tasks caused by Flink's insufficient semantic understanding capabilities.

[0073] As can be seen from the previous example, this application discloses a method for completing data recognition tasks by collaborating with Flink and a large model. This method can address the low accuracy and efficiency of data recognition tasks caused by Flink's insufficient semantic understanding capabilities. Next, we will describe this method in detail.

[0074] First, stream data access and preprocessing are implemented. For data access, Flink SQL (a structured query language) is used to parse the raw data stream. This supports access to a variety of data sources, such as Kafka (a distributed messaging system), MySQL (an open-source database), file systems, and message queues. Data source connection information and data formats are defined using Flink SQL's DDL statements (a data definition language). For example:

[0075] CREATE TABLE source_table (

[0076] data_type STRING,

[0077] timestamp BIGINT,

[0078] content STRING,

[0079] metadata MAP<STRING, STRING> ) WITH (

[0080] 'connector' = 'kafka',

[0081] 'topic' = 'real_time_data',

[0082] 'bootstrap.servers' = 'kafka:9092',

[0083] 'format' = 'json');

[0084] For the preprocessing of multimodal data alignment, we use Flink's ProcessFunction (a window aggregation function) to implement multimodal data alignment. Taking the synchronization of video frame and audio stream timestamps as an example, the specific implementation steps are as follows:

[0085] Define event structure: Define event structure containing timestamp and content for video frames and audio data, such as:

[0086] public class VideoEvent {

[0087] private long timestamp;

[0088] private byte[] frameData;

[0089] / / getter and setter}

[0090] public class AudioEvent {

[0091] private long timestamp;

[0092] private byte[] audioData;

[0093] / / getter and setter}

[0094] For time window processing operations, Flink's EventTime (event time processing) and TimeWindow (time window) mechanisms are used to divide the video stream and audio stream into windows respectively, and the window size is set to 500ms according to business requirements. For the data time alignment logic, two states, video buffer and audio buffer, are maintained in ProcessFunction to store unaligned video events and audio events. When a video event or audio event is received, it is stored in the corresponding buffer, and a check is made in the other buffer to see whether there is an event with a timestamp within the allowable error range (such as ±10ms). If so, the matching video and audio events are output; if not, continue to wait until the expired events are cleared when the window is triggered. The data preprocessing process is implemented through Flink's DataStream API (a class used to define data sources). The specific code example is as follows:

[0095] DataStream <multimodaldata>preprocessedStream = rawStream

[0096] .process(new DataCleaningFunction()) / / Data cleaning

[0097] .process(new FeatureExtractionFunction()) / / Feature extraction

[0098] .keyBy(data -> data.getMetadata().get("data_type"))

[0099] .window(TumblingEventTimeWindows.of(Time.seconds(10)))

[0100] .process(new MultiModalAlignmentFunction()); / / Multimodal alignment

[0101] Secondly, dynamic model routing is implemented to load the preset large model subgraph. In the design of the model registration center, a distributed key-value storage system such as Redis is used to store the metadata of the large model subgraph. The data structure is as follows:

[0102] json

[0103] {

[0104] "model_key": "text_understanding_subgraph_v1",

[0105] "function": "Text Semantic Understanding",

[0106] "input_format": "Text string",

[0107] "output_format": "Semantic vector",

[0108] "resource_requirements": { "gpu": 1, "memory": "4GB"},

[0109] "version": "1.0",

[0110] "model_path": "hdfs: / / model_repo / text_subgraph_v1"

[0111] }

[0112] Subsequently, use Flink UDF (User-Defined Function) to implement model routing, such as:

[0113] java

[0114] public class ModelRouter extends ProcessFunction<Row, ModelRequest> {

[0115] private transient MapState<String, String> modelRegistry;

[0116] @Override

[0117] public void open(Configuration parameters) {

[0118] MapStateDescriptor<String, String> descriptor = newMapStateDescriptor<>(

[0119] "modelRegistry",

[0120] TypeInformation.of(String.class),

[0121] TypeInformation.of(String.class) );

[0123] modelRegistry = getRuntimeContext().getMapState(descriptor);

[0124] }

[0125] @Override

[0126] public void processElement(Row row, Context ctx, Collector <modelrequest>out) {

[0127] String dataType = (String) row.getField("data_type");

[0128] String priority = (String) row.getField("priority");

[0129] String modelKey = ModelSelector.getModelKey(dataType,priority); / / Select the model key based on data type and priority

[0130] String modelPath = modelRegistry.get(modelKey); / / Get the model path from the model registry

[0131] out.collect(new ModelRequest(modelKey, modelPath, row));

[0132] }

[0133] }

[0134] Hybrid inference is then implemented. First, a large model service is deployed on a GPU cluster, providing inference services via gRPC (a remote procedure call framework) or RESTful API (a web application design and development style based on the Hypertext Transfer Protocol). The server receives data requests, preprocesses them, and then feeds them into the large model. It then returns a response containing the recognition result and confidence level. The code implementing the hybrid inference algorithm is as follows:

[0135] Python

[0136] def hybrid_inference(data):

[0137] # Large Model Inference

[0138] llm_input = preprocess_for_llm(data)

[0139] llm_result = grpc.infer(llm_input)

[0140] llm_confidence = llm_result.confidence

[0141] if llm_confidence < 0.8:

[0142] # Lightweight model secondary validation

[0143] xg_input = preprocess_for_xgboost(data)

[0144] xg_result = xgboost_model.predict(xg_input)

[0145] # Ensemble voting, large model weight 0.7, lightweight model weight 0.3

[0146] final_result = ensemble_vote(llm_result, xg_result, weights=[0.7, 0.3])

[0147] else:

[0148] final_result = llm_result

[0149] return final_result

[0150] Finally, the hot update of the model is implemented, and the model parameters are synchronized through the Flink BroadcastState (state broadcast) mechanism, as follows:

[0151] java

[0152] DataStream <modelupdate>modelStream = env.addSource(newModelUpdateSource()); / / Model update data stream

[0153] DataStream <data>dataStream = env.addSource(new DataSource()); / / Real-time data stream

[0154] BroadcastStream <modelupdate>broadcastStream = modelStream.broadcast(new ModelStateDescriptor());

[0155] dataStream.connect(broadcastStream)

[0156] .process(new ModelUpdater())

[0157] .name("Model Hot Update");

[0158] / / ModelUpdater implementation

[0159] public class ModelUpdater extends KeyedBroadcastProcessFunction<Key,Data, ModelUpdate, OutputData> {

[0160] private Model currentModel;

[0161] @Override

[0162] public void processBroadcastElement(ModelUpdate modelUpdate,Context ctx, Collector <outputdata>out) {

[0163] currentModel = modelUpdate.getNewModel(); / / Update local model

[0164] }

[0165] @Override

[0166] public void processElement(Data data, ReadOnlyContext ctx,Collector <outputdata>out) {

[0167] OutputData result = currentModel.process(data); / / Process data using the latest model

[0168] out.collect(result);

[0169] }

[0170] }

[0171] Specifically, the model output is periodically obtained from Flink's State Backend, the KS test statistic or average confidence score is calculated, and when a threshold is exceeded, a fine-tuning trigger signal is sent. After receiving the trigger signal, the latest data is sampled from Flink's data stream and small-batch fine-tuning is performed using a distributed training framework such as Horovod to generate updated model parameters. The updated parameters are encapsulated as ModelUpdate events, sent to the model stream, and synchronized to all pending tasks via BroadcastState.

[0172] Accordingly, see Figure 2 As shown, the embodiment of the present application also provides a device for completing data recognition tasks in collaboration with Flink and a large model, including:

[0173] The data preprocessing module 11 is used to perform preprocessing operations on multimodal data in the real-time data stream based on Flink, so as to allocate the obtained preprocessed data to corresponding preset time windows according to preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is the real-time data stream to be identified for the target data identification task;

[0174] A first semantic recognition module 12 is configured to analyze the preprocessed data in the preset time window, load a corresponding preset large model subgraph from a model registration center based on the obtained analysis result, and perform semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; the preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; the preset large model is a model whose model scale meets a preset large-scale determination condition;

[0175] a second semantic recognition module 13 configured to perform a confidence check on the first recognition result and, if the check fails, perform semantic recognition on the preprocessed data using a preset lightweight model, and fuse the obtained second recognition result with the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; the preset lightweight model is a model whose parameters satisfy a preset lightweight determination condition;

[0176] The model update module 14 is configured to perform concept drift detection on the target model using the Flink and the target recognition result, and, after determining that concept drift has occurred in the target model, update the target model based on the Flink so as to perform the data recognition task using the updated target model; the target model includes the preset large model and the preset lightweight model.

[0177] As can be seen from the above, the embodiment of the present application uses Flink to preprocess the real-time data stream containing multimodal data such as text, images, audio, video, etc., and allocates it to the corresponding preset time window according to preset business requirements. The preprocessed data in the preset time window is then analyzed to load the corresponding preset large model subgraph from the model registration center for semantic recognition based on the analysis results to obtain a first recognition result, and then the first recognition result is confidence checked. If it fails, the preset lightweight model is used to perform recognition again, and the second recognition result is fused with the first recognition result to obtain the target recognition result. Finally, Flink is used to detect concept drift on the target model including the preset large model and the preset lightweight model. Once concept drift is determined to occur, the target model is updated based on Flink to ensure that it can continue to accurately perform data recognition tasks. In this way, through the above process of the embodiment of the present application, by introducing the preset large model, Flink is given deep semantic understanding capabilities, enabling it to handle complex semantic analysis tasks and improve the accuracy of data recognition; the preset large model is split into preset large model subgraphs according to the type of data processing function to ensure that the subgraphs can be flexibly combined and replaced; the preset large model subgraphs are used in combination with the preset lightweight model, taking into account the accuracy and efficiency of recognition, and further improving the reliability of the recognition results; the model is subjected to concept drift detection and update, so that the target model can adapt to the dynamic changes of data distribution and maintain good data recognition performance, thereby providing stable and accurate support for the target data recognition task, thereby solving the problem of low recognition accuracy and efficiency of data recognition tasks due to insufficient semantic understanding capabilities of Flink.

[0178] In some specific implementations, the data preprocessing module 11 may specifically include:

[0179] The data cleaning submodule is used to perform data cleaning operations on multimodal data in the real-time data stream based on Flink's preset engine to process missing values ​​and outliers in the multimodal data and obtain preprocessed data;

[0180] a feature extraction unit, configured to extract data type features corresponding to the preprocessed data according to the data type of the preprocessed data and the target data identification task;

[0181] A time calibration unit, configured to calibrate the timestamps of the preprocessed data to unify the time base of the preprocessed data;

[0182] A data processing unit is configured to allocate the pre-processed data with a unified time reference and the corresponding data type characteristics to corresponding preset time windows according to preset business requirements through the time window mechanism of Flink, and to process the delayed data in the preset time windows using the event time processing function of Flink to address the data disorder problem in the preset time windows.

[0183] In some specific implementations, the data cleaning submodule may specifically include:

[0184] The data standardization unit is used to standardize the format of multimodal data in real-time data streams according to the preset data format standard based on Flink's preset engine to obtain standardized data;

[0185] a ratio determination unit, configured to fill missing values ​​in the standardized data using a preset missing value filling technique, and determine a data ratio of target missing values ​​in the standardized data; the target missing values ​​are missing values ​​in the standardized data that cannot be filled using the preset missing value filling technique;

[0186] a missing value discarding unit, configured to compare the data ratio with a preset data ratio threshold, and if the comparison result indicates that the data ratio is higher than the preset data ratio threshold, discard the target missing value from the filled normalized data to obtain missing value processed data;

[0187] An outlier processing unit is used to determine outliers in the missing value processed data by a preset outlier detection method, and process the outliers according to preset business rules to obtain pre-processed data.

[0188] In some specific implementations, the first semantic recognition module 12 may specifically include:

[0189] a data analysis unit, configured to analyze the pre-processed data in the preset time window to obtain an analysis result including data content features;

[0190] a rule matching unit, configured to match the data content features in the analysis result with a preset subgraph determination rule to determine a corresponding preset large model subgraph;

[0191] a version determining unit, configured to determine a target subgraph version with the best performance from among several subgraph versions of the preset large model subgraph based on historical operation performance data of the preset large model subgraph;

[0192] The loading strategy determination unit is used to determine the preset subgraph loading strategy for this subgraph loading based on the current resource utilization, so as to load the preset large model subgraph whose subgraph version is the target subgraph version from the model registration center according to the preset subgraph loading strategy.

[0193] In some specific implementations, the second semantic recognition module 13 may specifically include:

[0194] a score determination unit, configured to determine a confidence score corresponding to the first recognition result;

[0195] a semantic recognition unit, configured to compare the confidence score with a preset confidence score threshold, and if the confidence score is lower than the preset confidence score threshold, determine that the verification fails, and perform semantic recognition on the preprocessed data using a preset lightweight model to obtain a corresponding second recognition result;

[0196] A fusion strategy determination unit, configured to determine a corresponding preset fusion strategy according to a task type of the target data recognition task;

[0197] A result fusion unit is used to perform a fusion operation on the second recognition result and the first recognition result by using the preset fusion strategy to obtain a target recognition result for the target data recognition task.

[0198] In some specific implementations, the model updating module 14 may specifically include:

[0199] A first condition determination unit is configured to determine a confidence score corresponding to the target recognition result, and calculate an average confidence of the current preset large model based on the confidence score. If it is determined that a decrease in the average confidence compared to a historical average confidence satisfies a preset confidence decrease condition, it is determined that concept drift has occurred in the preset large model;

[0200] Alternatively, a second condition determination unit is configured to utilize the checkpoint mechanism of Flink to determine the current data distribution corresponding to the current target recognition result based on a preset test frequency, and to perform a KS test on the current data distribution and the historical data distribution to determine the corresponding cumulative distribution function difference, and to determine that concept drift has occurred in the target model after the cumulative distribution function difference is higher than a preset cumulative distribution function difference threshold.

[0201] In some specific implementations, the model updating module 14 may specifically include:

[0202] A data collection unit, configured to collect a preset number of newly added real-time data from the current real-time data stream of the Flink;

[0203] A model updating unit, configured to update the target model using the newly added real-time data and a preset model optimization algorithm to obtain corresponding updated model parameters;

[0204] A parameter synchronization unit is used to synchronize the updated model parameters to the pending tasks in the current task queue based on the broadcast function of Flink, so as to execute the pending tasks using the updated target model.

[0205] Furthermore, the embodiment of the present application also discloses an electronic device, Figure 3 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram should not be considered as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the method for completing the data recognition task of Flink and the large model in collaboration disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0206] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device. The communication protocol it follows is any communication protocol that can be applied to the technical solution of this application and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world. Its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0207] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or CD, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0208] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222. It can be Windows Server, NetWare, Unix, Linux, etc. In addition to including computer programs capable of completing the method for completing the data recognition task of Flink and the large model collaboration performed by the electronic device 20 as disclosed in any of the aforementioned embodiments, the computer program 222 can further include computer programs capable of completing other specific tasks.

[0209] Furthermore, this application discloses a computer-readable storage medium for storing a computer program. When executed by a processor, the computer program implements the aforementioned method for completing data recognition tasks using Flink and a large model. The specific steps of this method can be found in the corresponding content disclosed in the aforementioned embodiments and will not be further elaborated here.

[0210] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from the other embodiments. Reference can be made to the descriptions of the identical or similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and the relevant parts can be referred to the descriptions of the methods.

[0211] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0212] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0213] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise," "include," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus comprising a set of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus comprising the element.

[0214] The above is a detailed introduction to the technical solution provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.< / outputdata> < / outputdata> < / modelupdate> < / data> < / modelupdate> < / modelrequest> < / multimodaldata>

Claims

1. A method for completing data recognition tasks by collaborating with Flink and a large model, characterized in that: include: Preprocess multimodal data in real-time data streams based on Flink, and allocate the preprocessed data to corresponding preset time windows according to preset business requirements; The multimodal data includes text data, image data, audio data, and video data; the real-time data stream is a real-time data stream to be identified for performing a target data identification task; Analyzing the preprocessed data in the preset time window to load a corresponding preset large model subgraph from a model registration center according to the obtained analysis result, and performing semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; The preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; The preset large model is a model whose scale meets the preset large-scale determination conditions; Performing a confidence check on the first recognition result, and if the check fails, performing semantic recognition on the preprocessed data using a preset lightweight model, and fusing the obtained second recognition result with the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; The preset lightweight model is a model whose parameters meet the preset lightweight determination conditions; Concept drift detection is performed on the target model using the Flink and the target recognition result. After determining that concept drift has occurred in the target model, the target model is updated based on the Flink so as to perform the data recognition task using the updated target model; the target model includes the preset large model and the preset lightweight model.

2. The method for completing data recognition tasks by collaborating with Flink and a large model according to claim 1 is characterized in that: The Flink-based preprocessing operation on multimodal data in the real-time data stream to allocate the obtained preprocessed data to corresponding preset time windows according to preset business requirements includes: Perform data cleaning operations on multimodal data in real-time data streams based on a preset Flink engine to process missing values ​​and outliers in the multimodal data and obtain preprocessed data. Extracting data type features corresponding to the preprocessed data according to the data type of the preprocessed data and the target data identification task; Calibrating the timestamps of the preprocessed data to unify the time base of the preprocessed data; Through the time window mechanism of Flink, the pre-processed data with unified time bases and the corresponding data type characteristics are allocated to corresponding preset time windows according to preset business requirements, and the delayed data in the preset time windows are processed using the event time processing function of Flink to address the data disorder problem in the preset time windows.

3. The method for completing data recognition tasks by collaborating with Flink and a large model according to claim 2 is characterized in that: The Flink-based preset engine performs a data cleaning operation on the multimodal data in the real-time data stream to process missing values ​​and abnormal values ​​in the multimodal data to obtain preprocessed data, including: Based on Flink's preset engine, the multimodal data in the real-time data stream is formatted and standardized according to the preset data format standards to obtain standardized data; Filling missing values ​​in the standardized data using a preset missing value filling technique, and determining a data ratio of target missing values ​​in the standardized data; the target missing values ​​are missing values ​​in the standardized data that cannot be filled using the preset missing value filling technique; Comparing the data ratio with a preset data ratio threshold, and if the comparison result indicates that the data ratio is higher than the preset data ratio threshold, discarding the target missing value from the filled normalized data to obtain missing value processed data; The outliers in the missing value processed data are determined by a preset outlier detection method, and the outliers are processed according to preset business rules to obtain pre-processed data.

4. The method for completing data recognition tasks by collaborating with Flink and a large model according to claim 1 is characterized in that: The analyzing the pre-processed data in the preset time window to load the corresponding preset large model subgraph from the model registration center according to the obtained analysis result includes: Analyzing the pre-processed data in the preset time window to obtain an analysis result including data content features; Matching the data content features in the analysis results with preset subgraph determination rules to determine a corresponding preset large model subgraph; Determining a target subgraph version with the best performance from among several subgraph versions of the preset large model subgraph based on historical operation performance data of the preset large model subgraph; A preset subgraph loading strategy for this subgraph loading is determined based on current resource utilization, so as to load the preset large model subgraph whose subgraph version is the target subgraph version from the model registration center according to the preset subgraph loading strategy.

5. The method for completing data recognition tasks by collaborating with Flink and a large model according to claim 1 is characterized in that: The confidence check is performed on the first recognition result, and if the check fails, semantic recognition is performed on the preprocessed data using a preset lightweight model, and a second recognition result and the first recognition result are fused using a preset fusion strategy to obtain a target recognition result for the target data recognition task, including: Determining a confidence score corresponding to the first recognition result; Comparing the confidence score with a preset confidence score threshold; if the confidence score is lower than the preset confidence score threshold, determining that the verification fails, and performing semantic recognition on the preprocessed data using a preset lightweight model to obtain a corresponding second recognition result; Determining a corresponding preset fusion strategy according to the task type of the target data recognition task; The preset fusion strategy is used to perform a fusion operation on the second recognition result and the first recognition result to obtain a target recognition result for the target data recognition task.

6. The method for completing data recognition tasks by collaborating with Flink and a large model according to claim 1 is characterized in that: The performing concept drift detection on the target model using the Flink and the target recognition result includes: Determine the confidence score corresponding to the target recognition result, and calculate the average confidence of the current preset large model based on the confidence score. If it is determined that the decrease in the average confidence compared to the historical average confidence satisfies a preset confidence decrease condition, it is determined that concept drift has occurred in the preset large model; Alternatively, the checkpoint mechanism of Flink is utilized to determine the current data distribution corresponding to the current target recognition result based on a preset test frequency, and a KS test is performed on the current data distribution and the historical data distribution to determine the corresponding cumulative distribution function difference. When the cumulative distribution function difference is higher than a preset cumulative distribution function difference threshold, it is determined that concept drift has occurred in the target model.

7. The method for completing data recognition tasks in collaboration between Flink and a large model according to any one of claims 1 to 6, characterized in that: Updating the target model based on Flink so as to perform the data recognition task using the updated target model includes: Collect a preset number of new real-time data from the current real-time data stream of the Flink; The target model is updated using the newly added real-time data and a preset model optimization algorithm to obtain corresponding updated model parameters; Based on the broadcast function of Flink, the updated model parameters are synchronized to the pending tasks in the current task queue, so that the pending tasks are executed using the updated target model.

8. A device for completing data recognition tasks by collaborating with Flink and a large model, characterized in that: include: The data preprocessing module is used to preprocess the multimodal data in the real-time data stream based on Flink, so as to allocate the obtained preprocessed data to the corresponding preset time window according to the preset business requirements; the multimodal data includes text data, image data, audio data, and video data; the real-time data stream is the real-time data stream to be identified for the target data recognition task; a first semantic recognition module, configured to analyze the preprocessed data in the preset time window, load a corresponding preset large model subgraph from a model registration center according to the obtained analysis result, and perform semantic recognition on the preprocessed data based on the preset large model subgraph to obtain a corresponding first recognition result; The preset large model subgraph is an independent large model subgraph obtained by splitting the preset large model according to the type of data processing function; The preset large model is a model whose scale meets the preset large-scale determination conditions; a second semantic recognition module, configured to perform a confidence check on the first recognition result, and if the check fails, perform semantic recognition on the preprocessed data using a preset lightweight model, and fuse the obtained second recognition result with the first recognition result using a preset fusion strategy to obtain a target recognition result for the target data recognition task; The preset lightweight model is a model whose parameters meet the preset lightweight determination conditions; A model update module is configured to perform concept drift detection on a target model using Flink and the target recognition result, and, upon determining that concept drift has occurred in the target model, update the target model based on Flink so as to perform data recognition tasks using the updated target model; the target model includes the preset large model and the preset lightweight model.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor is configured to execute the computer program to implement the method for completing a data recognition task by collaborating with Flink and a large model as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that Used to store a computer program; wherein, when the computer program is executed by a processor, the method for completing a data recognition task in collaboration between Flink and a large model as described in any one of claims 1 to 7 is implemented.

Images