Smart city data and vision-language pre-training model bidirectional perception dialogue method

Abstract

The invention relates to a smart city data and vision-language pre-training model bidirectional perception dialogue method comprising the following steps: S1, obtaining metadata, and generating metadata DNA information of each data; s2, acquiring training and reasoning process data of the vision-language pre-training model, and generating a model utility report; s3, constructing a data agent, wherein the data agent generates a data processing appeal according to the metadata DNA information and the model utility report; and S4, updating the metadata or metadata DNA information according to the data processing appeal. The method has the beneficial effects that effective perceptual conversation can be performed between the data and the model, and the data management efficiency is high.

Description

Technical Field

[0001] This invention relates to the field of smart city data management technology, specifically to a two-way perception dialogue method between smart city data and a vision-language pre-trained model. Background Technology

[0002] With the continuous development of big data technology, data is rapidly expanding and growing in size. To facilitate the processing and management of this data, data models are becoming increasingly widespread and used in various fields. In scenarios such as video surveillance data analysis in smart cities and intelligent operation and maintenance of industrial equipment, systems generate massive amounts of multimodal data daily, including images and text logs. When using these multimodal data models, it is typically necessary to train and optimize various attention-based visual recognition or natural language processing models using this data. This ensures that the data processing models can adapt to data processing tasks, automating tasks such as anomaly detection and event classification, and improving the efficiency of data processing in smart city analysis and intelligent operation and maintenance scenarios. Currently, the management of this multimodal data after collection is unidirectional and static. After collection, storage, and processing, the data is provided as static material to the model training process. The model training team selects data based on experience; the system cannot identify which data is most valuable or which is outdated. Furthermore, data administrators and model trainers cannot quantify the potential value of a single data point in future training, preventing the precise utilization of data resources.

[0003] The effects of model training cannot be effectively and automatically fed back to the data management level to guide data updates, culling, or enrichment. Data remains completely passive, its lifecycle management relying on manual strategies and unable to adaptively adjust to the model's actual needs. Similarly, the data itself is unaware of its impact on the model and cannot proactively request optimization or attention. A large amount of useless or low-value data occupies storage and computing resources, while high-value data that could truly improve model performance may not be discovered and utilized in a timely manner. An insurmountable gap exists between data and the model, resulting in inefficient data utilization and highly arbitrary data selection for model training. Summary of the Invention

[0004] The purpose of this invention is to provide a bidirectional perception dialogue method between smart city data and a vision-language pre-trained model, in order to solve the technical problem of low data management efficiency in existing technologies where multimodal data and data processing models in smart cities cannot achieve effective perception.

[0005] The technical solution adopted by this invention to solve its technical problem is: a two-way perception dialogue method for smart city data and vision-language pre-trained models, comprising the following steps: S1. Obtain smart city operation and maintenance metadata and generate metadata DNA information for each type of smart city operation and maintenance metadata. S2. Obtain data on the training and inference process of the visual-language pre-trained model and generate a model utility report; S3. Construct a data intelligence agent, which generates data processing requests based on metadata DNA information and model utility reports. S4. Update the smart city operation and maintenance metadata or metadata DNA information according to data processing requirements.

[0006] The significant effects of this invention are as follows: By establishing metadata DNA information, this solution enables smart city operation and maintenance metadata to possess machine-readable archival information, allowing it to have a perceived identity and capabilities. By monitoring information during the training and inference process of the visual-language pre-training model and generating a structured model utility report, various information of a single data point during the training and inference process is quantitatively recorded. This facilitates dialogue between data and the model through the reading of parameters in the model utility report. Furthermore, by constructing data as a data intelligent agent—an autonomous software entity—data can generate decisions based on the matching degree between its own information and model indicators. This data intelligent agent allows data to proactively perceive the usage of smart city data in the visual-language pre-training model based on metadata DNA information and the model utility report, transforming data from a passive asset to an active perceiver. This enables data to express its own state and needs. In the process of data management, resource allocation (computation, storage, governance) shifts from being based on fixed strategies to being based on real-time and accurate data demands, significantly improving resource utilization.

[0007] Furthermore, in step S1, the metadata DNA information includes the semantic vector identifier, quality score, lineage record, value decay curve, and expected utility label of the smart city operation and maintenance metadata. This creates a dynamic and computable description system that transcends traditional metadata, encapsulating the semantics, quality, lineage, and expected value of the data into a machine-understandable and processable digital gene. This lays the data foundation for two-way perception and facilitates more accurate matching of the specific data processing requirements when comparing and analyzing the data with model utility reports.

[0008] Furthermore, in step S2, the model utility report includes data contribution, attention focus, prediction confidence, prediction accuracy, and performance on related tasks. During the training and inference process of the visual-language pre-trained model, quantitative indicators reflecting the contribution of individual data points and the model's attention are collected to form feedback on the use of smart city data by the visual-language pre-trained model, thus establishing a perception loop from the visual-language pre-trained model to smart city data.

[0009] Furthermore, the data contribution analysis includes: A1. Determine the checkpoints in the training process of the visual-language pre-trained model, and determine the reference points for data contribution analysis in the training process of the visual-language pre-trained model. A2. Calculate the training gradient of the training sample at the checkpoint and the reference gradient at the reference point; A3. Calculate the data contribution based on the training gradient of the checkpoint and the reference gradient of the reference point.

[0010] Furthermore, step A3 specifically involves: A301. Calculate the gradient dot product of the checkpoints based on the training gradients of the training samples at the checkpoints and the reference gradients at the reference points: In the formula, The gradient dot product of the checkpoints, For the summation function, To train the gradient, Use the reference gradient; A302. The total contribution is obtained by weighted summation of the gradient dot products of all checkpoints: In the formula, For total contribution, Number of checkpoints For the first Time decay weight of each checkpoint For the first Gradient dot product of checkpoints.

[0011] Furthermore, the specific steps of the attention focus analysis include: B1. Obtain the model attention weights of the vision-language pre-trained model, aggregate the model attention weights and normalize them to obtain normalized weights. B2. Map the normalized weights to the semantic labels to obtain the mapped attention weights; B3. Perform focus point quantification analysis on the mapped attention weights to obtain the key focus of the model; B4. Calculate the focus coverage rate based on the normalized weights; B5. Obtain the focus of attention based on key focus and focus coverage.

[0012] Attention focal points are used to record which semantic labels the visual-language pre-trained model primarily focuses on during inference, when processing the data. The attention mechanism indicates which parts of the input the visual-language pre-trained model pays more attention to. By recording these focal points, we can understand which parts of the data the visual-language pre-trained model bases its decisions on.

[0013] Furthermore, the calculation steps for prediction confidence and prediction accuracy are as follows: C1. Input test data into the vision-language pre-training model, and the vision-language pre-training model performs prediction processing on the test data to obtain the predicted raw score; C2. Convert the predicted raw scores into category probability values; C3. The highest value among the category probability values ​​is taken as the prediction confidence level for this prediction process; C4. Obtain the true labels of the test data and compare the prediction result corresponding to the prediction confidence with the true labels. If the prediction result matches the true label comparison result, the prediction accuracy is yes; otherwise, the prediction accuracy is no.

[0014] Furthermore, the specific steps for analyzing the performance of related tasks are as follows: D1. Determine the performance metrics for each model task and set the baseline performance for each model task; D2. Classify the test data according to the model task and record the usage information of the test data in the model task; D3. Based on the classification results and usage information of the test data in the model task, analyze the impact of the test data on the changes in model performance and obtain the performance of the related task.

[0015] By establishing a quantifiable causal relationship between data usage and the performance of visual-language pre-trained models, the system provides objective and accurate decision-making basis for why and how data governance is necessary. Through continuous performance correlation analysis, the system can consistently identify high-value data features that truly improve the performance of visual-language pre-trained models and recognize invalid or interfering data. This allows data storage, computing, and annotation resources to be continuously allocated to high-value data while eliminating low-value data, fundamentally improving the return on investment of data assets.

[0016] Furthermore, step D3 specifically involves: D301. Calculate the performance indicators corresponding to the model task according to the preset time window; D302. The trend of computational performance indicators over time window; D303. Analyze the correlation between test data usage information and performance indicator change trends. The correlation between test data usage and performance indicator change trends includes positive correlation, negative correlation, and no correlation.

[0017] Furthermore, in step S3, the specific steps for generating the data processing request include: E1. The data agent acquires metadata DNA information and model utility reports at a preset frequency, and listens for event information. E2. Match the metadata DNA information, model utility report, and event information obtained by the data agent with preset rules to obtain data processing decisions; the data processing decisions include requesting knowledge enrichment, requesting priority use, and requesting archiving or deletion. E3. Obtain data processing requirements based on data processing decisions.

[0018] Data intelligence agents, through the perception results of metadata DNA and model utility reports, enable high-value data to proactively request processing, and low-value data to proactively request archiving or deletion. This helps model training teams quickly locate high-quality data, avoiding blindly searching through a sea of ​​data, shortening model iteration cycles, and greatly improving the efficiency of discovering and utilizing model training data. Furthermore, each data entity is assigned a logical agent, enabling it to determine its state based on its own state (metadata DNA information) and external feedback (model utility reports) according to predefined rules, and proactively issue governance requests, achieving autonomous data management. Attached Figure Description

[0019] Figure 1 This is a flowchart of the two-way perception dialogue method for smart city data and vision-language pre-trained model in an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, a clear and complete description will be provided below in conjunction with the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the protection scope of the present invention.

[0021] See appendix Figure 1 The smart city data and vision-language pre-trained model bidirectional perception dialogue method shown includes the following steps: S1. Obtain smart city operation and maintenance metadata and generate metadata DNA information for each piece of metadata. Specifically, after the smart city operation and maintenance metadata is accessed by the system, the content of the smart city operation and maintenance metadata is analyzed, and the analysis results are used to generate a lightweight and machine-readable metadata DNA file associated with the metadata. The metadata DNA file can be a JSON-LD format file. The smart operation and maintenance metadata includes multimodal data such as images and text logs generated in scenarios such as smart city video surveillance data and industrial equipment operation and maintenance data. The metadata DNA information for smart city operation and maintenance includes semantic vector identifiers, quality scores, lineage records, value decay curves, and expected utility labels. Semantic vector labels can be generated from data content vectors using models such as Sentence-BERT to represent the core semantics of the data. The quality score is used to assess the quality of data. The quality score includes the basic quality score, semantic consistency score, and historical utility score. Bloodline records are used to document the source of this data and the relevant parameters of the data processing process; The value decay curve is used to represent the trend of the value of the data decaying over time. The value decay curve can be predicted based on historical data through a time series model. The expected utility label is used to represent the ability of the data to contribute to the model, specifically the ability to enhance spatial reasoning, enhance event causal judgment, etc. The expected utility label can be preset by the administrator or the system.

[0022] The specific details of semantic vector labels, quality score assessment, and value decay curve prediction generated by the model are existing technologies and will not be elaborated here.

[0023] S2. Obtain training and inference process data for the vision-language pre-trained model and generate a model utility report; whereby the vision-language pre-trained model (Vision-Language) Models (VLMs) are machine learning models with attention mechanisms. Specifically, in this embodiment, the visual-language pre-training model is a Transformer architecture model. The visual-language pre-training model is used to analyze and output information such as anomaly detection and event recognition classification generated during the smart city and intelligent operation and maintenance process through visual recognition or natural language processing inputs, such as images and text. The specific structure and data processing of the visual-language pre-training model are existing technologies and will not be described in detail here. When generating the model utility report, the state and data information of the visual-language pre-training model during the training and inference process are collected, and the model utility report is generated based on the collected model state and data. When collecting the state and information of the visual-language pre-training model during the training and inference process, a monitoring hook can be implanted in the visual-language pre-training model. The monitoring hook is a non-intrusive code interception technology that inserts monitoring points at key nodes of program execution to achieve real-time observation and data collection of the system's operating status. The specific content and implementation of the model hook are existing technologies and will not be described in detail here.

[0024] The model utility report includes data contribution, attention focus, prediction confidence, prediction accuracy, and related task performance. Data contribution is used to indicate the degree to which each training sample data contributes to the final parameters of the visual-language pre-trained model or the performance of a specific test set during the training process. Attention focus; When analyzing the contribution of data, the training state and data information obtained from the model hook can be acquired. The contribution of each training sample to the final parameters of the vision-language pre-trained model or the performance of a specific test set can be estimated using the influence function approximation method. Specifically, in this embodiment, TracIn is used to track the impact of training samples on the model parameters throughout the training process, and the contribution of each training sample is quantified by analyzing gradient similarity. The specific steps include: A1. Determine checkpoints in the training process of the visual-language pre-training model and determine reference points for data contribution analysis in the training process of the visual-language pre-training model. Specifically, when determining checkpoints, the parameters of the visual-language pre-training model in the prediction iteration training are periodically saved during the training process. These checkpoints represent the state of the visual-language pre-training model at different training and learning stages. Specifically, in this embodiment, the parameters of the visual-language pre-training model at the 100th, 500th, and 1000th iterations are saved as checkpoints.

[0025] When selecting reference points, the prediction results of the visual-language pre-trained model on different sample data are selected as reference points according to different evaluation objectives. Specifically, in this embodiment, the evaluation objectives may include the final model performance, the effect of specific test samples, and the validation set performance. The reference point for the final model performance evaluation objective is the model at the end of training. The specific test sample effect evaluation objective uses the prediction effect of the test data as a reference. The validation set performance evaluation objective uses the accuracy of the overall validation set as a reference.

[0026] A2. Calculate the training gradient of the training sample at the checkpoint and the reference gradient at the reference point; specifically, calculating the training gradient at the checkpoint includes the following steps: A201. Input the training samples into the visual-language pre-training model of the corresponding checkpoint, and calculate the predicted value of the training samples through the visual-language pre-training model; A202. Calculate the error between the predicted value and the true value. When calculating the error, a standard loss function can be used to calculate the error between the predicted value and the true value. Specifically, in this embodiment, "cross-entropy" is used for classification calculation and mean squared error is used for regression calculation. The specific content of calculating the error value through the loss function is existing technology and will not be described in detail here.

[0027] A203. Calculate the training gradient vector based on the error value, and use the training gradient vector as the reference gradient of the reference point. When calculating the training gradient vector, backpropagation of the error value is performed using automatic differentiation to calculate the partial derivative of the error value with respect to each parameter in the visual-language pre-trained model. These partial derivatives are combined to obtain the gradient vector of the training sample in the current checkpoint state of the visual-language pre-trained model.

[0028] The calculation of the reference gradient at the reference point specifically includes the following steps: A211. Input the test data corresponding to the reference point into the visual-language pre-trained model with the same checkpoint as the training gradient, and calculate the reference point loss. When the reference point is a single test sample, input the test sample and its true label, and calculate the reference point loss using the standard loss function consistent with the training phase. The specific formula for calculating the loss of a single test sample for the reference point is as follows: In the formula, The loss of the test sample is used as a reference point. For the standard loss function during the training phase, The parameter state of the visual-language pre-trained model corresponding to the t-th checkpoint. For test samples, The true labels for the test samples; When the reference point is the validation set, the overall loss of the validation set is calculated by averaging the losses of all validation samples. The specific formula for calculating the overall loss of the validation set is as follows: In the formula, The overall loss of the validation set is used as a reference point. For the size of the validation set, Standard loss function for the training phase To verify the first set One sample, To verify the first set The true labels of each sample; A212. Calculate the reference point gradient based on the reference point loss; specifically, when the reference point is a single test sample, the formula for calculating the reference point gradient is: In the formula, The loss for a single test sample reference point is relative to the visual-language pre-trained model at checkpoints. parameters gradient vector, For parameters Elevation step for partial derivatives Loss of the test sample at the reference point; When the reference point is the validation set, the formula for calculating the gradient of the reference point is: In the formula, To validate the reference point loss relative to the visual-language pre-trained model at checkpoints parameters gradient vector, The overall loss is used as the reference point to validate the set. A3. Calculate the data contribution based on the training gradient of the checkpoint and the reference gradient of the reference point; this includes the following steps: A301. Calculate the gradient dot product of the checkpoints based on the training gradients of the training samples at the checkpoints and the reference gradients at the reference points. This dot product reflects the similarity between the two gradient directions. The specific formula for calculating the gradient dot product of the checkpoints is: In the formula, The gradient dot product of the checkpoints, For the summation function, To train the gradient, Use the reference gradient; A302. The total contribution is obtained by weighted summation of the gradient dot products of all checkpoints. The formula is: In the formula, For total contribution, Number of checkpoints For the first Time decay weight of each checkpoint For the first The gradient dot product of checkpoints; where the time decay weight is smaller when the checkpoint is in the early stage of the visual-language pre-training model training, and larger when the checkpoint is near the end of the visual-language pre-training model training. The time decay weight is usually related to the learning rate. This weight is used to distinguish the importance of checkpoints at different training stages of the visual-language pre-training model when calculating the total contribution. The closer the checkpoint is to the end of the visual-language pre-training model training stage, the more direct and important the influence of its gradient direction on the final visual-language pre-training model is usually.

[0029] Specifically, in this embodiment, when calculating the contribution of training image A and test image B, the 100th iteration of the visual-language pre-training model is selected as checkpoint 1, the 500th iteration as checkpoint 2, and the 1000th iteration as checkpoint 3. The gradients of training image A and test image B at each checkpoint are shown in Table 1. Table 1 Gradients of Training and Testing Images In this embodiment, the weight of checkpoint 1 is 0.2, the weight of checkpoint 2 is 0.3, and the weight of checkpoint 3 is 0.5, meaning the total contribution is: When the total contribution is positive, it means that the training samples help improve the performance of the reference point; when the contribution is negative, it means that the training samples harm the performance of the reference point; when the contribution is zero, it means that the training samples have no significant impact on the reference point.

[0030] The data contribution score is also used to update the quality score of the metadata DNA. Specifically, after calculating the total contribution score of the data, the system updates the historical utility score of the quality score in the metadata DNA based on the currently calculated total contribution score. In this embodiment, the formula for updating the historical utility score is as follows: In the formula, For the updated historical utility score, As a weighting factor, This represents the current total contribution. Attention focal points are used to record which semantic labels the visual-language pre-trained model primarily focuses on during inference, when processing the data. The attention mechanism indicates which parts of the input the visual-language pre-trained model pays more attention to. By recording these focal points, we can understand which parts of the data the visual-language pre-trained model bases its decisions on. The specific steps of attention focal point analysis include: B1. Obtain the model attention weights of the visual-language pre-trained model, aggregate the model attention weights and normalize them to obtain normalized weights; where, when obtaining the attention weights of the visual-language pre-trained model.

[0031] Aggregating model attention weights specifically includes: The process involves determining whether the visual-language pre-trained model has multiple attention layers. If so, the attention layers selected for aggregation computation are chosen. Typically, the attention weights of the last few layers that have the most direct impact on decision-making are selected. The attention weights of the selected layers are then weighted or simply averaged to form a uniform attention distribution. Attention layer selection can be based on fixed rules based on task characteristics or dynamic analysis based on weight importance. The fixed rules based on task characteristics are based on a general understanding of model architectures such as Transformer: shallow layers (layers 1-4) typically learn local grammar and phrase-level features, middle layers (layers 5-8) learn sentence-level semantic relationships, and deep layers (layers 9-12 and beyond) capture global semantics and high-level abstract information most relevant to the final task objective. Therefore, the system defaults to a fixed ratio rule based on the total number of model layers to select aggregation layers. For example, it aggregates the last 1 / 3 of the layers (layers 9-12 for a 12-layer model), or the last two layers. This rule can be preset through a configuration file and validated on different tasks.

[0032] In dynamic analysis based on weighted importance, to achieve more accurate fitting, the system supports an offline analysis module. This module runs on a small calibration dataset before deploying the vision-language pre-trained model and performs the following analyses: Calculate the decision relevance of each layer of attention: For each sample, record the prediction confidence of the final output layer of the visual-language pre-trained model, and calculate the correlation with the attention weight distribution of each layer, such as calculating the mutual information between it and the prediction confidence, or observing the association between attention concentration and prediction correctness.

[0033] Selecting key layers: Select the layers whose attention distribution has the highest statistical correlation with the final prediction result of the visual-language pre-trained model as the aggregation objects. For example, if the analysis finds that the attention concentration of layers 10 and 11 is highly positively correlated with the prediction accuracy of the visual-language pre-trained model, while the attention of layer 3 is almost unrelated, the system will automatically select layers 10 and 11 for aggregation.

[0034] If the visual-language pre-trained model does not have multiple attention layers, multi-head attention aggregation is performed on the attention layers to obtain a comprehensive attention profile. Each attention layer typically contains multiple heads, and each attention head may focus on different semantic aspects. The weights of all attention heads are averaged to obtain a comprehensive attention profile, representing the overall gaze focus of the visual-language pre-trained model. Specifically, the comprehensive attention profile is calculated by arithmetically averaging the weight matrices of all attention heads in the attention layer. This matrix is ​​the comprehensive attention profile, which integrates the perspectives of multiple heads and more stably reflects the overall attention pattern of the visual-language pre-trained model at that attention layer. The specific calculation methods for the comprehensive attention profile are existing techniques and will not be elaborated upon here.

[0035] The aggregated attention weights are then normalized. The aggregated attention weight vector (typically corresponding to the attention given to all other positions by a particular query position) is normalized using the Softmax function. This function transforms a set of arbitrary real numbers into a probability distribution, ensuring that each element is non-negative and the sum of all elements is 1. The specific normalization formula is as follows: In the formula, To normalize the weights, It is a natural exponential function. Let the attention score vector be... All are indexes of semantic tags. This represents the total number of semantic units in the current input data. B2. Mapping normalized weights to semantic labels yields mapped attention weights; specifically, the mapping process includes text data mapping, image data mapping, and multimodal data mapping. In the text data mapping process, the input text data is segmented into basic semantic labels (such as words, phrases, or characters). The attention weight of each semantic label is associated with the semantic meaning itself. Specifically, in this embodiment, for the sentence "traffic light malfunction at the intersection", after mapping, it was found that the visual-language pre-trained model invested the most attention in "traffic light" (weight 0.6) and "malfunction" (weight 0.3).

[0036] When mapping image data, the image is divided into different regions (such as image patches in a Vision Transformer). A heatmap is generated based on the attention weight of each region; the darker the color, the more attention the visual-language pre-trained model pays to that region. Specifically, in this embodiment, in a street view image, the visual-language pre-trained model may focus high attention on traffic signs and vehicles.

[0037] When mapping multimodal data, attention is handled separately for text and image portions. Cross-modal attention is also analyzed, for example, which word in the text focuses on which region of the image.

[0038] The specific details of mapping weights to data are existing technologies and will not be elaborated here.

[0039] B3. Perform focus point quantification analysis on the mapped attention weights, extract meaningful indicators, and obtain the key focus of the model; specifically including: B301. Obtain the semantic labels of the top K mapping attention with the highest attention weights to obtain the key labels of the model; specifically, in this embodiment, obtain the semantic labels of the top 3 mapping attention with the highest attention weights ["streetlight", "not lit", "People's Road"]; directly indicating which specific information the visual-language pre-trained model is most concerned with.

[0040] B302. Calculate the attention concentration of each model's key labels. This calculation involves calculating the information entropy of each model's key labels. A low entropy value indicates high attention concentration, meaning the visual-language pre-trained model relies on a few key pieces of information for decision-making. A high entropy value indicates scattered attention, meaning the visual-language pre-trained model integrates multiple information sources. The formula for calculating information entropy is: In the formula, The information entropy value of the key labels in the model. The total number of keywords. For the first Normalized attention weights corresponding to each key label It is the natural logarithm function; B303. Calculate the consistency of key labels across models, analyze whether different attention heads focus on the same regions, and obtain the key focus of the models. High consistency indicates that the visual-language pre-trained models have a strong consensus on key label information. Compare the overlap of key label sets (such as Top-K lists) identified by different attention heads. This specifically includes the following steps: B3031. Extract the key labels of each attention head in each attention layer to obtain the key label set; for the attention layer to be analyzed, extract the top K semantic labels with the highest attention weight from each attention head of that attention layer to form the key label set of each head. For example, the first three semantic labels of attention head A are ["streetlight", "not lit", "Renmin Road"], and the first three semantic labels of attention head B are ["streetlight", "fault", "Renmin Road"].

[0041] B3032. Calculate the similarity between key tag sets. Specifically, when calculating the similarity between key tag sets, the Jaccard similarity coefficient can be used as the core quantitative indicator to calculate the degree of overlap between the key tag sets of any two attention heads. The similarity is the ratio of the number of elements in the intersection of the two key tag sets to the number of elements in the union of the two key tag sets. Specifically, in this embodiment, taking head A and head B as examples, the intersection is {"streetlight", "People's Road"} (2 elements), and the union is {"streetlight", "not lit", "People's Road", "faulty"} (4 elements). Then, the Jaccard similarity = 2 / 4 = 0.5.

[0042] B3033. The overall consistency score is obtained by aggregating the similarity between key tag sets. Specifically, the Jaccard similarity between all pairs of attention heads in this attention layer is calculated, and then the average of these similarity values ​​is taken to obtain the focus consistency score of this attention layer. The closer the focus consistency score is to 1, the more consistent the attention areas of different heads are; the lower the score, the more dispersed the attention points are and the lower the consensus.

[0043] B4. Calculate the focus coverage rate, where focus coverage rate is a quantitative metric used to measure how many semantic labels the attention of the vision-language pre-trained model needs to cover to achieve the main part of its overall attention. The specific steps are as follows: B401. Obtain a normalized list of attention weights, where each weight corresponds to a specific semantic label. The semantic label can be composed of data such as words or image patches, and the sum of the weights is 1. Sort the attention weight list in descending order of weight.

[0044] B402. Accumulate the attention weights in the attention weight list according to the descending order of weights. When the accumulated attention weights are equal to or greater than the accumulated weight threshold, record the number of semantic tags that have been accumulated. Otherwise, repeat B402. In this embodiment, the accumulated weight threshold is preferably 0.8, which represents 80% of the total attention. In other embodiments, other threshold parameters can be configured as needed.

[0045] B403. The number of recorded and accumulated semantic tags is used as the focus coverage rate. A small focus coverage rate, such as covering the first 3 units to achieve 80% attention, indicates highly concentrated attention. A large focus coverage rate, such as needing to cover the first 20 units, indicates very scattered attention.

[0046] Specifically, in this embodiment, when calculating the focus coverage rate of a data point with 10 semantic tags, its normalized attention weights, after sorting, are: [0.5, 0.3, 0.1, 0.05, 0.02, ...]. Accumulating the first one: 0.5 < 0.8; accumulating the first two: 0.5 + 0.3 = 0.8 ≥ 0.8; at this point, the accumulated number of units is 2, so the focus coverage rate is 2.

[0047] B5. Obtain attention focal points based on key focal points and focal coverage, and save them in the model utility report in a standardized format. The specific integrated attention focal point information includes the core focal point, distribution features, and contextual information of the attention focal point. The core focal point includes the top K semantic labels sorted by weight and their specific weights. The distribution characteristics of attention focus points include quantitative indicators such as attention concentration and focus coverage.

[0048] The contextual information for attention focus points includes data such as data ID, timestamp, and model version.

[0049] When saving attention focus points, structured formats such as JSON are used for storage, facilitating subsequent querying and analysis. These are then linked to the data's metadata DNA information to form a complete data-model interaction archive.

[0050] Prediction confidence and prediction accuracy record the output confidence and judgment correctness of the visual-language pre-trained model for this data, respectively. The specific calculation steps for prediction confidence and prediction accuracy are as follows: C1. Input test data into the visual-language pre-training model. The visual-language pre-training model performs prediction processing on the test data to obtain the predicted raw score. Specifically, in this embodiment, the visual-language pre-training model inputs an image of a cat. The visual-language pre-training model performs prediction and recognition processing on the input image and outputs all possible categories identified, such as cat, dog, car, etc., and outputs the score for each category. The larger the value, the more the visual-language pre-training model tends to that category. The combination of these category scores forms the raw score.

[0051] C2. Convert the predicted raw scores into category probability values. Specifically, when converting the raw scores into category probability values, the Softmax function is used to convert these raw scores into category probability values ​​corresponding to all categories, and the sum of all category probability values ​​is 1. The specific details of data processing using Softmax are existing technology and will not be elaborated here. Specifically, in this embodiment, the category probability values ​​output by the visual-language pre-trained model are {cat: 0.85, dog: 0.12, car: 0.03}. This indicates that the visual-language pre-trained model has an 85% confidence level in identifying this as a cat.

[0052] C3. The highest value among the category probability values ​​is taken as the prediction confidence level for this prediction; that is, the prediction confidence level in this embodiment is 0.85.

[0053] C4. Obtain the true labels of the test data and compare the prediction result corresponding to the prediction confidence with the true labels. If the prediction result matches the true label comparison result, the prediction accuracy is yes; otherwise, the prediction accuracy is no.

[0054] Correlation of task performance is used to establish a quantifiable causal relationship between data usage and the performance of the visual-language pre-trained model, providing objective and accurate decision-making basis for why and how data needs to be managed. By correlating each record of data used for training or inference with performance metrics of the visual-language pre-trained model on subsequent specific tasks, such as accuracy and F1 score, it clarifies which data usages are statistically correlated with the improvement or decline in the performance of the visual-language pre-trained model. It identifies the specific task dimension in which the data contributes value; for example, improving recall in nighttime recognition or impairing accuracy in congestion assessment. The specific steps of correlation of task performance analysis are as follows: D1. Determine the performance metrics for each model task and set the baseline performance for each model task. Specifically, in this embodiment, the model tasks include classification tasks, regression tasks, and generation tasks. The performance metrics for classification tasks include accuracy, F1 score, and recall. The performance metrics for regression tasks include mean squared error and mean absolute error. The performance metrics for generation tasks include the best linear unbiased estimate BLUE value and perplexity. When determining the baseline performance of each model task, it is based on the baseline performance of the vision-language pre-trained model on the validation set. Specifically, in this embodiment, the baseline performance of each model task is 5% of the baseline performance on the validation set. D2. Classify the data according to the model task and record the usage information of the data in the model task; specifically, when classifying the data according to the model task, classify the data according to the task type associated with each data point, label the data, and record in which model tasks each data point is used in training or inference, and record the time point and context information of the use.

[0055] D3. Based on the classification results and usage information of the test data in the model task, analyze the impact of the test data on changes in model performance to obtain the performance of the related task; specifically including: D301. Calculate the performance indicators corresponding to the model task according to the preset time window; where the preset time window can be a daily, weekly or other time period. When calculating the performance indicators of the model task, at the end of each time window, the core performance indicators of each model task in the current version of the vision-language pre-trained model are calculated through the model validation set. The specific calculation content of each performance indicator of the model task is existing technology and will not be described in detail here. D302. Calculate the performance index of the model task as a function of time windows; specifically, generate performance trend curves for the calculated model task performance indexes according to the time window sequence, and calculate the performance change value between adjacent time windows; and mark the performance improvement or decline corresponding to each time window based on the performance change value. D303. Analyze the correlation between data usage and performance changes. The correlation between data usage and performance changes includes positive correlation usage, negative correlation usage, and no correlation. Positive correlation usage indicates that the historical utility score in the metadata DNA information related to this type of data will be improved.

[0056] Negative correlation indicates that introducing this data into a vision-language pre-trained model will lead to a decrease in model performance, and this result will immediately trigger a data quality warning. The relevant data will be reviewed, and may be temporarily shelved or used with reduced weight. Simultaneously, a data quality inspection work order will be generated, prompting the administrator or automated process to perform quality review, re-annotation, or cleanup of the relevant data batch. No correlation indicates that the data is unrelated to performance changes; such data is considered neutral or of uncertain utility.

[0057] Specifically, the analysis of the correlation between data usage and performance changes includes: D3031. Within the same time window, aggregate the frequency of data being used for training or inference according to data attributes. The data attributes include information such as data source, label category, and data block ID cluster. The specific content of aggregating data attributes with training or inference frequency is existing technology and will not be elaborated here. D3032. Calculate the correlation coefficient between the change in the usage of data attributes and the change in task performance in subsequent time windows. Specifically, the correlation coefficient can be calculated using statistical methods such as Pearson correlation coefficient or lagged cross-correlation analysis. The specific calculation content is existing technology and will not be elaborated here. D3033. Compare the correlation coefficient with the correlation threshold. When the correlation coefficient is positive and greater than the correlation threshold, the correlation between data usage and performance change is positive. When the correlation coefficient is negative, the correlation between data usage and performance change is negative. Otherwise, the correlation between data usage and performance change is not positive. S3. Construct a data intelligent agent. The data intelligent agent generates data processing requests based on metadata DNA information and model utility reports. Each active smart city operation and maintenance data point is instantiated into a data intelligent agent. The data intelligent agent can operate based on rules and a lightweight machine learning state machine. The specific details of constructing the data intelligent agent are existing technologies and will not be elaborated upon here. As an autonomous software entity, the core of the data intelligent agent's operation is a continuous and dynamic "perception-decision-action" cycle. The specific content of its generated data processing requests includes: E1. The data agent acquires metadata DNA information and model utility reports at a preset frequency, and listens for event information. When acquiring metadata DNA information, the data agent acquires information such as quality score, value decay curve and historical utility related to its own information status in the metadata DNA information. When obtaining the model utility report, extract information related to your own data, such as usage records, data contribution, prediction confidence, and accuracy. When listening for events, listen for newly published system events such as training task startup and global resource status. E2. Based on the metadata DNA information, model utility report information, and event information obtained by the data intelligence agent, and matched with preset rules, a data processing decision is obtained. Specifically, a decision rule library is preset within the data intelligence agent, containing several preset decision rules. The various data obtained by the data intelligence agent are matched with the decision rules. When a data item matches a decision rule, the corresponding decision is output. In this embodiment, the decision rule library includes the following rules: Rule R1 (SOS and Request Enrichment): When the prediction accuracy of the data in the last 10 uses is less than 60% and the data's attention level is greater than the attention threshold, the decision to request knowledge enrichment is triggered.

[0058] Rule R2 (Self-Recommendation): When the data value decay curve prediction is about to fail and the semantic information is highly correlated with the current model training task, a decision is triggered to apply for priority use.

[0059] Rule R3 (Retirement Request): When data has not been used for nearly 3 months and the data value decay score is less than the very low threshold, a decision is made to request archiving or deletion.

[0060] Furthermore, when the data agent's performance analysis of the correlation between smart city operation and maintenance metadata and model performance changes shows a positive correlation, it strengthens its self-recommendation rule, proactively requesting to be used in subsequent corresponding model tasks, marking it as high-value training data, and giving it priority in storage and computing resources. When the performance analysis of the correlation between smart city operation and maintenance metadata and model performance changes shows no correlation, the agent's behavior will remain at the default or conservative state, and it will be given standard priority in system resource allocation, with continuous observation.

[0061] E3. Obtain data requests based on data processing decisions; the data agent transforms the data processing decisions into standard data processing request information and publishes it through a message queue.

[0062] S4. Update smart city operation and maintenance metadata or metadata DNA information according to data processing requests. Specifically, this is done by setting up a request executor, which listens for and executes the data request information published by the data intelligence agent. Data processing request execution includes enrichment execution, recommendation execution, and lifecycle execution. Enrichment execution is used when the data processing request is for knowledge enrichment. It calls existing knowledge graph construction or multi-source data fusion capabilities to supplement relevant information to the original data block, changing the content or semantic relationships of the data itself, making it a higher-quality resource.

[0063] When processing data requests for priority access, it is recommended to temporarily increase the weight of the data in the data sampler used for model training, thereby changing the priority or flow of the data in the resource scheduling system without altering the data content itself.

[0064] The lifecycle execution is used to handle data processing requests such as archiving or deletion. When such requests are made, the corresponding data is moved to cold storage or securely deleted, and the metadata is updated.

[0065] Compared to existing technologies, this solution achieves a paradigm shift in data from a passive asset to a proactive service. Data can express its own state and needs, and system resource allocation (including computing, storage, and governance) changes from being based on fixed strategies to being based on real-time, precise data demands, significantly improving resource utilization. By embedding dynamic metadata DNA information into each data unit, the data possesses machine-readable archival information containing parameters such as semantics, quality, and expected value, enabling the data to have a perceived identity and capabilities. The data is constructed as a data intelligent agent, which, as an autonomous software entity, continuously executes a "perception-decision-action" cycle, allowing the data to generate decisions based on the matching degree between its own information and model indicators. Furthermore, by designing data demand messages and a message queue-based communication mechanism, a standardized language and channel are provided for the intelligent agent to express its needs to the system, establishing a standardized data expression channel to facilitate data feedback and processing requests. A precise closed loop of model performance-driven data governance has been established. By creating a model utility report, feedback during model training can automatically and instantly trigger optimization actions at the data level, enabling the data system to adapt and evolve along with the model, continuously providing the model with the highest value nutrients.

[0066] By monitoring information during the model training and inference process and generating structured model utility reports, the system quantifies and records information such as the data contribution, attention focus, prediction accuracy, and performance of related tasks for individual data points during model training and inference. Furthermore, the data agent subscribes to and perceives model utility reports relevant to itself. Quantitative indicators in the reports (such as negative contribution or persistently low accuracy) directly serve as core triggering conditions for the agent's decision rule base. This facilitates a clear perception of the potential value of data within the vision-language pre-trained model, enabling the data to proactively make demands based on the perception results, thus facilitating efficient data utilization.

[0067] Data intelligence agents, through the perception results of metadata DNA and model utility reports, enable high-value data to proactively request processing and low-value data to proactively request archiving or deletion. This helps model training teams quickly locate high-quality data, avoid blindly searching in a sea of ​​data, shorten the iteration cycle of vision-language pre-training models, and greatly improve the efficiency of discovering and utilizing model training data.

[0068] Furthermore, by leveraging the value decay curve in the metadata DNA and the data contribution in the utility report, the system can dynamically assess the current value and expected lifespan of each piece of data, achieving dynamic quantification and early warning mechanisms for data value. A task-oriented proactive recommendation mechanism is also designed, proactively requesting priority use of data with high intrinsic value and strong relevance to the current training task. Simultaneously, a value-based automatic elimination mechanism is designed, proactively requesting archiving or deletion of data that has been idle for a long time and has extremely low value. This allows data to proactively propose optimization or request attention, preventing low-value data from occupying storage and computing resources while ensuring that high-value data that can improve model performance is discovered and utilized in a timely manner, thus improving the efficiency of model data management.

[0069] Furthermore, a loosely coupled, event-based system architecture was designed to standardize the requests generated by the data agent into messages, which are then responded to and processed by specialized executor components, thereby transforming the dialogue content into actual system operations and completing the closed loop.

[0070] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific implementation method of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the scope of the present invention should be included within the protection scope of the present invention.

Claims

1. A bidirectional perception dialogue method using smart city data and vision-language pre-trained models, characterized in that, Includes the following steps: S1. Obtain smart city operation and maintenance metadata and generate metadata DNA information for each type of smart city operation and maintenance metadata. S2. Obtain data on the training and inference process of the visual-language pre-trained model and generate a model utility report; S3. Construct a data intelligence agent, which generates data processing requests based on metadata DNA information and model utility reports. S4. Update the smart city operation and maintenance metadata or metadata DNA information according to data processing requirements.

2. The perceptual dialogue method according to claim 1, characterized in that, In step S1, the metadata DNA information includes the semantic vector identifier, quality score, lineage record, value decay curve, and expected utility label of the smart city operation and maintenance metadata.

3. The perceptual dialogue method according to claim 1, characterized in that, In step S2, the model utility report includes data contribution, attention focus, prediction confidence, prediction accuracy, and associated task performance.

4. The perceptual dialogue method according to claim 3, characterized in that, The steps for calculating data contribution include: A1. Determine the checkpoints in the training process of the visual-language pre-trained model, and determine the reference points for data contribution analysis in the training process of the visual-language pre-trained model. A2. Calculate the training gradient of the training sample at the checkpoint and the reference gradient at the reference point; A3. Calculate the data contribution based on the training gradient of the checkpoint and the reference gradient of the reference point.

5. The perceptual dialogue method according to claim 4, characterized in that, Step A3 is as follows: A301. Calculate the gradient dot product of the checkpoints based on the training gradients of the training samples at the checkpoints and the reference gradients at the reference points: In the formula, The gradient dot product of the checkpoints, For the summation function, To train the gradient, Use the reference gradient; A302. The total contribution is obtained by weighted summation of the gradient dot products of all checkpoints: In the formula, For total contribution, Number of checkpoints For the first Time decay weight of each checkpoint For the first Gradient dot product of checkpoints.

6. The perceptual dialogue method according to claim 3, characterized in that, The steps involved in analyzing attention focus include: B1. Obtain the model attention weights of the vision-language pre-trained model, aggregate the model attention weights and normalize them to obtain normalized weights. B2. Map the normalized weights to the semantic labels to obtain the mapped attention weights; B3. Perform focus point quantification analysis on the mapped attention weights to obtain the key focus of the model; B4. Calculate the focus coverage rate based on the normalized weights; B5. Obtain the focus of attention based on key focus and focus coverage.

7. The perceptual dialogue method according to claim 3, characterized in that, The steps for calculating prediction confidence and prediction accuracy are as follows: C1. Input test data into the vision-language pre-training model, and the vision-language pre-training model performs prediction processing on the test data to obtain the predicted raw score; C2. Convert the predicted raw scores into category probability values; C3. The highest value among the category probability values ​​is taken as the prediction confidence level for this prediction process; C4. Obtain the true labels of the test data and compare the prediction result corresponding to the prediction confidence with the true labels. If the prediction result matches the true label comparison result, the prediction accuracy is yes; otherwise, the prediction accuracy is no.

8. The perceptual dialogue method according to claim 3, characterized in that, The specific steps for analyzing the performance of related tasks are as follows: D1. Determine the performance metrics for each model task and set the baseline performance for each model task; D2. Classify the test data according to the model task and record the usage information of the test data in the model task; D3. Based on the classification results and usage information of the test data in the model task, analyze the impact of the test data on the changes in model performance and obtain the performance of the related task.

9. The perceptual dialogue method according to claim 8, characterized in that, Step D3 specifically involves: D301. Calculate the performance indicators corresponding to the model task according to the preset time window; D302. The trend of computational performance indicators over time window; D303. Analyze the correlation between test data usage information and performance indicator change trends. The correlation between test data usage and performance indicator change trends includes positive correlation, negative correlation, and no correlation.

10. The perceptual dialogue method according to claim 1, characterized in that, In step S3, the specific steps for generating the data processing request include: E1. The data agent acquires metadata DNA information and model utility reports at a preset frequency, and listens for event information. E2. Match the metadata DNA information, model utility report, and event information obtained by the data agent with preset rules to obtain data processing decisions; the data processing decisions include requesting knowledge enrichment, requesting priority use, and requesting archiving or deletion. E3. Obtain data processing requirements based on data processing decisions.

Images