Dynamic monitoring and management platform based on vision and sensing fusion

Abstract

The invention provides a dynamic monitoring and management platform based on vision and sensing fusion, and relates to the technical field of dynamic monitoring and management, and the platform comprises a vision sensor module which is used for collecting image data of a target object; the physical sensor module is used for collecting physical parameter data of a target object, and the physical parameter data comprises temperature, humidity, pressure, acceleration and angular velocity; and the data processing module is used for carrying out feature extraction on the visual and physical sensor data, creating a category score matrix, fusing classification results and weights of images and physical features, taking a final category label as a fusion result, and obtaining dynamic state information of the target object according to the fusion result. Through fusion of visual and sensing data, real-time dynamic monitoring and an optimization management strategy based on a firefly algorithm, accurate and efficient state monitoring, early warning, evaluation and maintenance decision making of the target object are realized.

Description

Technical Field

[0001] The present invention relates to the field of dynamic monitoring and management technology, and in particular to a dynamic monitoring and management platform based on vision and sensor fusion. Background Art

[0002] Traditional monitoring methods mostly rely on a single sensor or visual technology, which may limit the monitoring range. For example, in some large industrial scenarios, a single sensor may not be able to cover the entire area, causing some areas to become monitoring blind spots. In terms of accuracy, some traditional sensors may be greatly affected by environmental factors, such as temperature and humidity changes, resulting in errors in monitoring data. For example, in environmental monitoring, traditional temperature and humidity sensors may cause reading drift due to changes in external conditions, affecting the accuracy of the data.

[0003] Secondly, the data processing and transmission speed of traditional monitoring systems may be slow and cannot meet the needs of real-time monitoring. For example, in traffic monitoring, if traditional image processing technology is relied upon for vehicle identification and counting, data delays may occur due to processing speed limitations, and traffic conditions cannot be reflected in real time.

[0004] In addition, some traditional sensors may have delays in the process of data collection and transmission, especially in long-distance transmission or poor network conditions, such delays may be more obvious, affecting the real-time and effectiveness of monitoring. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a dynamic monitoring and management platform based on vision and sensor fusion to achieve multi-dimensional and high-precision dynamic monitoring and management.

[0006] In order to solve the above technical problems, the technical solution of the present invention is as follows:

[0007] First, a dynamic monitoring and management platform based on vision and sensor fusion, including:

[0008] A visual sensor module, used to collect image data of the target object;

[0009] A physical sensor module is used to collect physical parameter data of the target object, wherein the physical parameter data includes temperature, humidity, pressure, acceleration, and angular velocity;

[0010] The data processing module is used to extract features from visual and physical sensor data, create a category score matrix, fuse the classification results and weights of images and physical features, use the final category label as the fusion result, and obtain the dynamic state information of the target object based on the fusion result;

[0011] The dynamic monitoring module is used to monitor and analyze the dynamic state of the target object in real time based on the fused data through a time series analysis model to obtain analysis results;

[0012] The management module is used to determine the health management objective function based on the analysis results of the dynamic monitoring module, and initialize the firefly population. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions.

[0013] Furthermore, feature extraction is performed on the visual and physical sensor data to create a category score matrix, the classification results and weights of the image and physical features are fused, the final category label is used as the fusion result, and the dynamic state information of the target object is obtained based on the fusion result, including:

[0014] Extract features from image data of visual sensors, including shape features, texture features, and color features; extract features from physical parameter data, including statistical features, time domain features, and frequency domain features;

[0015] The extracted image features and physical features are used as input data, and the data is classified and processed through the classification model to obtain the output result, that is, the category of the target object;

[0016] Create a category score matrix, fuse the image and physical feature classification results with the corresponding weights, and fill in the corresponding positions; calculate the fusion score of each category in the matrix, and determine the final category label as the fusion result based on the fusion score;

[0017] According to the fusion results, the dynamic state information of the target object is obtained, including the position, velocity, acceleration, posture, temperature and humidity information of the target object.

[0018] Furthermore, the extracted image features and physical features are used as input data, and the data is classified and processed through the classification model to obtain the output result, that is, the category of the target object, including:

[0019] Use random forest as a classification model, and input historical sample data, including known feature vectors and corresponding category labels, into the random forest model. By learning the mapping relationship between features and categories in the sample data, a trained random forest model is obtained.

[0020] The image features and physical features are input into the trained random forest model. According to the learned mapping relationship, the input data is processed and the classification result is output. The classification result is a category label indicating the category to which the target object belongs.

[0021] Furthermore, a category score matrix is ​​created, and the classification results of the image and physical features are fused with the corresponding weights respectively and filled in the corresponding positions; the fusion score of each category in the matrix is ​​calculated, and the final category label is determined as the fusion result according to the fusion score, including:

[0022] Create a category score matrix, where the rows represent different category labels and the columns represent the scores of the corresponding categories in the image feature classification results and the physical feature classification results;

[0023] According to the classification results of image features, the score of the corresponding category is multiplied by the weight , and fill in the corresponding position of the score matrix; according to the physical feature classification results, multiply the score of the corresponding category by the weight , and accumulated to the corresponding position of the score matrix;

[0024] For each category, the image classifier obtains the prediction probability of the corresponding category based on the image feature vector, and multiplies the prediction probability by the weight coefficient to obtain the image feature score; the physical feature classifier obtains the prediction probability of the same category based on the physical feature vector, and multiplies the prediction probability by the weight coefficient to obtain the physical feature score;

[0025] Add the image feature score and the physical feature score to get the total score of the corresponding category in the score matrix;

[0026] According to the total score of the corresponding category in the score matrix, the final category label is used as the fusion result.

[0027] Furthermore, based on the fused data, the dynamic state of the target object is monitored and analyzed in real time through a time series analysis model to obtain analysis results, including:

[0028] Arrange the pre-fused data in chronological order to form a time series, where each time point corresponds to a dynamic state feature value of a target object;

[0029] Use historical data, including past fusion data and corresponding dynamic state labels, to train the time series analysis model to obtain a trained time series analysis model;

[0030] The real-time fusion data is input into the trained time series analysis model to conduct real-time monitoring and analysis of the dynamic state, and the dynamic state analysis results of the target object at each time point are output, including state prediction value, anomaly detection alarm, and trend analysis.

[0031] Furthermore, according to the results of dynamic monitoring and analysis, the health management objective function is determined, and the firefly population is initialized. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions, including:

[0032] According to the analysis results of the dynamic monitoring module, the objective function of health management is determined, and the firefly population is initialized according to the objective function;

[0033] Set the flash brightness, attractiveness, and step factor parameters of the fireflies. For each firefly, calculate the objective function value of the corresponding health management strategy as the flash brightness of the corresponding firefly.

[0034] Traverse the firefly population, for each pair of fireflies, calculate the attraction between every two fireflies, and determine the direction and step length of each firefly moving towards the final firefly based on the attraction;

[0035] Update the position of the firefly, that is, update the parameters of the health management strategy to obtain a new set of health management strategy parameters;

[0036] The process of calculating the brightness of fireflies, calculating their attractiveness and updating their positions is repeated until the preset number of iterations is reached. The health management strategy corresponding to the final fireflies is used as the final result for early warning, evaluation and maintenance decision-making.

[0037] Furthermore, in the firefly population, each firefly represents a health management strategy.

[0038] Furthermore, the health management objective function includes:

[0039] Calculate the ratio of the number of correct warnings to the total number of warnings to obtain the accuracy of the warning;

[0040] According to the accuracy of the early warning and the preset weight coefficient, the share of the early warning accuracy in the objective function value is determined;

[0041] For each sample, calculate the square of the difference between the actual value and the predicted value of each sample to obtain the square difference corresponding to each sample, and combine the square differences of all samples to obtain the overall deviation degree between the predicted value and the actual value;

[0042] Based on the sample size and the overall degree of deviation between the predicted value and the actual value, the prediction measurement index is obtained;

[0043] According to the predicted measurement index and the preset weight coefficient, determine the contribution of the predicted part in the objective function value;

[0044] Calculate the product of the total output after maintenance and the time period before maintenance to obtain the expansion value of the output after maintenance in the time dimension;

[0045] Calculate the product of the total output before maintenance and the time period after maintenance to obtain the expansion value of the output before maintenance in the time dimension;

[0046] Obtain the difference between the value of the expanded output after maintenance and the value of the expanded output before maintenance to determine the net gain value of output in the time dimension;

[0047] Calculate the maintenance cost and the product of the pre-maintenance time period and the post-maintenance time period to obtain the total maintenance cost value of the time span;

[0048] According to the net gain value of output in the time dimension and the total maintenance cost value of the time span, the measurement index of maintenance benefit is obtained;

[0049] According to the measurement index of maintenance benefit and the preset weight coefficient, the contribution of the maintenance benefit part in the objective function value is determined;

[0050] The contributions of the warning accuracy part, the prediction part and the maintenance benefit part in the objective function value are accumulated to obtain the final objective function value.

[0051] In a second aspect, a computing device includes:

[0052] one or more processors;

[0053] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic monitoring and management platform based on vision and sensor fusion.

[0054] In a third aspect, a computer-readable storage medium stores a program, which, when executed by a processor, implements the dynamic monitoring and management platform based on vision and sensor fusion.

[0055] The above solution of the present invention includes at least the following beneficial effects:

[0056] Through the collaborative work of the visual sensor module and the physical sensor module, it is possible to simultaneously collect image data and multiple physical parameter data (such as temperature, humidity, pressure, acceleration, and angular velocity) of the target object, providing a rich data foundation for subsequent comprehensive analysis and accurate judgment.

[0057] The data processing module extracts features from visual and physical sensor data and creates a category score matrix. The final fusion result is obtained by fusing the classification results and weights of the image and physical features. This fusion method can make full use of the advantages of different sensors and improve the accuracy and reliability of target object state recognition.

[0058] The dynamic monitoring module uses the fused data to monitor and analyze the dynamic state of the target object in real time through a time series analysis model, which helps to detect the state changes of the target object in a timely manner. The management module determines the health management objective function based on the analysis results of the dynamic monitoring module and initializes the firefly population for optimization. By calculating the attraction to update the strategy parameters, after a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions. This strategy formulation method based on the optimization algorithm can more scientifically determine the health management strategy and improve the accuracy and effectiveness of the decision.

[0059] The entire platform can significantly improve the efficiency and accuracy of monitoring and management of target object status through the fusion of visual and sensor data, real-time dynamic monitoring, and scientific management strategy formulation. This helps to timely discover potential problems, take effective measures for early warning and maintenance, reduce the probability of failure, and improve the reliability and stability of the system. The platform is designed to be highly adaptable and flexible, and can be customized and optimized according to different application scenarios and needs. For example, the type and number of sensors can be adjusted, the method of feature extraction and fusion can be modified, and the time series analysis model and management strategy can be optimized to meet the monitoring and management needs of different target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of a dynamic monitoring and management platform based on vision and sensor fusion provided by an embodiment of the present invention.

[0061] Figure 2 It is a flowchart of an embodiment of the present invention, in which the health management objective function is determined according to the dynamic monitoring and analysis results, and the firefly population is initialized. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attractiveness. After a preset number of iterations, the final firefly corresponding strategy is used to make early warning, evaluation and maintenance decisions. DETAILED DESCRIPTION

[0062] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0063] like Figure 1 As shown, the embodiment of the present invention proposes a dynamic monitoring and management platform based on vision and sensor fusion, including:

[0064] A visual sensor module 1, used to collect image data of a target object;

[0065] Physical sensor module 2, used to collect physical parameter data of the target object, the physical parameter data including temperature, humidity, pressure, acceleration, and angular velocity;

[0066] Data processing module 3 is used to extract features from visual and physical sensor data, create a category score matrix, fuse the classification results and weights of images and physical features, use the final category label as the fusion result, and obtain the dynamic state information of the target object based on the fusion result;

[0067] The dynamic monitoring module 4 is used to monitor and analyze the dynamic state of the target object in real time through a time series analysis model according to the fused data to obtain analysis results;

[0068] Management module 5 is used to determine the health management objective function according to the analysis results of the dynamic monitoring module, and initialize the firefly population. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions.

[0069] In the embodiment of the present invention, the visual sensor is responsible for collecting image data of the target object, providing intuitive and detailed visual information; the physical sensor is responsible for collecting physical parameter data such as temperature, humidity, pressure, acceleration, angular velocity, etc., ensuring the comprehensiveness and accuracy of monitoring. The combination of the two greatly expands the scope of monitoring and improves the accuracy of identifying the state of the target object. The data processing module processes the data collected by the visual and physical sensors through the classification model, and performs data fusion and analysis to obtain the dynamic state information of the target object. The dynamic monitoring module uses the time series analysis model to monitor and analyze the fused data in real time, which can quickly respond to the state changes of the target object and provide timely and accurate analysis results.

[0070] The management module uses the firefly algorithm to manage the health of the target object based on the analysis results of the dynamic monitoring module. The firefly algorithm simulates the flash signal communication and attraction behavior between fireflies and performs a global search in the parameter space to find a better health management strategy. This not only improves the intelligence level of health management, but also ensures the accuracy and effectiveness of early warning, evaluation and maintenance decisions.

[0071] In a preferred embodiment of the present invention, the visual sensor module 1 is used to collect image data of the target object; the physical sensor module 2 is used to collect physical parameter data of the target object, and the physical parameter data includes temperature, humidity, pressure, acceleration, angular velocity, and may include:

[0072] In the embodiment of the present invention, the layout and selection of visual sensors and physical sensors are designed. The visual sensor can be a high-definition camera, an infrared camera, etc., and is selected according to the characteristics of the target object and the monitoring environment. The physical sensor is selected and configured according to the physical parameters to be monitored (such as temperature, humidity, pressure, acceleration, angular velocity, etc.). The selected visual sensors and physical sensors are integrated into the monitoring system to ensure that they can work stably and reliably and can communicate with the data processing module.

[0073] The visual sensor begins to collect image data of the target object. This data can be a continuous video stream or a timed static image. The collected image data needs to be preprocessed, such as denoising, enhancement, correction, etc., to improve the accuracy of subsequent processing. At the same time, the physical sensor also begins to collect physical parameter data of the target object. This data is real-time numerical data and needs to be preprocessed by filtering, calibration, etc. to ensure the accuracy and stability of the data.

[0074] The collected image data and physical parameter data need to be transmitted to the data processing module through the network or other communication methods. During the transmission process, the integrity and real-time of the data need to be ensured. In order to ensure that the data collected by the visual sensor and the physical sensor are synchronized in time, timestamps or other synchronization mechanisms can be used for data alignment. After the data processing module receives the data transmitted by the visual sensor and the physical sensor, it starts data processing and analysis. For image data, image processing algorithms can be used for target recognition, tracking, classification and other operations. For physical parameter data, statistical analysis, feature extraction and other operations can be performed.

[0075] The processed visual data and physical parameter data need to be fused. Data fusion can be performed in a variety of ways, such as weighted average. Through data fusion, more comprehensive and accurate information about the target object's state can be obtained. The dynamic monitoring module monitors and analyzes the dynamic state of the target object in real time based on the fused data, and uses time series analysis for dynamic monitoring and prediction. The monitoring results can be fed back to the user through interface display, alarm prompts, etc., so that the user can understand the state of the target object in a timely manner and take corresponding measures.

[0076] In a preferred embodiment of the present invention, feature extraction is performed on visual and physical sensor data, a category score matrix is ​​created, the classification results and weights of the image and physical features are fused, the final category label is used as the fusion result, and the dynamic state information of the target object is obtained according to the fusion result, which may include:

[0077] Extract features from image data of visual sensors, including shape features, texture features, and color features; extract features from physical parameter data, including statistical features, time domain features, and frequency domain features;

[0078] The extracted image features and physical features are used as input data, and the data is classified and processed through the classification model to obtain the output result, that is, the category of the target object;

[0079] Create a category score matrix, fuse the image and physical feature classification results with the corresponding weights, and fill in the corresponding positions; calculate the fusion score of each category in the matrix, and determine the final category label as the fusion result based on the fusion score;

[0080] According to the fusion results, the dynamic state information of the target object is obtained, including the position, velocity, acceleration, posture, temperature and humidity information of the target object.

[0081] In the embodiment of the present invention, the image data of the target object is first obtained through the visual sensor, and then the image is preprocessed using the image processing algorithm, such as grayscale, binarization, edge detection, etc., and a specific feature extraction method is used, such as shape feature extraction (through contour analysis, Fourier description, etc.), texture feature extraction (through grayscale co-occurrence matrix, wavelet transform, etc.), color feature extraction (through color histogram, color space conversion, etc.), to extract the shape, texture, color and other features of the target object from the image. The physical parameter data of the target object, such as temperature, humidity, pressure, acceleration, angular velocity, etc., are collected through the physical sensor.

[0082] The collected physical parameter data is preprocessed, such as filtering, denoising, and normalization, to improve the accuracy and stability of the data. According to the characteristics of the physical parameter data, statistical features (such as mean, variance, standard deviation, etc.), time domain features (such as peak, mean absolute difference, waveform factor, etc.) and frequency domain features (such as power spectrum density, frequency center of gravity, etc.) are extracted, and the extracted image features and physical features are combined into a feature vector as the input data of the classification model. The target object is classified according to the feature vector, and the classification model is trained. The model training and parameter optimization are performed using the labeled sample data until the model reaches the predetermined accuracy. The new feature vector is classified using the trained classification model to obtain the output result, that is, the category of the target object.

[0083] Perform weighted averaging on the image feature classification results and the physical feature classification results, assign weights according to the importance of the image features and physical features in the classification, and calculate the weighted average result as the fused classification result. Determine the category of the target object based on the fused classification result. Obtain the dynamic state information of the target object, such as position, velocity, acceleration, attitude, temperature, humidity, etc., based on the category of the target object and the specific value in the feature vector.

[0084] Suppose you want to monitor machine parts on an industrial production line. Visual sensors are used to collect image data of the parts, and physical sensors are used to collect physical parameter data such as temperature and acceleration of the parts.

[0085] The visual sensor collects an image of a part and extracts the shape, texture, color and other features of the part through the feature extraction algorithm. The physical sensor collects the temperature and acceleration data of the part and extracts the statistical features, time domain features and frequency domain features through the feature extraction algorithm. The extracted image features and physical features are combined into a feature vector and input into the classification model. The classification model divides the parts into two categories: "normal" and "abnormal" according to the feature vector. Assume that the output result of the classification model is "normal", and the weight of the image feature classification result is 0.6, and the weight of the physical feature classification result is 0.4. Through weighted average processing, the fused classification result is still "normal". According to the fused classification result and the specific values ​​in the feature vector, the dynamic state information of the part can be obtained, such as the position, speed, acceleration, posture, temperature, etc. of the part.

[0086] By combining the data of visual sensors and physical sensors, more comprehensive and accurate feature information of target objects can be obtained, thereby improving the accuracy of classification models. The weighted average fusion method can assign weights according to the importance of different features in classification, further optimizing the classification results. Real-time acquisition of dynamic state information of target objects, such as position, speed, acceleration, posture, temperature, humidity, etc., helps to discover and solve problems in a timely manner. Real-time processing and analysis of data through classification models can quickly respond to changes in the state of target objects. Automatically collecting, processing and analyzing data of target objects reduces the need for manual intervention, improves production efficiency and automation level, and can automatically adjust production line parameters or take other measures based on monitoring results to achieve more intelligent production and management.

[0087] In another preferred embodiment of the present invention, the extracted image features and physical features are used as input data, and the data is classified by a classification model to obtain an output result, that is, the category of the target object, which may include:

[0088] Use random forest as a classification model, and input historical sample data, including known feature vectors and corresponding category labels, into the random forest model. By learning the mapping relationship between features and categories in the sample data, a trained random forest model is obtained.

[0089] The image features and physical features are input into the trained random forest model. According to the learned mapping relationship, the input data is processed and the classification result is output. The classification result is a category label indicating the category to which the target object belongs.

[0090] In an embodiment of the present invention, a random forest is selected as a classification model to process feature data extracted by visual sensors and physical sensors. Random forest is an integrated learning method that can improve the accuracy and robustness of classification by constructing multiple decision trees and combining their outputs. Collect historical sample data, which include known feature vectors (composed of image features and physical features) and corresponding category labels (indicating the category to which the target object belongs), and input these historical sample data into the random forest model as training data. The random forest model constructs multiple decision trees through internal algorithms (such as Bootstrap sampling and feature selection). Each decision tree is trained according to the mapping relationship between features and categories in the training data to form its own classification rules. After the training is completed, a trained random forest model is obtained, which can accurately reflect the mapping relationship between features and categories.

[0091] When new target object data is received, image features and physical features are extracted through visual sensors and physical sensors, and these features are combined into a feature vector as input data and input into the trained random forest model. Each decision tree in the random forest model processes the input data according to its own classification rules and outputs a category label as its own classification result. The classification results of all decision trees are combined through a voting mechanism (such as majority voting) to obtain the final classification result, which is the category label to which the target object belongs.

[0092] Suppose you want to monitor the types of fruit in an orchard. The visual sensor is used to collect image data of the fruit, and the physical sensor is used to collect physical parameter data such as the weight and size of the fruit.

[0093] Collect historical sample data, including known fruit image features and physical features (such as color, shape, texture, weight, size, etc.) and corresponding fruit category labels (such as apple, banana, orange, etc.), and input these historical sample data into the random forest model as training data. The random forest model obtains the trained random forest model by learning the mapping relationship between the features in the sample data and the fruit category. When new fruit data is received, the image features and physical features of the fruit are extracted through visual sensors and physical sensors, and these features are combined into a feature vector and input into the trained random forest model. The random forest model processes the input data according to the learned mapping relationship and outputs the final classification result, such as "apple".

[0094] The random forest model can effectively reduce the overfitting problem of a single decision tree and improve classification accuracy by integrating the outputs of multiple decision trees. By learning the mapping relationship between features and categories in historical sample data, the random forest model can more accurately identify the category of the target object. When the data of the target object fluctuates or changes to a certain extent, the random forest model can still maintain a high classification accuracy.

[0095] By automatically extracting features, training models, and performing classification, machines can quickly and accurately identify target objects, which helps reduce the need for human intervention and improves production efficiency and automation levels, especially in scenarios that require large-scale and rapid identification of target objects.

[0096] In another preferred embodiment of the present invention, a category score matrix is ​​created, and the classification results of the image and physical features are respectively fused with the corresponding weights and filled in the corresponding positions; the fusion score of each category in the matrix is ​​calculated, and the final category label is determined as the fusion result according to the fusion score, which may include:

[0097] Create a category score matrix, where the rows represent different category labels and the columns represent the scores of the corresponding categories in the image feature classification results and the physical feature classification results;

[0098] According to the classification results of image features, the score of the corresponding category is multiplied by the weight , and fill in the corresponding position of the score matrix; according to the physical feature classification results, multiply the score of the corresponding category by the weight , and accumulated to the corresponding position of the score matrix;

[0099] For each category, the image classifier obtains the prediction probability of the corresponding category based on the image feature vector, and multiplies the prediction probability by the weight coefficient to obtain the image feature score; the physical feature classifier obtains the prediction probability of the same category based on the physical feature vector, and multiplies the prediction probability by the weight coefficient to obtain the physical feature score;

[0100] Add the image feature score and the physical feature score to get the total score of the corresponding category in the score matrix;

[0101] According to the total score of the corresponding category in the score matrix, the final category label is used as the fusion result.

[0102] In an embodiment of the present invention, a two-dimensional array is initialized as a category score matrix. The number of rows in the matrix is ​​equal to the number of possible categories, and the number of columns is fixed to two, which are used to store the scores of image features and physical features for each category. The row labels of the matrix correspond to different category labels, such as "category A", "category B", etc.; the column labels are respectively identified as "image feature score" and "physical feature score". The predicted probability for each category is obtained from the image feature classifier. These probabilities constitute the score of the image feature vector for each category. The image feature score of each category is multiplied by the preset image feature weight coefficient , fill the weighted image feature scores into the corresponding rows and “image feature score” columns of the category score matrix.

[0103] The predicted probability of each category is obtained from the physical feature classifier. These probabilities constitute the scores of the physical feature vector for each category. The physical feature score of each category is multiplied by the preset physical feature weight coefficient. The weighted physical feature scores are added to the corresponding rows of the category score matrix and the "physical feature score" column to achieve score accumulation.

[0104] Traverse each row of the category score matrix and calculate the total score of each category. The total score is the weighted sum of the image feature score and the physical feature score, that is, ;in, The image classifier is based on the image feature vector For Category The predicted probability of is a physical feature classifier based on the physical feature vector For Category The predicted probability is compared, the total scores of all categories are compared, the category with the highest score is found, and the category label with the highest score is output as the fusion result.

[0105] Suppose that to classify an unknown object, it has two classifiers: one based on image features and another based on physical features (such as weight, texture, etc.).

[0106] Create a category score matrix containing three categories: "apple", "banana", and "orange".

[0107] The image feature classifier gives a predicted probability of 0.8 for "apple", 0.1 for "banana", and 0.1 for "orange". These scores are multiplied by the image feature weight coefficient. = 0.6 and fill in the score matrix. The physical feature classifier gives a predicted probability of 0.7 for "apple", 0.2 for "banana", and 0.1 for "orange". These scores are multiplied by the physical feature weight coefficient =0.4, and added to the corresponding position of the score matrix. Calculate the total score of each category, and get the total score of "apple" is 0.6×0.8 +0.4×0.7 = 0.76, "banana" is 0.6×0.1 + 0.4×0.2 = 0.14, and "orange" is 0.6×0.1 + 0.4×0.1 = 0.1.

[0108] Compare the total scores and determine that "apple" is the fused result.

[0109] By fusing the scores of image features and physical features, it is possible to more comprehensively consider the various aspects of the target object, thereby improving the accuracy of classification. The setting of weight coefficients allows adjustments based on the importance of different features to further optimize the classification results. When a single feature (such as an image or physical feature) is affected by interference or noise, the fusion method can use the information of another feature to compensate and improve the robustness of the classification. The fusion method can handle data sets containing multiple features and is suitable for complex and changing classification scenarios.

[0110] In a preferred embodiment of the present invention, the dynamic state of the target object is monitored and analyzed in real time through a time series analysis model based on the fused data to obtain analysis results, which may include:

[0111] Arrange the pre-fused data in chronological order to form a time series, where each time point corresponds to a dynamic state feature value of a target object;

[0112] Use historical data, including past fusion data and corresponding dynamic state labels, to train the time series analysis model to obtain a trained time series analysis model;

[0113] The real-time fusion data is input into the trained time series analysis model to conduct real-time monitoring and analysis of the dynamic state, and the dynamic state analysis results of the target object at each time point are output, including state prediction value, anomaly detection alarm, and trend analysis.

[0114] In an embodiment of the present invention, pre-fused data is obtained from a data source, and these data contain dynamic state feature values ​​of the target object at different time points. These data are arranged in chronological order according to the timestamp to form an ordered time series, and each time point corresponds to a dynamic state feature value of a target object, such as position, speed, acceleration, etc. Historical data are collected, which include past fused data and corresponding dynamic state labels. The fused data is the result of weighted averaging of image features and physical features, and the dynamic state label describes the actual state of the target object at each time point. A suitable time series analysis model is selected, such as ARIMA, LSTM, Prophet, etc., and the model is trained using historical data. During the training process, the model learns the patterns, trends and periodic changes in the time series, as well as the relationship between these changes and the dynamic state labels.

[0115] After training, the time series analysis model can accurately capture the dynamic changes in the time series and predict the future state of the target object based on these changes. When receiving new real-time fusion data, it arranges the data in chronological order and inputs it into the trained time series analysis model. The model monitors and analyzes the dynamic state of the target object in real time based on the input data and the historical patterns in the time series. The time series analysis model outputs the dynamic state analysis results of the target object at each time point, including state prediction values, anomaly detection alarms, and trend analysis. The state prediction value represents the model's prediction result of the future state of the target object; the anomaly detection alarm is used to identify abnormal points or abnormal patterns in the time series; and the trend analysis helps the machine understand the long-term trends and periodic changes in the time series.

[0116] Suppose you want to monitor and analyze an autonomous vehicle in real time to predict its driving status and detect potential anomalies.

[0117] The pre-fused data is obtained from the sensor. These data include the dynamic state feature values ​​of the car, such as acceleration, speed, and direction, and are arranged in chronological order to form a time series. The time series analysis model (such as LSTM) is trained using historical data (including past fused data and corresponding driving state labels). During the training process, the model learns the driving mode and state changes in the time series. After the training is completed, the real-time fused data is input into the trained LSTM model. The LSTM model monitors and analyzes the driving state of the car in real time based on the input data and the historical patterns in the time series. It outputs the state prediction value at each time point (such as speed and direction in the next few seconds), abnormal detection alarm (such as sudden braking or acceleration), and trend analysis (such as the trend of the driving route).

[0118] Through the time series analysis model, the dynamic state of the target object can be accurately predicted, and potential anomalies and trend changes can be discovered in time. Real-time monitoring and analysis can help to respond and make decisions quickly, and improve the overall efficiency and safety of the system. The time series analysis model can identify abnormal points and abnormal patterns in the time series and issue alarm signals in time, which can help prevent potential failures and accidents and ensure the stable operation and reliability of the system. Trend analysis helps understand the long-term trends and periodic changes in the time series, and provides a basis for future decision-making and planning. Prediction capabilities can predict the future state of the target object in advance, providing strong support for system optimization and adjustment.

[0119] In a preferred embodiment of the present invention, the health management objective function is determined according to the dynamic monitoring and analysis results, and the firefly population is initialized. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the strategy corresponding to the final firefly is used for early warning, evaluation and maintenance decision-making, which may include:

[0120] According to the analysis results of the dynamic monitoring module, the objective function of health management is determined, and the firefly population is initialized according to the objective function;

[0121] Set the flash brightness, attractiveness, and step factor parameters of the fireflies. For each firefly, calculate the objective function value of the corresponding health management strategy as the flash brightness of the corresponding firefly.

[0122] Traverse the firefly population, for each pair of fireflies, calculate the attraction between every two fireflies, and determine the direction and step length of each firefly moving towards the final firefly based on the attraction;

[0123] Update the position of the firefly, that is, update the parameters of the health management strategy to obtain a new set of health management strategy parameters;

[0124] The process of calculating the brightness of fireflies, calculating their attractiveness and updating their positions is repeated until the preset number of iterations is reached. The health management strategy corresponding to the final fireflies is used as the final result for early warning, evaluation and maintenance decision-making.

[0125] In an embodiment of the present invention, the dynamic state analysis results of the target object are obtained from the dynamic monitoring module, including state prediction values, abnormal detection alarms and trend analysis, and the objective function of health management is determined based on these analysis results. The objective function is an indicator that reflects the health state of the target object, such as performance degradation rate, failure probability, etc., or a composite function related to these indicators. A set of initial solutions are randomly generated in the parameter space, and these solutions represent different combinations of health management strategy parameters. Each solution (i.e., each health management strategy parameter combination) is represented as a firefly individual, and all firefly individuals constitute a firefly population. A series of parameters are set for fireflies, including flash brightness, attractiveness and step factor. The flash brightness is usually related to the objective function value. The better the objective function value (such as the smaller or larger, depending on the nature of the objective function), the higher the flash brightness. The attractiveness represents the mutual attraction between fireflies, which is related to the flash brightness and distance. The step factor controls the moving step of the fireflies in the parameter space.

[0126] Traverse each firefly individual in the firefly population and calculate the objective function value of its corresponding health management strategy. Convert the objective function value to a fluorescence brightness value. Traverse each pair of fireflies in the firefly population and calculate the attraction between them. The attraction is inversely proportional to the fluorescence brightness difference and the distance. The higher the fluorescence brightness and the closer the distance, the greater the attraction. According to the attraction, calculate the direction and step length of each firefly moving towards other fireflies (especially fireflies with higher fluorescence brightness). The moving direction and step length are jointly determined by the attraction and step length factor. Update the position of each firefly (i.e., the parameters of the health management strategy) according to the calculated moving direction and step length. The updated firefly positions constitute a new firefly population.

[0127] The above steps are repeated to continuously calculate the fluorescence brightness, attractiveness and update the firefly position. The iteration process continues until the preset number of iterations is reached or other termination conditions (such as fluorescence brightness convergence) are met. The firefly individual with the highest fluorescence brightness is selected as the final result. The health management strategy corresponding to the firefly individual is used for early warning, evaluation and maintenance decision-making to optimize the health management of the target object.

[0128] Assume that an industrial equipment is to be managed for health to prevent failures and extend its service life. The equipment's operating status data, including characteristic values ​​such as vibration, temperature, and pressure, are obtained from the dynamic monitoring module, and the equipment's performance degradation trend and potential failure risk are analyzed. Based on the analysis results, the objective function of health management is determined to minimize the equipment's failure probability. A set of initial solutions are randomly generated in the parameter space, which represent different combinations of maintenance strategy parameters (such as maintenance cycles, inspection items, etc.). The flash brightness, attraction, and step factor parameters of the fireflies are set, and the objective function value (i.e., failure probability) of each individual firefly is calculated. The firefly population is traversed, the attraction between each pair of fireflies is calculated, and the direction and step length of each firefly moving toward other fireflies are determined based on the attraction. The position of the firefly (i.e., maintenance strategy parameters) is updated, and the above steps are repeated until the preset number of iterations is reached. Finally, the firefly individual with the highest fluorescence brightness is selected as the final result, and the maintenance strategy corresponding to this individual is used for the health management of the equipment to prevent failures and extend its service life.

[0129] The firefly algorithm simulates the flash signal communication and attraction behavior between fireflies and performs a global search in the parameter space to find a better health management strategy, which helps to improve the health level of the target object and reduce the risk of failure and maintenance costs. The firefly algorithm can handle complex and changeable health management problems and adapt to different target objects and dynamic environments. Through multiple iterations and updates, the algorithm can gradually approach the optimal solution and improve the robustness and stability of health management. The firefly algorithm can analyze the data of the dynamic monitoring module in real time and detect potential problems and abnormal trends in time. By outputting the final health management strategy, the algorithm provides strong support for early warning, evaluation and maintenance decisions, which helps to respond and make decisions quickly.

[0130] In another preferred embodiment of the present invention, in the firefly population, each firefly represents a health management strategy, which may include:

[0131] In an embodiment of the present invention, all parameters involved in the health management strategy and their value ranges are clarified. These parameters include maintenance cycle, inspection frequency, maintenance threshold, spare parts reserve, etc., which depend on the characteristics of the target object and the health management requirements. In a determined parameter space, a set of solutions is randomly generated, each solution represents a parameter combination of a health management strategy, and these solutions are encoded as firefly individuals, each of which contains a specific set of parameter values. All generated firefly individuals are combined into a firefly population. The size of the population (i.e., the number of fireflies) can be adjusted according to the complexity of the problem and the computing resources. A mapping relationship is established between the parameter value of each firefly individual and the specific operation of the target health management strategy. For example, a parameter value of a firefly corresponds to the specific number of days in the equipment maintenance cycle. Based on the parameter combination of the firefly individuals, a specific health management strategy can be generated.

[0132] In a preferred embodiment of the present invention, the health management objective function includes:

[0133] Calculate the ratio of the number of correct warnings to the total number of warnings to obtain the accuracy of the warning;

[0134] According to the accuracy of the early warning and the preset weight coefficient, the share of the early warning accuracy in the objective function value is determined;

[0135] For each sample, calculate the square of the difference between the actual value and the predicted value of each sample to obtain the square difference corresponding to each sample, and combine the square differences of all samples to obtain the overall deviation degree between the predicted value and the actual value;

[0136] Based on the sample size and the overall degree of deviation between the predicted value and the actual value, the prediction measurement index is obtained;

[0137] According to the predicted measurement index and the preset weight coefficient, determine the contribution of the predicted part in the objective function value;

[0138] Calculate the product of the total output after maintenance and the time period before maintenance to obtain the expansion value of the output after maintenance in the time dimension;

[0139] Calculate the product of the total output before maintenance and the time period after maintenance to obtain the expansion value of the output before maintenance in the time dimension;

[0140] Obtain the difference between the value of the expanded output after maintenance and the value of the expanded output before maintenance to determine the net gain value of output in the time dimension;

[0141] Calculate the maintenance cost and the product of the pre-maintenance time period and the post-maintenance time period to obtain the total maintenance cost value of the time span;

[0142] According to the net gain value of output in the time dimension and the total maintenance cost value of the time span, the measurement index of maintenance benefit is obtained;

[0143] According to the measurement index of maintenance benefit and the preset weight coefficient, the contribution of the maintenance benefit part in the objective function value is determined;

[0144] The contributions of the warning accuracy part, the prediction part and the maintenance benefit part in the objective function value are accumulated to obtain the final objective function value.

[0145] In the embodiment of the present invention, various types of relevant data are collected from the dynamic monitoring module, including warning record data, including the number of correct warnings and total number of warnings , sample data, contains the actual value of each sample and predicted values , and record the number of samples , production output data, including total output after maintenance , Total production before maintenance , time period data, that is, the time period before maintenance and the time period after maintenance , maintenance cost data, record maintenance costs . Clean the collected data and remove duplicate, erroneous or incomplete data records. For example, check whether there are abnormal situations in the warning records where the total number of warnings is 0. If so, correct or eliminate them. Standardize the data to ensure that different types of data are consistent and comparable in subsequent calculations. For example, for production data and maintenance cost data, they can be normalized according to their value ranges so that their values ​​are between [0, 1].

[0146] The correct number of warnings Divide by the total number of warnings , the accuracy of early warning . According to the preset weight coefficient , multiply the warning accuracy by , and obtain the share of the warning accuracy in the objective function value For each sample , calculate its actual value With the predicted value The square of the difference between the two samples is obtained. . The squared differences of all samples are accumulated to obtain the overall deviation between the predicted value and the actual value. . Based on the sample size As a benchmark, the number of samples Divide by the overall degree of deviation , get the predicted metrics . According to the preset weight coefficient , multiply the predicted measure by , get the contribution of the predicted part in the objective function value .

[0147] The total output after maintenance The period before maintenance Multiply them together to get the extended value of production in the time dimension after maintenance. The total output before maintenance The time period after maintenance Multiply them together to get the expansion value of production before maintenance in the time dimension . Calculate the difference between the output expansion value after maintenance and the output expansion value before maintenance to obtain the net gain value of output in the time dimension . Maintenance costs Before maintenance period and post-maintenance period Multiply the product of to get the total maintenance cost value of the time span The net production gain is divided by the total maintenance cost to obtain the maintenance benefit measure. . According to the preset weight coefficient , multiply the maintenance benefit measure by , and obtain the contribution of the maintenance benefit part to the objective function value The contribution of the warning accuracy part, the prediction part and the maintenance benefit part in the objective function value is accumulated to obtain the final objective function value .

[0148] By comprehensively considering the early warning accuracy, prediction accuracy and the ratio of maintenance benefit to cost, the objective function can comprehensively evaluate the effectiveness of the health management strategy, thereby selecting a more targeted and effective strategy.

[0149] The firefly algorithm can perform a global search in a complex and ever-changing parameter space to find the final health management strategy, which enables the strategy to show good robustness and adaptability in different target objects and dynamic environments. By optimizing the ratio of maintenance benefits to costs, the objective function can guide the health management strategy to develop in a more economical and efficient direction, which helps to optimize resource allocation, reduce maintenance costs, and improve overall operational efficiency. Real-time calculation of the objective function value provides a strong basis for decision support. At the same time, the rapid convergence of the firefly algorithm enables the health management strategy to be quickly determined and applied, improving the response speed.

[0150] The embodiment of the present invention further provides a computing device, comprising: a processor, a memory storing a computer program, wherein when the computer program is executed by the processor, the dynamic monitoring and management platform based on vision and sensor fusion as described above is executed. All implementation methods in the above embodiment of the dynamic monitoring and management platform based on vision and sensor fusion are applicable to this embodiment, and can also achieve the same technical effect.

[0151] The embodiment of the present invention further provides a computer-readable storage medium storing instructions, which, when executed on a computer, enable the computer to execute the dynamic monitoring and management platform based on vision and sensor fusion as described above. All implementation methods in the above-mentioned dynamic monitoring and management platform based on vision and sensor fusion are applicable to this embodiment and can achieve the same technical effect.

[0152] The above is a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A dynamic monitoring and management platform based on vision and sensor fusion, characterized by: include: A visual sensor module, used to collect image data of the target object; A physical sensor module is used to collect physical parameter data of the target object, wherein the physical parameter data includes temperature, humidity, pressure, acceleration, and angular velocity; The data processing module is used to extract features from visual and physical sensor data, create a category score matrix, fuse the classification results and weights of images and physical features, use the final category label as the fusion result, and obtain the dynamic state information of the target object based on the fusion result; The dynamic monitoring module is used to monitor and analyze the dynamic state of the target object in real time based on the fused data through a time series analysis model to obtain analysis results; The management module is used to determine the health management objective function based on the analysis results of the dynamic monitoring module, and initialize the firefly population. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions.

2. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 1 is characterized in that: Extract features from visual and physical sensor data, create a category score matrix, fuse the classification results and weights of images and physical features, use the final category label as the fusion result, and obtain the dynamic state information of the target object based on the fusion result, including: Extract features from image data of visual sensors, including shape features, texture features, and color features; extract features from physical parameter data, including statistical features, time domain features, and frequency domain features; The extracted image features and physical features are used as input data, and the data is classified and processed through the classification model to obtain the output result, that is, the category of the target object; Create a category score matrix, fuse the image and physical feature classification results with the corresponding weights, and fill in the corresponding positions; calculate the fusion score of each category in the matrix, and determine the final category label as the fusion result based on the fusion score; According to the fusion results, the dynamic state information of the target object is obtained, including the position, velocity, acceleration, posture, temperature and humidity information of the target object.

3. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 2 is characterized in that: The extracted image features and physical features are used as input data, and the data is classified and processed through the classification model to obtain the output result, that is, the category of the target object, including: Use random forest as a classification model, and input historical sample data, including known feature vectors and corresponding category labels, into the random forest model. By learning the mapping relationship between features and categories in the sample data, a trained random forest model is obtained. The image features and physical features are input into the trained random forest model. According to the learned mapping relationship, the input data is processed and the classification result is output. The classification result is a category label indicating the category to which the target object belongs.

4. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 3 is characterized in that: Create a category score matrix, merge the image and physical feature classification results with the corresponding weights, and fill in the corresponding positions; Calculate the fusion score of each category in the matrix, and determine the final category label as the fusion result based on the fusion score, including: Create a category score matrix, where the rows represent different category labels and the columns represent the scores of the corresponding categories in the image feature classification results and the physical feature classification results; According to the classification results of image features, the score of the corresponding category is multiplied by the weight , and fill in the corresponding position of the score matrix; according to the physical feature classification results, multiply the score of the corresponding category by the weight , and accumulated to the corresponding position of the score matrix; For each category, the image classifier obtains the prediction probability of the corresponding category based on the image feature vector, and multiplies the prediction probability by the weight coefficient to obtain the image feature score; the physical feature classifier obtains the prediction probability of the same category based on the physical feature vector, and multiplies the prediction probability by the weight coefficient to obtain the physical feature score; Add the image feature score and the physical feature score to get the total score of the corresponding category in the score matrix; According to the total score of the corresponding category in the score matrix, the final category label is used as the fusion result.

5. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 4 is characterized in that: Based on the fused data, the dynamic state of the target object is monitored and analyzed in real time through the time series analysis model to obtain analysis results, including: Arrange the pre-fused data in chronological order to form a time series, where each time point corresponds to a dynamic state feature value of a target object; Use historical data, including past fusion data and corresponding dynamic state labels, to train the time series analysis model to obtain a trained time series analysis model; The real-time fusion data is input into the trained time series analysis model to conduct real-time monitoring and analysis of the dynamic state, and the dynamic state analysis results of the target object at each time point are output, including state prediction value, anomaly detection alarm, and trend analysis.

6. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 5 is characterized in that: According to the dynamic monitoring and analysis results, the health management objective function is determined, and the firefly population is initialized. The objective function value is used as the firefly flash brightness, and the strategy parameters are updated by calculating the attraction. After a preset number of iterations, the final firefly corresponding strategy is used for early warning, evaluation and maintenance decisions, including: According to the analysis results of the dynamic monitoring module, the objective function of health management is determined, and the firefly population is initialized according to the objective function; Set the flash brightness, attractiveness, and step factor parameters of the fireflies. For each firefly, calculate the objective function value of the corresponding health management strategy as the flash brightness of the corresponding firefly. Traverse the firefly population, for each pair of fireflies, calculate the attraction between every two fireflies, and determine the direction and step length of each firefly moving towards the final firefly based on the attraction; Update the position of the firefly, that is, update the parameters of the health management strategy to obtain a new set of health management strategy parameters; The process of calculating the brightness of fireflies, calculating their attractiveness and updating their positions is repeated until the preset number of iterations is reached. The health management strategy corresponding to the final fireflies is used as the final result for early warning, evaluation and maintenance decision-making.

7. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 6 is characterized in that: In the firefly population, each firefly represents a health management strategy.

8. The dynamic monitoring and management platform based on vision and sensor fusion according to claim 7 is characterized in that: The health management objective functions include: Calculate the ratio of the number of correct warnings to the total number of warnings to obtain the accuracy of the warning; According to the accuracy of the early warning and the preset weight coefficient, the share of the early warning accuracy in the objective function value is determined; For each sample, calculate the square of the difference between the actual value and the predicted value of each sample to obtain the square difference corresponding to each sample, and combine the square differences of all samples to obtain the overall deviation degree between the predicted value and the actual value; Based on the sample size and the overall degree of deviation between the predicted value and the actual value, the prediction measurement index is obtained; According to the predicted measurement index and the preset weight coefficient, determine the contribution of the predicted part in the objective function value; Calculate the product of the total output after maintenance and the time period before maintenance to obtain the expansion value of the output after maintenance in the time dimension; Calculate the product of the total output before maintenance and the time period after maintenance to obtain the expansion value of the output before maintenance in the time dimension; Obtain the difference between the value of the expanded output after maintenance and the value of the expanded output before maintenance to determine the net gain value of output in the time dimension; Calculate the maintenance cost and the product of the pre-maintenance time period and the post-maintenance time period to obtain the total maintenance cost value of the time span; According to the net gain value of output in the time dimension and the total maintenance cost value of the time span, the measurement index of maintenance benefit is obtained; According to the measurement index of maintenance benefit and the preset weight coefficient, the contribution of the maintenance benefit part in the objective function value is determined; The contributions of the warning accuracy part, the prediction part and the maintenance benefit part in the objective function value are accumulated to obtain the final objective function value.

9. A computing device, characterized in that include: one or more processors; A storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic monitoring and management platform based on vision and sensor fusion as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, which, when executed by a processor, implements a dynamic monitoring and management platform based on vision and sensor fusion as described in any one of claims 1 to 8.

Images