Perception collaborative decision-making method and system based on multi-modal heterogeneous data fusion

Abstract

The invention provides a perception collaborative decision-making method and system based on multi-modal heterogeneous data fusion, and relates to the technical field of artificial intelligence, and the method comprises the steps: inputting a global environment situation perception graph into a pre-trained multi-target collaborative decision-making model; the multi-target collaborative decision-making model forms a multi-target decision-making feature set by analyzing the resource entities and the incidence relation in the graph; based on the multi-target decision feature set, decision optimization is carried out to obtain a comprehensive collaborative scheduling scheme; performing instruction analysis and packaging on the comprehensive collaborative scheduling scheme to obtain an executable instruction sequence; and issuing the executable instruction sequence to a corresponding decision node and a control terminal in parallel through a distributed communication architecture to complete real-time scheduling of resources and collaborative issuing of control instructions. According to the invention, by constructing a linkage mechanism of multi-modal data fusion, dynamic environment perception and collaborative decision execution, intelligent perception and quick response to a complex environment are realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a perception collaborative decision-making method and system based on multi-modal heterogeneous data fusion. BACKGROUND

[0002] With the evolution of artificial intelligence (AI) technology from single-modal processing to multi-modal fusion and collaborative decision-making, intelligent systems are increasingly applied in the fields of government services, AI breeding, smart home and digital agriculture. These systems need to process images, texts, time series data and graph structure data from multiple sensing devices at the same time, forming a typical multi-modal heterogeneous data environment. However, the related technology has significant defects in data processing, which seriously restricts the system performance.

[0003] Firstly, the data island problem caused by modal heterogeneity is prominent. The spatial features of images, the semantic information of texts, the dynamic rules of time series data and the topological relationships of graph structure data belong to different feature spaces, and there is a lack of unified representation model. Related technologies mostly use shallow fusion strategies such as independent processing of each modal data and then simple splicing or weighted averaging, which is difficult to mine the deep semantic association between modalities, may lead to insufficient discriminability of fused features, and cannot accurately describe the complex system state.

[0004] Secondly, the data processing pipeline is long and real-time performance is difficult to guarantee. Traditional batch processing mode cannot adapt to continuous high-speed data flow. There is a delay from data acquisition to decision output, which makes the system make decisions based on outdated information and performs poorly in application scenarios that require fast response. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a perception collaborative decision-making method and system based on multi-modal heterogeneous data fusion, which realizes intelligent perception and rapid response to complex environment by constructing a linkage mechanism of multi-modal heterogeneous data fusion, dynamic environment perception and collaborative decision-making execution.

[0006] To solve the above technical problems, the technical solutions of the present application are as follows: In a first aspect, a perception collaborative decision-making method based on multi-modal heterogeneous data fusion, the method comprising: Collecting multi-modal heterogeneous data, the multi-modal heterogeneous data including images, texts, time series data and graph structure data; preprocessing the multi-modal heterogeneous data to obtain standardized data flow; The standardized data stream is parsed and feature extraction is performed, alignment and coding are performed, and a multi-source feature vector is obtained; the multi-source feature vector is weighted and fused to obtain a preliminary index set; based on the preliminary index set, a predefined key area is grid processed, and the dynamic correlation characteristics of each spatial grid unit are analyzed to obtain a spatial dynamic weight matrix; the spatial dynamic weight matrix and the preliminary index set are weighted and fused, and through spatial context calibration, an optimized comprehensive state index is obtained; based on the optimized comprehensive state index, a global environment situation awareness map is generated; The global environment situation awareness map is input into a pre-trained multi-target collaborative decision-making model; the multi-target collaborative decision-making model forms a multi-target decision-making feature set by analyzing the resource entities and associated relationships in the global environment situation awareness map; based on the multi-target decision-making feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme; The comprehensive collaborative scheduling scheme is parsed and encapsulated to obtain an executable instruction sequence; the executable instruction sequence is parallelly issued to the corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative publishing of control instructions.

[0007] In a second aspect, a perception collaborative decision-making system based on multi-modal heterogeneous data fusion includes: A multi-source acquisition module is configured to acquire multi-modal heterogeneous data, and the multi-modal heterogeneous data includes images, texts, time series data, and graph structure data; the multi-modal heterogeneous data is preprocessed to obtain standardized data streams; A fusion analysis module is configured to parse and extract features from the standardized data streams, perform alignment and coding, and obtain a multi-source feature vector; the multi-source feature vector is weighted and fused to obtain a preliminary index set; based on the preliminary index set, a predefined key area is grid processed, and the dynamic correlation characteristics of each spatial grid unit are analyzed to obtain a spatial dynamic weight matrix; the spatial dynamic weight matrix and the preliminary index set are weighted and fused, and through spatial context calibration, an optimized comprehensive state index is obtained; based on the optimized comprehensive state index, a global environment situation awareness map is generated; A collaborative decision-making module is configured to input the global environment situation awareness map into a pre-trained multi-target collaborative decision-making model; the multi-target collaborative decision-making model forms a multi-target decision-making feature set by analyzing the resource entities and associated relationships in the global environment situation awareness map; based on the multi-target decision-making feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme; An execution module is configured to parse and encapsulate the comprehensive collaborative scheduling scheme to obtain an executable instruction sequence; the executable instruction sequence is parallelly issued to the corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative publishing of control instructions.

[0008] In a third aspect, a computing device includes: one or more processors; a storage device storing one or more programs, when executed by the one or more processors, cause the one or more processors to implement the method.

[0009] In a fourth aspect, a computer-readable storage medium stores a program, when executed by a processor, implements the method.

[0010] The above scheme of the present application at least includes the following beneficial effects: Firstly, by collecting multi-modal heterogeneous data and preprocessing, the heterogeneous barriers of multi-modal data are broken, and by integrating scattered images, texts, time series data and graph structure data into standardized data streams in a unified format, the differences in data formats, redundant information and abnormal interference can be eliminated, and the consistency and reliability of input data can be strengthened. Then, by analyzing and extracting the standardized data streams, aligning and encoding and weighted fusion, the deep association between multi-modal data can be mined, and a multi-source feature vector with completeness and discriminability can be generated. Through grid processing and dynamic association characteristic analysis, a spatial dynamic weight matrix is constructed to realize the precise binding of features and spatial positions. Combined with spatial context calibration, feature optimization is completed to form a structured comprehensive state index and a global environment situation awareness graph, which can strengthen the expression of spatial association features and improve the spatial perception and feature learning ability of complex environments. Next, the global environment situation awareness graph is input into the pre-training model to analyze the resource entities and associated relationships to form a multi-target decision feature set, which can realize the conversion of visual spatial information to structured decision features. Through a multi-objective collaborative optimization algorithm, decision deduction is completed to construct a multi-objective coordinated decision-making framework. It promotes the evolution from single-target decision-making to multi-target collaborative decision-making, realizes efficient collaborative reasoning, and improves the systematicness of artificial intelligence in processing complex decision-making tasks. Finally, by analyzing and packaging the collaborative scheduling scheme, an abstract decision to concrete execution instruction conversion bridge is constructed to ensure the executability of the decision output. Through the distributed communication architecture, the instructions are issued in parallel to play the advantages of distributed processing and improve the execution efficiency. Combined with the state feedback and dynamic adjustment mechanism, a closed-loop system of decision, execution and feedback is constructed to strengthen the adaptive ability and dynamic self-adaptation ability, so that the artificial intelligence decision can quickly respond to the real-time changes of the actual scene. BRIEF DESCRIPTION OF DRAWINGS

[0011] Figure 1 A flowchart of a perception collaborative decision-making method based on multi-modal heterogeneous data fusion is provided for the embodiments of the present application.

[0012] Figure 2A schematic diagram of a perception collaborative decision system based on multi-modal heterogeneous data fusion is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0013] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms without being limited by the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure can be more thoroughly understood, and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0014] An embodiment of the present application proposes a perception collaborative decision method based on multi-modal heterogeneous data fusion, as shown in Figure 1 A flowchart of a perception collaborative decision method based on multi-modal heterogeneous data fusion is provided for an embodiment of the present application, which includes the following steps: Step 100, multi-modal heterogeneous data is collected, including images, texts, time series data, and graph structure data; the multi-modal heterogeneous data is preprocessed to obtain a standardized data stream.

[0015] Step 200, the standardized data stream is parsed and feature extracted, and is aligned and encoded to obtain a multi-source feature vector; the multi-source feature vector is weighted fused to obtain a preliminary index set; based on the preliminary index set, a predefined key area is grid processed, and dynamic correlation characteristics of each spatial grid unit are analyzed to obtain a spatial dynamic weight matrix; the spatial dynamic weight matrix and the preliminary index set are weighted fused, and are calibrated through a spatial context to obtain an optimized comprehensive state index; based on the optimized comprehensive state index, a global environment situation awareness graph is generated.

[0016] Step 300, the global environment situation awareness graph is input to a pre-trained multi-target collaborative decision model; the multi-target collaborative decision model forms a multi-target decision feature set by analyzing resource entities and correlation relationships in the graph; based on the multi-target decision feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme.

[0017] Step 400, the comprehensive collaborative scheduling scheme is parsed and encapsulated for instructions to obtain an executable instruction sequence; the executable instruction sequence is parallelly issued to corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative publishing of control instructions.

[0018] In the embodiments of the present application, data islands are broken by collecting multi-modal heterogeneous data; the data is pre-processed to form a standardized data stream, which can unify the data format standard and improve data adaptability; the standardized data stream is analyzed to extract features and complete alignment coding, which strengthens the expression ability of multi-source features; the preliminary index set is obtained by weighted fusion, combined with key area gridding processing and spatial dynamic correlation characteristic analysis, a spatial dynamic weight matrix is constructed, which can improve the pertinence and rationality of feature fusion; the comprehensive state index is optimized by spatial context calibration, the global environment situation awareness map is generated, the structured situation representation is constructed, which is helpful for deep understanding of complex environment; the global environment situation awareness map is input into the pre-training model, the resource entity and the correlation relationship are analyzed to enrich the decision feature dimension; the multi-agent collaborative algorithm is used to optimize the decision, efficient collaborative reasoning is realized, which can improve the integrity and collaboration of the decision; the executable instruction sequence is obtained by analyzing and packaging the collaborative scheduling scheme, the standardized execution instruction is used to ensure the accuracy of the landing of artificial intelligence decision; the distributed communication architecture is used to issue instructions in parallel, which improves the efficiency and real-time performance of artificial intelligence decision execution, and strengthens the closed-loop ability from decision to execution.

[0019] In some embodiments, the step 100 of collecting multi-modal heterogeneous data includes images, texts, time series data, and graph structure data; the multi-modal heterogeneous data is pre-processed to obtain a standardized data stream, including: a1-a5.

[0020] a1, the multi-modal heterogeneous data from multiple heterogeneous data sources is collected in parallel, the data sources include at least one of the following: monitoring images, sensor nodes, mobile units, and environmental information data sources.

[0021] In some embodiments, the multi-modal heterogeneous data includes images, texts, time series data, and graph structure data.

[0022] In some embodiments, a1 can be specifically implemented as follows: deploying distributed data collection nodes, capturing scene image data in real time through high-definition cameras for monitoring image data sources, synchronously detecting the posture parameters of the captured images using a posture calibration algorithm, correcting the pitch, yaw and roll deviations of the images according to a preset calibration model, and ensuring the consistency of the image space posture; collecting time series sensing data such as temperature, humidity and air pressure for the sensing nodes, collecting dynamic data such as position trajectory and state parameters for the mobile units, and collecting text data such as weather reports and geographical descriptions and graph structure data such as regional associated topology for the environment information data sources, fusing a k-neighborhood spatial index algorithm during the collection process, constructing a k-neighborhood index structure based on spatial position, quickly matching the adjacent associated data of each heterogeneous data source in the spatial dimension, and realizing the synchronous associated collection of spatial related data; using a multi-thread concurrent collection mechanism to synchronously receive multi-modal data from each heterogeneous data source, optimizing the collection scheduling through spatial indexing, ensuring the image data quality through posture calibration, and ensuring the comprehensiveness, parallelism and spatial consistency of data collection.

[0023] a2. Format analysis and data cleaning of multi-modal heterogeneous data to remove invalid and redundant data and obtain structured data.

[0024] In some embodiments, a2 can be specifically implemented as follows: for the collected multi-modal heterogeneous data, using format analysis tools adapted to different data types to convert image data into a unified pixel matrix format, convert text data into a standard character encoding format, arrange time series data into a timestamp associated numerical sequence format, and convert graph structure data into a standardized storage format associated with nodes and edges; removing redundant information such as duplicate records and null data through rule matching screening, eliminating invalid data with format errors or beyond the reasonable range through data validity checking, and regularizing the processed data of various types into structured data with a unified structure.

[0025] a3. Abnormal value detection and missing data repair processing of structured data to obtain normalized data.

[0026] In some embodiments, the above a3 can be specifically implemented as follows: first, using statistical analysis method combined with domain threshold standard, outlier identification is performed on numerical time series data in structured data, node edge association anomaly detection is performed on graph structure data, and semantic contradiction information in text data is determined; then, the detected abnormal values are processed using reasonable replacement or rejection strategy, and for missing values in the data set, according to the data type classification processing, for non-spatially related data, the mean filling method or the similarity-based association filling method is selected to repair, and for numerical time series data containing spatial position information (such as sensor data distributed in different spatial nodes), a spatial geometric interpolation algorithm is used to repair, and the specific calculation process is as follows: first, extract the spatial coordinate information of the missing data point, select the effective adjacent data points within the preset range of the coordinate, determine the spatial distance and geometric position relationship of each adjacent data point and the missing point, assign weight coefficients according to the inverse proportion of the spatial distance, the adjacent data points closer to the missing point have higher weight proportion, calculate the initial interpolation result of the missing point by weighted summation, and then fine-tune the interpolation value combined with the time series change rule of the data, to ensure that the repaired data not only conforms to the spatial distribution characteristics but also maintains the time series continuity; finally, the normalized data with unified data format, complete content and consistent space-time association is obtained.

[0027] a4, time series alignment and spatial position matching are performed on the normalized data, a unified space-time reference is established, and space-time aligned data is obtained.

[0028] In some embodiments, a4 can be specifically implemented as follows: extracting timestamp information in each protocol data, taking the high-precision time synchronization protocol as a reference, adjusting the timing data of different data sources to the same time granularity, and realizing the alignment of each modal data in the time dimension; for image data containing spatial information, mobile unit position data, and graph structure topology data, first, a unified geographic coordinate system is used for preliminary spatial position calibration, and each data is associated with the corresponding spatial coordinate point or area, and then the iterative closest point algorithm is used to optimize the spatial alignment accuracy, and the specific calculation process is as follows: selecting spatial features clear and evenly distributed data sources as reference data sets, and other data sets to be calibrated as target data sets, extracting key spatial feature points in the two data sets, including image pixel coordinate feature points, mobile unit positioning coordinate points, and graph structure topology node coordinate points; initializing the spatial transformation matrix, including translation, rotation, and scaling parameters; iteratively finding the nearest neighbor point pairs of the target data set feature points and the reference data set feature points, calculating the spatial distance error of each pair of corresponding points, and constructing an error function; solving the optimal transformation matrix that minimizes the error function based on the least squares method, updating the spatial coordinates of the target data set; repeating the nearest neighbor point pair matching, error calculation, and transformation matrix optimization steps until the error change of the adjacent two iterations is less than the preset threshold, and stopping iteration; correcting the position deviation of spatial data from different data sources through the iterative closest point algorithm, establishing more accurate spatial correlation mapping between data, and forming spatio-temporal alignment data with unified time reference and high-precision spatial reference.

[0029] a5, normalizing the spatio-temporal alignment data to eliminate dimensional differences to obtain standardized data flow.

[0030] In some embodiments, a5 can be specifically implemented as follows: for numerical data in the spatio-temporal alignment data (including timing perception data, position coordinate data, etc.), using the min-max normalization or z-score standardization method, mapping data of different dimensions and different numerical ranges to a unified numerical interval, eliminating the problem of data incomparability caused by dimensional differences, while maintaining the original distribution characteristics and correlation rules of the data, and finally forming a standardized data flow with unified format, consistent space-time, and unified dimension, providing a strong data basis for subsequent feature extraction and fusion.

[0031] In the embodiments of the present application, by collecting multi-modal data of multiple heterogeneous data sources in parallel, data islands can be broken, and data input dimensions can be enriched; the data coverage range can be widened to provide a multi-element data basis for multi-source information fusion; by performing format analysis and cleaning on multi-modal heterogeneous data, invalid and redundant information can be removed, and data form can be regularized; data noise interference can be reduced, and data quality can be improved; by conducting outlier detection and missing data repair, data defects can be made up, data integrity and consistency can be guaranteed, and data reliability can be strengthened; by performing time series alignment and spatial position matching, a unified time-space reference can be established; the time-space misplacement problem of multi-source data can be eliminated, and the time-space correlation of data can be given; finally, by performing normalization processing on the time-space aligned data, dimension differences can be eliminated; data scale standards can be unified, and data comparability can be improved.

[0032] In some embodiments, the step 200 described above, the standardized data stream is analyzed and feature extraction is performed, and alignment and encoding are performed to obtain a multi-source feature vector; the multi-source feature vector is weighted and fused to obtain a preliminary index set; based on the preliminary index set, a predefined key area is grid processed, and dynamic correlation characteristics of each spatial grid unit are analyzed to obtain a spatial dynamic weight matrix; the spatial dynamic weight matrix and the preliminary index set are weighted and fused, and spatial context calibration is performed to obtain an optimized comprehensive state index; based on the optimized comprehensive state index, a global environment situation awareness map is generated, including: b1-b16.

[0033] b1, performing time dimension analysis on the standardized data stream, and dividing the continuous standardized data stream into an equal-interval time slice sequence, specifically comprising: first, analyzing the collection frequency of the standardized data stream and the application scenario decision delay requirement, wherein the collection frequency is, for example, 10 Hz, and the application scenario decision delay requirement is, for example, 500 ms; determining a time interval parameter of 100 ms by calculating the matching degree of data transmission bandwidth and processing algorithm; adopting a sliding window division method with a step length of 50 ms, and simultaneously optimizing window division by fusing a sliding window geometric constraint algorithm; first extracting the spatial coordinate boundaries of each modality data in the standardized data stream, determining the total number of preset spatial grids in the key region, such as 10,000, and setting a window spatial coverage completeness threshold of 90%, that is, each sliding window needs to cover no less than 90% of the preset spatial grid; for each potential time slice, calculating the number of effective spatial grids covered based on the spatial coordinates of each modality data, and obtaining the spatial coverage completeness through the ratio of the number of effective grids to the total number of grids; if the ratio is less than 90%, dynamically adjusting the window start time with a step length of 1 ms, and the adjustment range is not more than 20 ms before and after, recalculating the spatial coverage completeness of the adjusted window, until the threshold requirement is met; the window meeting the geometric constraint is taken as the final time slice, each slice contains complete multi-modal heterogeneous data in the 100 ms time period, each slice is subjected to timestamp continuity verification, the error allowable range is ±1 ms, the data segment with repeated or missing timestamp is removed, the orderliness, integrity and spatial coverage consistency of the time series data are ensured, and a regular, non-data redundant and time-space adaptive time unit is provided for subsequent time period processing.

[0034] b2, based on the equal-interval time slice sequence, performing spatial registration processing on the multi-modal data in each time slice to establish spatial reference data under a unified coordinate system, the spatial reference data including mobile object trajectory data, fixed sensor data and mobile sensor data, specifically including: based on the equal-interval time slice sequence, selecting a World Geodetic System 1984 (WGS-84) geographic coordinate system as a unified spatial reference, and optimizing spatial registration accuracy by a projection transformation algorithm; first, identifying the original projection type of each modal data through data source metadata analysis or pre-set configuration, including but not limited to Universal Transverse Mercator (UTM) projection, Gauss-Kruger projection, Mercator projection, extracting the key parameters of the corresponding original projection, including central meridian longitude, reference ellipsoid parameters, projection scale factor, offset; the specific calculation process is that, for the original projection coordinate data, the geodetic coordinates are inversely deduced based on the projection forward solution model, the geodetic coordinates (longitude and latitude) under the original reference surface are converted into the geodetic coordinates under the WGS-84 reference surface through ellipsoid parameter conversion, and the projection deformation error is compensated through an error correction model to ensure that the coordinate error after projection transformation is less than or equal to 0.3m; the spatial registration processing is performed on the multi-modal data in each time slice, the mobile object trajectory data is first converted into WGS-84 longitude and latitude coordinates through projection transformation, and then the GPS positioning deviation is corrected through a pre-set coordinate conversion matrix, the correction accuracy is ±0.5m, the original installation position projection coordinates of the fixed sensor data are converted into WGS-84 longitude and latitude coordinates through the above process, accurate to six decimal places, and then the sensor data are one by one bound and stored in a geographic JSON (GeoJSON) format, the original projection coordinates output by the positioning module are first subjected to projection transformation, and then the WGS-84 coordinates output by the Beidou positioning module are synchronized in real time, and the coordinate association relationship is updated once every 10ms; finally, the spatial reference data under the unified coordinate system including the mobile object trajectory data, the fixed sensor data and the mobile sensor data are integrated.

[0035] b3, determine the object distribution density in each grid cell based on the mobile object trajectory data in the spatial reference data, obtain the spatio-temporal feature matrix; based on the fixed sensor data, by statistical analysis of the correlation between event interval and event occurrence number per unit time, construct the flow feature vector; use the mobile sensor data to remove outliers and calculate the mean value of the dynamic parameter sampling sequence, obtain the regional average dynamic feature, specifically including: based on the spatial reference data, first set the grid size to 10m x 10m according to the application scene accuracy requirement, determine the grid division rule according to the key area range, determine the latitude and longitude coordinates of the grid cell boundary, quickly locate the grid cell to which the mobile object trajectory data belongs through R-tree spatial index, count the real-time number of mobile objects in each grid cell, calculate the object distribution density (unit: person / m 2 ), construct the spatio-temporal feature matrix (dimension: grid number x time slice number) containing time slice identifier, grid number and corresponding distribution density, based on the fixed sensor data, use 1s sliding time window to extract the event trigger record of each sensor, count the event occurrence interval and event occurrence number per unit time (number of events in the window / window duration) in the window, analyze the positive correlation or negative correlation between the two, construct the flow feature vector reflecting the event occurrence frequency and regularity, dimension: sensor number, event frequency and interval average, use the mobile sensor data to divide the dynamic parameter sampling sequence according to the 100m x 100m spatial area, use the 3σ criterion to remove outliers in the sequence, calculate the arithmetic mean of the effective dynamic parameters in each region, obtain the regional average dynamic feature reflecting the overall dynamic state of the region.

[0036] b4, align the feature dimensions of the spatio-temporal feature matrix, the flow feature vector and the regional average dynamic feature, and obtain the multi-source feature vector by vectorization splicing, specifically including: first analyze the dimension structure and data type of the spatio-temporal feature matrix, the flow feature vector and the regional average dynamic feature (convert to 32-bit floating point numbers (float32) uniformly), among which the dimension of the spatio-temporal feature matrix is grid number x time slice number, the dimension of the flow feature vector is sensor number x 2, and the dimension of the regional average dynamic feature is region number x 1, perform flattening processing on the spatio-temporal feature matrix to convert it into a one-dimensional vector, the dimension is grid number x time slice number, expand the spatial dimension of the flow feature vector according to the grid to which the sensor belongs, the dimension is grid number x time slice number x 2, expand the spatial dimension of the regional average dynamic feature according to the grid contained in the region, the dimension is grid number x time slice number x 1, ensure that the dimensions of the three are completely consistent and the data types are matched, complete the feature dimension alignment, then according to the time slice order, vectorize and splice the three types of aligned features along the feature dimension, form the multi-source feature vector corresponding to each time slice, the dimension is grid number x time slice number x 4, realize the effective integration of multi-type feature information.

[0037] b5, based on the multi-source feature vector, determining an attention weight matrix of each feature vector through a preset fully connected layer and a normalized index (softmax) function, specifically including: presetting a fully connected layer structure based on multi-source feature vector dimensions, feature conversion requirements and gradient stability targets, the fully connected layer containing an input layer, two hidden layers and an output layer, the number of input layer neurons being consistent with the multi-source feature vector dimensions, wherein the multi-source feature vector dimensions are grid number x time slice number x 4, the number of neurons in the first layer of hidden layer is set to 1 / 2 of the input layer, a rectified linear unit (ReLU) activation function is selected to realize feature nonlinear transformation, the number of neurons in the second layer of hidden layer is set to 1 / 4 of the input layer, a leaky rectified linear unit (LeakyReLU) activation function is selected to avoid gradient disappearance, the number of neurons in the output layer is the same as that in the input layer, and an L2 regularization mechanism is preset to suppress overfitting; after the multi-source feature vector is reshaped into a one-dimensional vector, it is input into the fully connected layer, and after being transmitted by the input layer, it is sequentially subjected to feature conversion by the two layers of hidden layer, and the output layer outputs the original scores corresponding to each feature and performs preset L2 regularization processing; the original scores after regularization are input into the softmax function, the function is used to normalize the original scores of each feature, map the scores to the interval of 0 to 1, and the sum of all feature scores is 1, the generated value is the attention weight corresponding to each feature, which is used to quantify the contribution degree of different features to the overall feature expression; finally, the attention weight matrix corresponding to each feature vector is obtained, the dimension is consistent with the multi-source feature vector, and the weight proportion of the key feature is ensured to be not less than 30% through gradient descent optimization, which directly reflects the importance of each feature.

[0038] b6, the attention weight matrix is weighted and summed with the multi-source feature vector to obtain an attention weighted fusion feature, specifically including: first, the dimension consistency of the attention weight matrix and the multi-source feature vector is confirmed one by one, both are three-dimensional structures of grid number x time slice number x 4, ensuring that each weight value and the corresponding feature element are one-to-one corresponding in grid number, time slice number and feature dimension; for each weight value in the attention weight matrix and the feature element in the multi-source feature vector corresponding to the position, element-by-element multiplication operation is performed, and six significant figures after the decimal point are retained during the operation process to obtain a three-dimensional structure of weighted feature component set, the dimension of which remains consistent with the weight matrix and the multi-source feature vector; each grid and each feature dimension are traversed in turn according to the time slice dimension, and the weighted feature components corresponding to all time slices under the same grid and the same feature dimension are accumulated and summed to obtain the time aggregation value of each grid and each feature dimension; then, all time aggregation values of each grid are integrated according to the feature dimension to form a four-dimensional feature vector corresponding to each grid; through the above two summation operations, the weighted contributions of each feature in different time and space dimensions are aggregated, and finally the attention weighted fusion feature with the dimension of grid number x 4 is generated; through the differential allocation of weight values, the weighted components of key features occupy a dominant position after summation, thereby highlighting the supporting role of key features in the overall feature expression, while weakening the influence of the weighted components corresponding to secondary features.

[0039] b7, the attention weighted fusion feature is subjected to feature dimension reduction and standardization processing to obtain a preliminary index set, specifically including: first, the attention weighted fusion feature with the dimension of grid number x 4 is subjected to data regularization, and is arranged into a two-dimensional data matrix with 4 feature dimensions as columns and feature data corresponding to each grid as rows; based on the two-dimensional data matrix, the covariance between each feature dimension is calculated, a 4x4 covariance matrix is constructed, the covariance calculation takes the feature data of all grids as a complete sample set, and the covariance value of all sample data under any two feature dimensions is solved one by one to ensure that each element in the covariance matrix corresponds to the linear correlation degree of two feature dimensions; the constructed covariance matrix is subjected to eigenvalue decomposition, and 4 eigenvalues and the feature vectors corresponding to each eigenvalue are obtained, the 4 eigenvalues are sequentially sorted in descending order of value, the variance contribution rate of each eigenvalue to the total sum of all eigenvalues is calculated one by one, and the cumulative variance contribution rate is obtained by accumulating the variance contribution rates in the sorted order; when the cumulative variance contribution rate first reaches the preset screening condition of greater than or equal to 95%, the accumulation operation is stopped, and all feature vectors corresponding to the eigenvalues participating in the accumulation at this time are recorded, these feature vectors are combined into a projection matrix according to the sorted order of the corresponding eigenvalues, and the number of columns of the projection matrix is the number of principal components retained, that is, the number of features after dimension reduction; the original two-dimensional data matrix is subjected to matrix multiplication operation with the projection matrix to obtain a feature matrix after dimension reduction, the number of rows of which remains the grid number, and the number of columns is the number of retained principal components.

[0040] Subsequently, z-score standardization is performed on the dimension-reduced feature matrix. First, the feature mean and sample standard deviation of all elements in the dimension-reduced feature matrix are calculated. The feature mean is the arithmetic mean of all elements, and the sample standard deviation is the unbiased standard deviation calculated based on all elements, with six significant digits after the decimal point. For each element in the dimension-reduced feature matrix, the difference between the original value and the feature mean is calculated, and then the difference is divided by the sample standard deviation to complete the standardization conversion of a single element. Repeat the conversion process until all elements in the dimension-reduced feature matrix are processed. Map the standardized feature matrix to a unified numerical interval to completely eliminate the dimensional differences and numerical range differences between different feature dimensions. Finally, a preliminary index set with dimensions of grid number x dimension-reduced feature number is obtained.

[0041] b8, based on the preliminary index set, and in combination with the pre-defined key area boundary, a regular grid topology structure covering the area is constructed, specifically including: in combination with the functional requirements of the specific application scenario, the actual management range, and the effective coverage range of the heterogeneous data source, the key area boundary is pre-defined through a preset manner, it is determined that the boundary is composed of a set of continuous polygon vertex latitude and longitude coordinates, forming a standardized boundary definition file; read the boundary definition file through the GIS interface to obtain the complete boundary latitude and longitude coordinate information of the pre-defined key area, including the longitude and latitude values of the boundary vertices; calculate the actual area of the key area according to the obtained boundary coordinates, such as 10km 2 , in combination with the accuracy requirement of the scene for data processing, such as 1m, determine the grid size parameter as 1m x 1m; the latitude difference between the northernmost vertex and the southernmost vertex of the boundary is converted to obtain the north-south span of the area (unit: m), and the north-south span value is taken as the number of rows of the grid; the longitude difference between the easternmost vertex and the westernmost vertex of the boundary is converted to obtain the east-west span of the area (unit: m), and the east-west span value is taken as the number of columns of the grid, to ensure that the grid structure can completely cover the key area without spatial omission; adopt a rectangular grid division rule to construct a regular grid topology structure according to the set number of rows, number of columns, and size parameter; start from the top-left corner grid in row-major order to assign a unique number to each grid, with the number format being row number, column number; based on the number of rows, number of columns, and size parameter of each grid, in combination with the starting latitude and longitude coordinates of the key area boundary, the latitude and longitude coordinates of the four corners of each grid are calculated one by one to establish a one-to-one mapping relationship between the grid unique number and the grid four-corner latitude and longitude coordinates; construct an R-tree index structure, store the grid number and corresponding coordinate mapping relationship into the index, realize the function of quickly querying the corresponding coordinates through the grid number or quickly positioning the grid number through the coordinates, and finally form a clear spatial grid system to provide a structured carrier for spatial feature mapping.

[0042] b9, based on the rule grid topology, mapping the preliminary index set to each spatial grid cell, calculating the statistical distribution characteristics of the environmental state indicators in each grid cell, specifically including: first extracting the latitude and longitude coordinates associated with each index data in the preliminary index set, comparing the pre-set four corner latitude and longitude coordinate ranges of each spatial grid cell, and determining whether the coordinates of the index data fall between the upper and lower limits of the longitude and latitude of the target grid cell, completing the preliminary matching of the index data and the spatial grid cell to which it belongs; checking the matching results through the mapping relationship between the grid number and the spatial coordinates, eliminating invalid index data whose coordinates exceed the range of all grids, and ensuring that each valid index data corresponds to a spatial grid cell; grouping each environmental state indicator data in each grid cell according to the unique identifier of the time slice, wherein the unique identifier of the time slice is, for example, the time slice number, so that the index data of the same grid cell and the same time slice are grouped together, forming a three-dimensional grouping structure of grid cell, time slice and index data.

[0043] For each single environmental state indicator data in each group of data, first count the total number of valid data, that is, the remaining data amount after excluding null values and outliers. When calculating the arithmetic mean, all valid index data in the group are added up, and the sum is divided by the total number of valid data, retaining six significant digits after the decimal point. When calculating the unbiased variance, first subtract the arithmetic mean of the group from each valid data in the group to obtain the difference, square each difference, and accumulate all squared differences to obtain the total sum of squares. Divide the total sum of squares by the total number of valid data minus one (i.e. degrees of freedom n-1), and also retain six significant digits after the decimal point.

[0044] When calculating the median, all valid index data in the group are sorted in ascending order. If the total number of valid data is odd, the value at the middle position after sorting (the position is the total number of valid data plus one divided by two) is determined as the median. If the total number of valid data is even, the values at the two middle positions after sorting (the positions are the total number of valid data divided by two and the total number of valid data divided by two plus one, respectively) are calculated, and the arithmetic mean of the two values is calculated as the median.

[0045] When calculating the mode, the frequency of each different index value in the group is counted, and the value with the highest frequency is recorded. If there are two or more values with the same frequency and the highest frequency, the arithmetic mean of these values is calculated as the mode of the group index data. For all environmental state index data in each grid cell, repeat the calculation process of the arithmetic mean, unbiased variance, median, and mode to fully depict the distribution and dispersion of environmental state indicators in each grid cell at different time slices. Integrate the four statistical results corresponding to each grid cell and each time slice by index type to form a grid cell-specific statistical distribution feature with dimensions of grid number x time slice number x 4. Use indexed storage to associate grid numbers, time slice identifiers, and statistical results to support quick retrieval and call of corresponding statistical data according to the time or spatial dimension.

[0046] b10, based on the statistical distribution characteristics of the environmental state indicators in each grid cell, calculates the dynamic correlation strength between adjacent grid cells through spatial autocorrelation analysis, specifically including: using spatial autocorrelation analysis method, selecting Moran's index as global correlation calculation index, selecting Local Indicators of Spatial Association (LISA) statistics as local correlation calculation index, using Queen adjacency rule (grid sharing edge or vertex is determined as adjacent grid), performing correlation operation on the mean value of the environmental state index statistical distribution characteristics of each grid cell and the mean value of the statistical distribution characteristics of the surrounding adjacent grid cells, quantifying the correlation closeness between adjacent grid cells through the calculation result, mapping the correlation strength value range to the interval of 0 to 1, wherein correlation strength greater than or equal to 0.7 is determined as strong correlation, 0.3 to 0.7 is medium correlation, and less than or equal to 0.3 is weak correlation, obtaining the dynamic correlation strength between each grid cell and each adjacent grid, and capturing the linkage characteristics in the spatial dimension.

[0047] b11, the statistical distribution characteristics of the environmental state indicators in each grid cell and the dynamic correlation strength between adjacent grid cells are integrated, and a spatial dynamic weight matrix is calculated by weighted fusion, specifically including: an adaptive learning method based on a gradient boosting tree model, setting the initial weight coefficients of the statistical distribution characteristics and the dynamic correlation strength, i.e. the statistical distribution characteristics 0.6 and the dynamic correlation strength 0.4, optimizing the weight coefficients through 5-fold cross-validation to ensure that the feature fusion error of the validation set is minimized, and the sum of the weight coefficients is constrained to be 1, multiplying the mean value of the environmental state indicator statistical distribution characteristics of each grid cell by the corresponding weight coefficient to obtain the statistical feature weighted value, multiplying the dynamic correlation strength values of the grid cell and the adjacent grid by the corresponding weight coefficients, and then summing the correlation strength weighted values of all adjacent grids to obtain the total correlation strength weighted value, adding the statistical feature weighted value and the total correlation strength weighted value to obtain the dynamic weight value of each grid cell, and arranging the dynamic weight values of all grid cells according to the grid number to form a spatial dynamic weight matrix (dimension: grid number x grid number), with the row corresponding to the target grid and the column corresponding to the adjacent grid.

[0048] b12, multiply the spatial dynamic weight matrix and the preliminary indicator set element by element to realize feature weighted fusion in the spatial domain and obtain a weighted feature mapping, specifically including: first, extract the dimension parameters of the spatial dynamic weight matrix and the preliminary indicator set, determine that the spatial dynamic weight matrix is a two-dimensional structure, the row dimension corresponds to the total number of target grids, and the column dimension corresponds to the total number of adjacent grids, and the preliminary indicator set is a two-dimensional structure, the row dimension corresponds to the total number of target grids, and the column dimension corresponds to the total number of features; perform a tensor dimension compatibility check to verify whether the row dimension of the spatial dynamic weight matrix and the row dimension of the preliminary indicator set are completely equal, to ensure that the index identification of each target grid in the two types of data is one-to-one corresponding, and to meet the dimension matching core requirement of broadcast operation.

[0049] Determine the broadcast expansion rule, expand the spatial dynamic weight matrix from a two-dimensional structure to a three-dimensional tensor, expand the dimension to a feature dimension, and expand the matrix in each grid correlation weight value along the feature dimension to make each correlation weight and the corresponding grid in the preliminary indicator set form a one-to-one mapping relationship, and the expanded three-dimensional tensor dimension is target grid number x adjacent grid number x feature number; at the same time, expand the preliminary indicator set from a two-dimensional structure to a three-dimensional tensor, expand the dimension to an adjacent grid dimension, and expand the feature vector of each grid along the adjacent grid dimension to make each feature value and the correlation weight between the corresponding grids in the spatial dynamic weight matrix form a one-to-one mapping relationship, and the expanded three-dimensional tensor dimension is consistent with the expanded dimension of the weight matrix.

[0050] The target grid dimension of the three-dimensional weight tensor is traversed in row priority order, and each target grid corresponding three-dimensional sub-tensor is processed in turn, and the sub-tensor dimension is adjacent grid number x feature number; for each target grid three-dimensional sub-tensor, an element-by-element multiplication operation is performed with the three-dimensional index tensor corresponding to the target grid three-dimensional sub-tensor, that is, the weight value corresponding to the same adjacent grid index and the same feature index in the sub-tensor is directly multiplied with the feature value, and during the operation process, the last six significant digits after the decimal point are retained, and the remainder is processed using the rounding rule to ensure controllable calculation accuracy.

[0051] After completing the element-by-element multiplication, an accumulated summation operation is performed on the three-dimensional product sub-tensor of each target grid according to the adjacent grid dimension, the weighted contribution of all adjacent grids to each feature dimension of the target grid is summarized, and the last six significant digits after the decimal point are maintained during the accumulation process to avoid precision loss; after traversing all target grids and completing the above operation, the accumulated result of each target grid is arranged as a one-dimensional feature vector, and the vector dimension is the number of features; all one-dimensional feature vectors are arranged in turn according to the target grid index order to form a weighted feature mapping with a dimension of target grid number x feature number; the mapping aggregates the element-by-element weighting and adjacent grid contribution of the preliminary index set through the spatial dynamic weight matrix, realizes the deep integration of the spatial dynamic correlation relationship into the feature expression, achieves the accurate adjustment of the preliminary index, and strengthens the expression effect of the spatial correlation feature.

[0052] b13, based on the weighted feature mapping, context calibration is performed on the feature propagation of adjacent network nodes to obtain context-aware features; the context-aware features are compressed and standardized to obtain optimized comprehensive state indicators, which specifically include: first, based on the adjacent grid dynamic correlation strength obtained in b10, a graph neural network-based adjacent grid node feature propagation network is constructed, taking each spatial grid unit as a network node, each node carrying the weighted feature vector output by b12, and taking whether there is an adjacent relationship (including direct adjacent and 3-order indirect adjacent) between grids as a network edge, and the initial weight of the edge is assigned as the dynamic correlation strength between the corresponding grids; the definition of 3-order neighborhood is determined, the 1-order neighborhood is the direct adjacent grid sharing the edge or vertex with the target grid, the 2-order neighborhood is the grid directly adjacent to the 1-order neighborhood grid and not included in the 1-order neighborhood, and the 3-order neighborhood is the grid directly adjacent to the 2-order neighborhood grid and not included in the 1-order to 2-order neighborhood, and the propagation distance threshold is set to 3 grids, that is, only allows features to propagate within 3-order neighborhood.

[0053] The calculation rule of the propagation weight coefficient is set, and the target grid is taken as the center. The propagation weight coefficient of the first-order neighborhood is the corresponding dynamic correlation strength multiplied by the first power of 0.8, the propagation weight coefficient of the second-order neighborhood is the corresponding dynamic correlation strength multiplied by the second power of 0.8, and the propagation weight coefficient of the third-order neighborhood is the corresponding dynamic correlation strength multiplied by the third power of 0.8. After calculation, the weight coefficient of each neighborhood is checked for effectiveness, and the propagation path corresponding to the weight coefficient less than 0.01 is removed to avoid the interference of weak correlation on feature propagation.

[0054] The feature propagation process is started. Each grid unit splits the weighted feature vector borne by itself into multiple feature components according to the propagation weight coefficients of the corresponding neighborhoods, and transmits the feature components to the corresponding adjacent nodes in the third-order neighborhood. At the same time, each grid node receives the feature components transmitted by all adjacent nodes in its third-order neighborhood, classifies and counts all received feature components, and records the source grid, propagation order and propagation weight coefficient corresponding to each feature component.

[0055] The weight normalization coefficient of the received feature is calculated. The propagation weight coefficients of all received feature components are accumulated and summed to obtain the weight sum. For each received feature component, the propagation weight coefficient is divided by the weight sum to obtain the normalized weight of the feature component, ensuring that the sum of the normalized weights of all received feature components is 1. The received feature components are weighted and averaged according to the feature dimension. For each feature dimension, the values of all received feature components in that dimension are multiplied by the corresponding normalized weight, and all product results are accumulated and summed to obtain the calibrated value of that dimension. The weighted average calculation of all feature dimensions is completed in turn to form the context-aware feature vector of the grid node, realizing the interactive calibration of adjacent node features. Feature compression processing is performed on the context-aware feature vectors of all grid nodes. A maximum pooling kernel of 2x2 size is used, and the pooling step is set to 1. Each context-aware feature vector is divided into a group according to consecutive 4 feature dimensions, forming a total of feature number divided by 4 groups. The 4 feature values in each group are compared in size, and the largest feature value is selected as the pooling result of the group. The pooling operation of all groups is completed in turn to obtain the compressed feature with the feature dimension reduced to one-fourth of the original feature number.

[0056] The min-max standardization method is used for scale unification of the compressed features. The minimum value and the maximum value of each dimension in the compressed features of all grid nodes are counted. The value of each dimension of the compressed features of each grid node is obtained by subtracting the minimum value of the corresponding dimension from the value and dividing the obtained difference by the difference between the maximum value and the minimum value of the corresponding dimension. The feature value is mapped to the interval of 0 to 1, and the six significant digits after the decimal point are retained during the calculation. Finally, the feature vectors of all grid nodes after the standardization processing are arranged in the order of grid numbers to obtain the optimized comprehensive state index with a dimension of one quarter of the product of the grid number and the feature number, so as to ensure that the index has context correlation, dimension reduction and scale unification.

[0057] b14, based on the optimized comprehensive state index, performing feature upsampling and spatial resolution enhancement to obtain a high-resolution feature mapping, specifically including: first extracting the dimension parameter of the optimized comprehensive state index to determine the corresponding original feature map resolution, which is directly corresponding to the grid number, for example, when the grid number is 1000x1000, the original feature map resolution is 1000x1000, and the target resolution is determined according to the demand for spatial details of the application scene, for example, 2000x2000. The upsampling multiple is calculated by the ratio of the target resolution to the original resolution, and when the target resolution is twice the original resolution, the upsampling multiple is determined to be 2. At the same time, the optimized comprehensive state index is arranged in the order of grid numbers to regularize the two-dimensional feature map format. The row dimension of the feature map corresponds to the north-south direction distribution of the grid, and the column dimension corresponds to the east-west direction distribution of the grid. Each pixel point carries the feature value of the corresponding grid, so as to ensure that the data structure is adapted to the subsequent upsampling algorithm.

[0058] If the bilinear interpolation method is used for up-sampling processing, the specific calculation process is as follows: a blank interpolation feature map is created according to the target resolution, and the number of rows and columns of the interpolation feature map is consistent with the number of rows and columns of the target resolution; for each pixel point to be interpolated in the interpolation feature map, the mapping coordinates of the pixel point in the original feature map are calculated in reverse through the up-sampling multiple, that is, the original coordinates = the coordinates of the pixel point to be interpolated / the up-sampling multiple; the nearest four original feature map pixel points around the pixel point to be interpolated are determined according to the mapping coordinates, that is, the four adjacent original pixels in the upper, lower, left and right, and the row and column coordinates and the corresponding feature values of the four original pixels in the original feature map are recorded; the row and column coordinate difference values of the pixel point to be interpolated and each adjacent original pixel are calculated to obtain the horizontal distance and the vertical distance, and the distance unit is the number of pixels; the weight is allocated according to the inverse square law of distance, that is, the smaller the square value of the distance between the pixel point to be interpolated and the original pixel, the greater the corresponding weight, and the weight is normalized after calculation to ensure that the total weight of the four original pixels is 1; the feature value of each original pixel is multiplied by the corresponding weight to obtain four weighted feature values, and the four weighted feature values are accumulated to obtain the feature value of the pixel point to be interpolated, and the calculation process retains six significant digits after the decimal point; all the pixel points to be interpolated in the interpolation feature map are processed in row priority order, and the pixel gaps in the low-resolution feature map are filled to finally generate a high-resolution feature map consistent with the target resolution.

[0059] If the transpose convolution layer is used to perform the deconvolution operation for up-sampling processing, the specific calculation process is as follows: the core parameters of the transpose convolution layer are preset, the convolution kernel size is set to 3x3, the step size is set to 2, the padding method is edge padding, and the number of padding pixels is set to 1 to ensure that the output feature map resolution is exactly twice that of the input feature map; the 3x3 convolution kernel weight is initialized, the weight value is initialized by the Xavier method, the initial value is adaptively allocated based on the input feature dimension and the output feature dimension to avoid unstable training caused by too large or too small initial weight; the two-dimensional feature map corresponding to the optimized comprehensive state index is taken as the input feature map, the edges of the input feature map are filled, and 1 pixel of padding value is added to each of the four edges of the feature map, the padding value is the average value of the edge pixels of the input feature map, and the edge information is avoided to be lost; the deconvolution operation is performed on the filled input feature map according to the set step size 2, the 3x3 convolution kernel is slid to cover each pixel area of the input feature map, each element of the convolution kernel is multiplied by the pixel value of the corresponding area of the input feature map, and then all the products are accumulated to obtain the intermediate result of the convolution operation; according to the sliding rule with a step size of 2, the intermediate results are arranged in the output feature map with an interval of 1 pixel, the pixel gap is automatically filled, and the chessboard artifact after up-sampling is avoided through the weight balanced distribution of the convolution kernel; after all the convolution sliding operations are completed, the output feature map consistent with the target resolution (such as 2000x2000) is obtained.

[0060] After the upsampling process is completed by any one of the above two methods, the output feature map is re-regularized into a tensor format according to the grid number and feature dimension, and a high-resolution feature mapping is obtained, which has a dimension of target grid number (corresponding to the product of the row and column of the target resolution) x target feature number (consistent with the optimized comprehensive state index feature number). The spatial resolution of the feature mapping is significantly improved compared to the original feature map, and the spatial detail information is richer, which can accurately depict the feature differences of different grid units.

[0061] b15, based on the high-resolution feature mapping, a continuous spatial distribution feature field is obtained through a spatial interpolation algorithm, specifically including: first, extract the grid parameters corresponding to the high-resolution feature mapping, determine the spatial coordinate boundary, size and feature value of each grid unit. The feature mapping has covered the main spatial range of the pre-defined key area, and there is only a blank space area between the grid units; according to the spatial detail expression requirement, the blank space area and the original grid coverage area are uniformly divided into 0.5m x 0.5m uniform interpolation grids, each interpolation grid corresponds to a spatial point of the to-be-estimated feature value, and the latitude and longitude coordinates of all interpolation grid points are recorded to ensure that the feature field after interpolation can completely cover the entire key area.

[0062] If the ordinary Kriging interpolation algorithm is selected for feature value estimation, the specific calculation process is as follows: first, pre-process all grid unit feature values in the high-resolution feature mapping, eliminate outliers, and retain effective feature values and corresponding grid coordinates as interpolation sample points through 3σ criterion judgment; the spatial distance and corresponding feature value difference between any two sample points are counted, and a Gaussian-type variation function is fitted based on these statistical data to determine the base value, nugget value and range parameters of the variation function, to ensure that the goodness of fit R 2 is greater than or equal to 0.95; for each interpolation grid point, 8 effective sample points, i.e. 8 nearest grid units, are selected according to the spatial distance from the center of the point to the farthest; if there are less than 8 sample points within the range, all sample points within the range are used as the basis for interpolation; based on the fitted Gaussian-type variation function, the spatial correlation between the interpolation point and each selected sample point is calculated, and a Kriging equation set is constructed combining the spatial distribution law of the sample point feature value, and the weight coefficients of each sample point are obtained by solving the equation set, and the sum of the weight coefficients is 1; the feature value of each sample point is multiplied by the corresponding weight coefficient, and the product of all the results is accumulated to obtain the feature estimation value of the interpolation grid point; the average absolute error of the estimated value and the feature values of the nearest 3 sample points is calculated, if the error is less than or equal to 5%, the estimated value is retained, if the error exceeds 5%, 10 nearest sample points are reselected and the above estimation process is repeated until the error requirement is met.

[0063] If the radial basis function interpolation algorithm is selected for eigenvalue estimation, the specific calculation process is as follows: the smoothing parameter of the Gaussian radial basis function is determined, which is set to 1 / 2 of the grid size in the high-resolution feature map, such as 0.5m if the original grid size is 1m; for each interpolation grid point, sort and select 8 nearest effective sample points (grid elements in the high-resolution feature map) according to the spatial distance, record the eigenvalues of the sample points and the spatial distance from the interpolation point; the spatial distance between each sample point and the interpolation point is substituted into the Gaussian radial basis function to calculate the correlation degree between the sample point and the interpolation point (the correlation degree decreases with the increase of the distance); the correlation degrees of all sample points are normalized to obtain the weight coefficient of each sample point; the eigenvalue of each sample point is multiplied by the corresponding weight coefficient, and the product of all is accumulated to obtain the estimated value of the eigenvalue of the interpolation grid point; the difference between the estimated value and the eigenvalue of the surrounding sample points is also checked to ensure that the difference is less than or equal to 5%, if it does not meet the requirements, the smoothing parameter is adjusted (within 0.8 to 1.2 times of the original parameter) to recalculate until the continuous consistency requirement is met; the eigenvalue estimation of all interpolation grid points is completed in row priority order, and finally the grid eigenvalues of the original high-resolution feature map and the estimated eigenvalues of all interpolation grid points are integrated to form a continuous spatial distribution feature field covering the entire key area without spatial breakpoints.

[0064] b16, based on the continuous spatial distribution feature field, the feature field value is mapped to the corresponding color space value to obtain a grid image; based on the grid image, the gradient distribution of the feature field is calculated to obtain contour line vector data; the grid image and the contour line vector data are subjected to layer superposition and semantic annotation processing to obtain a global environment situation awareness map, which specifically includes: first, a one-to-one mapping rule between the feature field value and the Red Green Blue (RGB) color space value is established, the range of the eigenvalues of all spatial points in the continuous spatial distribution feature field is first counted and normalized to the interval of 0 to 1, and then five continuous numerical intervals are divided, which are 0 to 0.2, 0.2 to 0.4, 0.4 to 0.6, 0.6 to 0.8, and 0.8 to 1.0; each numerical interval is assigned a corresponding RGB color value, wherein the 0 to 0.2 interval corresponds to RGB255, 255, 255, the 0.2 to 0.4 interval corresponds to RGB200, 200, 255, the 0.4 to 0.6 interval corresponds to RGB150, 150, 255, the 0.6 to 0.8 interval corresponds to RGB100, 100, 255, and the 0.8 to 1.0 interval corresponds to RGB50, 50, 255, ensuring that the higher the value, the higher the color saturation, and the color transition between adjacent intervals is natural.

[0065] Traverse each spatial point in the continuous spatial distribution characteristic field, read the characteristic value of the point, judge its belonging numerical interval, determine the corresponding RGB color value according to the mapping rule, the color value is an integer (0 to 255 range); create a blank raster image according to the resolution of 0.5m / pixel, the row and column number of the raster image is determined by the latitude and longitude range of the key area and the pixel resolution (row number = north-south span of the area / 0.5m, column number = east-west span of the area / 0.5m); the RGB color value of each spatial point is written into the corresponding pixel position of the raster image according to its latitude and longitude coordinates, after the color value mapping of all spatial points is completed, the raster image in the format of Portable Network Graphics (PNG) is generated, which directly reflects the spatial distribution of the characteristic field.

[0066] The Sobel operator is used to calculate the gradient value of each spatial point in the characteristic field, and the specific process is as follows: a 3*3 Sobel horizontal operator and a vertical operator are constructed, a 3*3 neighborhood spatial point is selected with each spatial point as the center, and the characteristic values of the points in the neighborhood are extracted; the neighborhood characteristic values are convolved with the horizontal operator to obtain the horizontal gradient value of the point; the neighborhood characteristic values are convolved with the vertical operator to obtain the vertical gradient value of the point; the gradient strength is calculated by summing the square of the horizontal gradient value and the square of the vertical gradient value and then taking the square root, and the last three significant digits after the decimal point are retained; the contour interval is set according to the gradient strength, the contour interval is set to 0.1 when the gradient strength is less than or equal to 0.1, and the contour interval is set to 0.05 when the gradient strength is greater than 0.1; traverse all spatial points in the characteristic field, extract spatial points with equal characteristic values, connect these points in the order of spatial position to form closed or continuous contour lines, and all contour lines constitute contour line vector data, which contains coordinate string, corresponding characteristic value and interval information of the contour line.

[0067] The raster image generated is imported as a bottom layer of the Geographic Information System (GIS) software by using the layer fusion function of the GIS software, and the contour line vector data is imported as a superimposed layer of the GIS software, the transparency of the superimposed layer is adjusted to 50%, the color of the bottom layer and the superimposed contour line are ensured to be clear and visible, semantic annotation information is added, the annotation content includes grid number (the number of the original rule grid is annotated at the corresponding grid center position), index name (annotated at the upper left corner of the image), numerical range (annotated at the upper right corner of the image), legend (annotated at the lower right corner of the image, including numerical interval and corresponding color block), the annotation font is Arial, and the font size is set to 10pt; the annotation position avoids the feature value mutation area, the gradient strength is judged, the area with gradient strength greater than 0.3 is the mutation area, and the annotation is avoided to shield the key feature distribution information; after the layer superposition and semantic annotation are completed, the result is exported as a global environmental situation awareness map in the Geographic Tagged Image File Format (GeoTIFF), which supports latitude and longitude coordinate positioning and feature value attribute query function, and ensures the practicability and ease of use of the situation awareness map.

[0068] In the embodiment of the application, by dividing the continuous data stream into an equal-interval time slice sequence, an ordered time framework can be established, and data time sequence controllability can be improved; spatial registration is performed on multi-modal data in each time slice to establish spatial reference data in a unified coordinate system; spatial misplacement of multi-source data is eliminated, and data spatial correlation is strengthened, thereby building a unified foundation for artificial intelligence to fuse spatial heterogeneous data; time-space feature matrices, flow feature vectors and regional average dynamic features are extracted for different types of spatial reference data; feature expression dimensions are enriched, and exclusive characteristics of various data are mined; multi-type features are dimensionally aligned and vectorized and spliced to obtain multi-source feature vectors; multi-source feature dimension inconsistency problems are solved, structured feature sets are constructed, and complexity is reduced.

[0069] The attention weight matrix is calculated through a full connection layer and a softmax function; key feature components are automatically identified to improve the proportion of core information weight; the attention weight matrix and the multi-source feature vector are weighted and summed to obtain an attention weighted fusion feature; key feature expression is strengthened and irrelevant feature interference is suppressed; the attention weighted fusion feature is dimensionally reduced and standardized to obtain a preliminary index set, which simplifies the feature dimension and unifies the feature scale; a rule grid topology structure is constructed in combination with a key region boundary to map abstract indexes to specific spatial units, establish a spatial structured framework, and help understand the spatial distribution rule of data; the preliminary index set is mapped to each grid unit to calculate the statistical distribution characteristics of the environmental state index, mine the state distribution rule in the grid, and provide data support; the dynamic correlation strength of adjacent grid units is calculated through spatial autocorrelation analysis to capture the dynamic dependence relationship between grids, construct a spatial correlation network, and enhance the perception ability of spatial linkage characteristics; the spatial dynamic weight matrix is obtained by comprehensively considering the statistical distribution characteristics and the dynamic correlation strength, the spatial adaptive adjustment of the weight is realized, the spatial feature weighting logic is optimized, and the spatial adaptability of feature fusion is improved.

[0070] The spatial dynamic weight matrix and the preliminary index set are multiplied element by element to obtain a weighted feature mapping, the spatial dynamic correlation is integrated into feature expression, and the spatial differentiation representation ability of the feature is improved; context calibration is performed on adjacent node feature propagation, and an optimized comprehensive state index is obtained through compression and standardization, thereby improving the context consistency of the feature; the comprehensive state index is feature up-sampled and spatial resolution is improved to obtain a high-resolution feature mapping, thereby strengthening the spatial detail expression of the feature and laying a foundation for generating a fine-grained situation awareness map; the high-resolution feature mapping is converted into a continuous spatial distribution feature field by a spatial interpolation algorithm, and discrete grid features are converted into continuous spatial representations; the feature field is mapped into a raster image, contour line vector data is calculated, and layer overlay and semantic labeling are performed to obtain a global environmental situation awareness map, construct an intuitive and semantically rich situation representation, and improve the explainability of the decision-making process.

[0071] In some embodiments, the step 300 inputs the global environmental situation awareness map into a pre-trained multi-target collaborative decision-making model; the multi-target collaborative decision-making model forms a multi-target decision-making feature set by analyzing the resource entities and the correlation relationships in the global environmental situation awareness map; based on the multi-target decision-making feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme, including: c1-c5.

[0072] c1, analyze the global environment situation awareness graph, extract key state indicators for representing task processing efficiency, resource utilization balance, and service demand satisfaction, construct a multi-objective decision feature set, which specifically includes: first, load the global environment situation awareness graph, which contains multi-modal data association information after spatial alignment and feature fusion, extract resource entity information in the graph through the geographic information system (GIS) analysis interface, including the identification, spatial coordinates, and running state data of fixed sensors, mobile devices, task execution units, and other entities, and analyze the association between entities, including spatial proximity association, task dependency association, and resource supply and demand association; based on the core objectives of the application scenario, determine the extraction rules of three types of key state indicators, the task processing efficiency indicator is calculated by counting the number of tasks completed per unit time, the average time from task reception to completion, and the task queue length, the resource utilization balance indicator is obtained by statistical analysis of the load rate, resource occupation time, and idle time proportion of each resource unit, and the service demand satisfaction indicator is determined by parameters such as the ratio of demand response compliance quantity to total demand quantity and whether the demand response delay is within the preset threshold; perform data cleaning on the extracted indicators to remove outliers and missing values, unify the data type and dimension of the indicators, and construct a structured multi-objective decision feature set according to entity identification, timestamp, and indicator category, the feature set is stored in tensor format for easy reading and processing by subsequent models.

[0073] c2, based on the multi-objective decision feature set, construct a global multi-objective optimization function that simultaneously optimizes running efficiency, resource cost, and service quality, and set the running constraint conditions of each functional unit, which specifically includes: based on the multi-objective decision feature set, determine the core dimensions of global multi-objective optimization, the running efficiency dimension aims to maximize task throughput and minimize average processing delay, the resource cost dimension aims to minimize total energy consumption and device wear coefficient, and the service quality dimension aims to maximize demand satisfaction rate and optimize user satisfaction; construct the global multi-objective optimization function using weighted summation, the weight coefficients are determined by the analytic hierarchy process combined with scene demand, construct the judgment matrix according to the 1-9 scale method, determine the final weight after normalization processing and consistency test (consistency ratio (CR) less than or equal to 0.1), and ensure that the importance of each optimization objective matches the actual application demand.

[0074] The constraint convex set projection algorithm optimizes the construction of the constraint system. The specific calculation process is as follows: first, map each type of constraint to the corresponding convex set. The hardware constraint convex set takes the maximum load threshold of the device and the upper limit of the operation processing capacity as the boundary. The time constraint convex set takes the maximum delay allowed by the task response and the minimum interval of resource scheduling as the boundary. The resource constraint convex set takes the total amount of energy supply limit and the usage quota of key consumables as the boundary. For each convex set, define the mathematical boundary conditions of the convex set to ensure that the constraints meet the closure and convexity requirements of the convex set. Calculate the intersection of each type of constraint convex set to obtain the global constraint convex set. Generate 100 to 200 groups of candidate solutions, i.e., parameter combinations covering device load, task delay, resource consumption, etc., and substitute the candidate solutions into the global constraint convex set for projection operation to calculate the shortest distance from the candidate solution to the convex set boundary. If the distance from a candidate solution to the convex set boundary is 0, it means that the candidate solution falls within the convex set, and the constraint system has a feasible solution. If the distance from all candidate solutions to the convex set boundary is greater than 0, it means that the constraint convex set is empty, and the rigid constraint threshold needs to be adjusted by 5% to 10%, such as adjusting the maximum load threshold of the device from 80% of the rated load to 85%. Reconstruct the constraint convex set and repeat the projection verification until there is a candidate solution that falls within the convex set. Finally, determine the boundary threshold of each constraint condition. The hardware constraint threshold is set based on 20% redundancy reserved for device performance parameters. The time constraint threshold is determined based on the response delay upper limit of 95% of the tasks according to the service level agreement. The resource constraint threshold is set by deducting 15% fluctuation reservation from the resource supply capacity. A complete constraint system with feasibility and strictness is formed. The constraint convex set projection algorithm ensures that the optimization process is within the convex feasible region, and the global multi-objective optimization function has an optimal solution.

[0075] c3, decomposing the global multi-objective optimization function into local value functions associated with each agent; each agent generates a local policy based on its local value function and the environment state; all local policies are integrated by a centralized coordinator, which evaluates its global coordination utility using an attention mechanism, and obtains a non-dominated solution set as the initial coordination policy set by iterative search within the constraint solution space, which specifically includes: adopting a multi-agent reinforcement learning paradigm based on value function decomposition, selecting a multi-objective evolutionary algorithm based on decomposition (Multi-Objective Evolutionary Algorithm based on Decomposition, MOEAD) to divide the global multi-objective optimization function according to the responsibilities of the agents, and dividing the agents according to the types of functional units or spatial regions, each agent corresponds to a local sub-objective, and the decomposition process ensures that the target direction of each local value function is consistent with the global optimization function, and the cumulative result of all local value functions is equivalent to the global optimization function; configure an independent reinforcement learning policy network for each agent, the network input is the local features in the multi-objective decision feature set corresponding to the agent's responsible range and the real-time environment state, the environment state includes the current resource occupation, the task queue length, the service demand change trend, etc.; each agent generates a local policy based on the optimization objective of its local value function through the policy network, the local policy includes resource allocation ratio, task execution priority, device start-stop scheme, etc. specific operation instructions.

[0076] Fusion of distributed consensus algorithm optimizes the consistency and effectiveness of local strategy, selects a practical Byzantine fault tolerance consensus algorithm to construct a distributed consensus mechanism, sets the consensus participating nodes as all agents, the upper limit of consensus rounds is 3 rounds, and the fault tolerance threshold is one third of the total number of agents, that is, at most one third of the total number of agents is allowed to output abnormal strategy; specifically, each agent attaches its own identification information and strategy generation timestamp to the generated local strategy, and broadcasts it to all other agent nodes; after each agent receives all the broadcasted local strategies from other nodes, it first verifies the format specification and its own identification validity of the strategy, and eliminates the strategies with incorrect format or invalid identification; then, based on the constraint conditions of the local value function, it checks whether the received strategy meets the requirements of the local sub-objective of the agent, and counts the number of valid strategies that meet the requirements.

[0077] Enter the consensus preparation phase, each agent performs a hash operation on the valid strategy to generate a strategy digest, and broadcasts the digest and its own vote (support or opposition) to all nodes again; after all nodes complete the vote broadcast, each agent counts the number of support votes for each local strategy, and calculates the proportion of support votes to the total number of valid agents; when the support proportion of a local strategy reaches two-thirds or more, the strategy is determined to pass the consensus verification; if there are strategies whose support proportion does not reach the threshold, feedback to the corresponding generating agent, requiring it to adjust the strategy parameters based on the reasons for not passing the consensus feedback (such as violating local constraints, conflicting with other strategies), regenerate the local strategy and repeat the above broadcast and verification process until all local strategies of the agents pass the consensus verification.

[0078] A centralized coordinator is built, and the coordinator is built-in with an attention mechanism module, which allocates attention weights by calculating the correlation between each consensus-passing local strategy and the global optimization goal, and focuses on local strategies that have a significant impact on global performance; the coordinator imports all consensus-passing local strategies into the constraint solution space, sets an upper limit on the number of iterations and a convergence error threshold, and gradually iteratively optimizes according to the greedy search algorithm, evaluates the global collaborative utility of the strategy combination at each iteration, and removes strategy combinations that violate the constraint conditions or have low utility, until the number of iterations reaches the upper limit or the utility improvement is less than the convergence threshold, and outputs a non-dominated solution set that satisfies the Pareto optimality condition, which is used as the initial collaborative strategy set.

[0079] c4, multi-dimensional performance evaluation and comprehensive trade-off analysis are performed on the initial collaborative strategy set, and the comprehensive utility value of each candidate strategy is calculated; the candidate strategies are sorted according to their comprehensive utility values to obtain a comprehensive utility ranking, which includes: constructing a hierarchical multi-dimensional performance evaluation system, determining a three-level structure, the target layer is the comprehensive performance evaluation result, the dimension layer corresponds to three core dimensions of running efficiency, resource cost and service quality, and the index layer is the specific quantitative index under each dimension, the running efficiency dimension sets the task completion rate and average processing delay, the resource cost dimension sets the unit task energy consumption and equipment depreciation loss, and the service quality dimension sets the demand satisfaction rate and response compliance rate, all indexes are positively processed to ensure that the higher the value, the better the performance.

[0080] Determine the weights of each level based on the analytic hierarchy process, construct a hierarchical model of the target layer, dimension layer and index layer, score the relative importance between dimensions and between indexes of the same dimension according to the 1-9 scale method, and form the corresponding judgment matrix; calculate the maximum eigenvalue and eigenvector of the matrix, and obtain the initial weights after normalization; verify the rationality through consistency check (CR less than or equal to 0.1), if not qualified, adjust the score until it passes, and finally determine the weights of the dimension layer and the index layer, and the sum of the weights of the same level is 1.

[0081] The fusion hierarchical geometric weighting algorithm calculates the comprehensive utility value, and the specific process is as follows: for each candidate strategy in the initial cooperative strategy set, the evaluation system is substituted one by one, the actual scores of each index are calculated based on the strategy simulation execution data, and the actual scores are uniformly normalized to the interval of 0 to 1; the hierarchical rules are used to calculate step by step, first, for each dimension, the scores of the two indexes under the dimension are multiplied after taking the corresponding index weight to the power, to obtain the dimension geometric weighted score, and the last four digits after the decimal point are retained; then the scores of the three dimensions are multiplied after taking the corresponding dimension weight to the power, to obtain the comprehensive utility value of the candidate strategy, and the last four digits after the decimal point are retained; the comprehensive utility values of all candidate strategies are sorted in descending order to obtain the comprehensive utility sorting result; the index original score, dimension weighted score and comprehensive utility value of each strategy are recorded, and a structured evaluation report is formed by arranging the strategy number.

[0082] c5, based on the comprehensive utility sorting, selecting a dominant strategy from the initial cooperative strategy set to obtain a comprehensive cooperative scheduling scheme, specifically including: based on the comprehensive utility sorting result, preferentially screening the candidate strategies with high ranking, and combining the real-time demand priority of the application scene to perform secondary verification, if the scene is an emergency response type, the service quality dimension score is mainly checked, if the scene is a cost sensitive type, the resource cost dimension score is mainly checked; from the candidate strategies that pass the verification, the strategy with the highest comprehensive utility value and relatively balanced dimension scores is selected as the dominant strategy, if there are multiple strategies with the same comprehensive utility value, the final dominant strategy is determined by random sampling combined with scene adaptability analysis; the dominant strategy is disassembled into specific executable operation instructions, including resource allocation scheme, task scheduling order, running parameter setting, etc. of each functional unit, the execution subject, execution time and execution standard of the instruction are clear; the feasibility of the dominant strategy is verified, the resource consumption, task completion and service quality performance in the strategy execution process are simulated, to ensure that the strategy meets all constraint conditions and has no logical conflict; finally, a structured comprehensive cooperative scheduling scheme is formed, the scheme includes strategy details, execution steps, expected performance and emergency adjustment mechanism, and supports direct import system execution.

[0083] In the embodiments of the present application, the global environment situation awareness map is parsed and key state indicators are extracted to construct a multi-objective decision feature set; the visual space situation is converted into structured decision features to strengthen the relevance of the features to task processing efficiency, resource utilization balance, and service demand satisfaction; a global multi-objective optimization function is constructed based on the multi-objective decision feature set, and functional unit operation constraints are set; the optimization direction and boundary limit of the decision are determined to establish a standardized multi-objective optimization framework; the decision logic of the artificial intelligence system is made clearer to improve the systematicness and controllability of the multi-objective optimization task; the global optimization function is converted into a local value function to generate local strategies and integrate them through a centralized coordinator to realize the coordination and unity of global objectives and local behaviors, and to exert the advantages of distributed processing and the global control ability of centralized coordination; the initial collaborative strategy set is evaluated in multiple dimensions for performance to calculate and sort the comprehensive utility value, and a quantitative strategy evaluation system is established to provide an objective basis for artificial intelligence strategy selection; a dominant strategy is selected based on the comprehensive utility sorting to form a comprehensive collaborative scheduling scheme; the optimal strategy is accurately screened to construct a complete closed loop from feature extraction to decision output, and the comprehensive adaptation ability of the decision scheme is strengthened.

[0084] In some embodiments, the above step 400, the comprehensive collaborative scheduling scheme is parsed and encapsulated for instructions to obtain an executable instruction sequence; the executable instruction sequence is parallelly issued to the corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative publishing of control instructions, including: d1-d4.

[0085] d1, semantic analysis and instruction conversion are performed on the comprehensive collaborative scheduling scheme to generate an executable instruction sequence including a device control instruction set, a resource scheduling instruction set, and a service publishing instruction set, specifically including: first, loading the comprehensive collaborative scheduling scheme, which includes global resource allocation strategy, device operation parameter configuration, service execution standard and constraint condition and other structured information, the scheme is hierarchically disassembled by a semantic analysis engine, the decision target, execution subject identifier, operation object type and core parameter in the scheme are extracted first, and then the logical association and execution time sequence requirement of each decision item are disassembled; based on the preset instruction conversion rule library, the disassembled decision information is mapped to specific executable instructions to generate three types of instruction sets, the device control instruction set includes the unique identifier, operation type, target parameter value and execution time limit of each execution device, wherein the operation type is, for example, start, stop, parameter adjustment, the resource scheduling instruction set includes resource type, allocation object identifier, allocation proportion, occupation time length and release condition, wherein the resource type is, for example, computing resource, energy resource, storage resource, the service publishing instruction set includes service identifier, service provider, target service object, service start threshold and service quality monitoring index; the generated instruction set is subjected to syntax checking and logical conflict detection, the instructions with format errors, parameters exceeding device thresholds and logical contradictions are eliminated, the necessary fields are supplemented, the instructions are sorted according to the execution time sequence, an ordered executable instruction sequence is formed, the sequence is stored in JavaScript Object Notation (JSON) format, ensuring the readability and analyzability of the instructions.

[0086] d2, encapsulate the executable instruction sequence into a standardized instruction data packet recognizable by each control terminal based on a preset communication protocol, specifically including: presetting the configuration of the communication protocol in advance, the specific process being to investigate the compatibility, transmission efficiency and terminal adaptation range of mainstream industrial-grade communication protocols, combine the terminal type characteristics of target application scenarios such as government service, AI breeding, smart home and digital agriculture, and filter out protocol types with strong stability and wide adaptability, including Message Queuing Telemetry Transport (MQTT) protocol, Hypertext Transfer Protocol version 2 (HTTP / 2) protocol, Modicon Bus Transmission Control Protocol (ModbusTCP) and the like; collect parameters such as hardware models, operating system versions, communication interface specifications and data transmission requirements of various control terminals, establish an adaptation rule library of terminal parameters and protocol characteristics, and determine the optimal communication protocol corresponding to different parameter combinations; solidify the filtered protocol types and the adaptation rule library to the system configuration module to form a preset communication protocol set, supporting automatic matching and calling when subsequent terminals are accessed.

[0087] According to the hardware model, operating system and communication interface capability of the control terminal, the adaptation rule library preset by the system is called to match the corresponding preset communication protocol for different types of control terminals, ensuring the communication compatibility of the protocol and the terminal; the unified structure of the standardized instruction data packet is defined, the data packet is divided into three parts of a packet header, a packet body and a check bit, the packet header includes terminal unique identifier, instruction set type, data packet length, time stamp and the like, the packet body is a binary data stream after protocol adaptation of the executable instruction sequence, the check bit is generated by using a 32-bit Cyclic Redundancy Check 32 (CRC32) algorithm for calculation, and is used for verifying the data transmission integrity; the executable instruction sequence is serialized according to the matched communication protocol, the instruction sequence in JSON format is converted into binary data supported by the corresponding protocol, the packet header information is filled according to the defined data packet structure, the serialized binary instruction data is written into the packet body, and the check bit is calculated and filled; the encapsulated standardized instruction data packet is verified in format, the header field integrity, the consistency of the packet body data length and the identifier length, and the validity of the check bit are checked, and after verification, it is stored according to the terminal type, preparing for subsequent parallel distribution.

[0088] d3, the standardized instruction data packets are distributed in parallel to the corresponding decision nodes and control terminals through a distributed message middleware, specifically including: deploying a distributed message middleware cluster, selecting Kafka as the core middleware, configuring multiple broker nodes to achieve load balancing and fault redundancy, creating a dedicated message queue based on the terminal identifier and decision node number, establishing a one-to-one mapping relationship between the instruction data packet and the message queue; starting the data packet distribution service, reading the classified and stored standardized instruction data packets, obtaining the target terminal and decision node identifier corresponding to each data packet, and delivering the data packet to the corresponding message queue according to the mapping relationship; enabling a parallel distribution mechanism, using the multi-thread processing capability of the middleware to simultaneously push instruction data packets to multiple message queues, and using a flow control strategy during the distribution process to dynamically adjust the data packet sending rate according to the real-time state of the network bandwidth to avoid network congestion; the middleware cluster monitors the distribution process in real time, records the distribution time, target queue, and sending state of each data packet, wherein the sending state is success, failure, or pending retry, and for the data packets that fail to distribute, they are re-pushed according to the preset retry mechanism (up to 3 retries, with a 1-second interval between each retry) until the distribution is successful or the retry limit is reached; the decision nodes and control terminals listen to the corresponding message queues through a long connection and receive the instruction data packets in real time to ensure the rapid transmission and synchronous reception of instructions.

[0089] d4, receiving the state feedback data returned by each control terminal, and dynamically adjusting the distribution strategy of subsequent instructions based on the state feedback data through a feedback control mechanism to complete the real-time scheduling of global resources and the coordinated issuance of control instructions, specifically including: building a state feedback receiving interface based on Transmission Control Protocol / Internet Protocol (TCP / IP), the interface supports high-concurrency data reception, and each control terminal is allocated an independent feedback data receiving port to ensure the ordered reception of feedback data; receiving the state feedback data returned by each control terminal, which includes terminal identifier, instruction reception state, instruction execution progress, current device operating parameters, resource occupation, service execution result, and abnormal alarm information, etc., unpacking and analyzing the received feedback data, extracting key state indicators, and eliminating invalid and duplicate data; constructing a feedback control mechanism, which includes a state evaluation module and a strategy adjustment module, the state evaluation module compares the analyzed state indicators with the preset targets of the coordinated scheduling scheme to evaluate the instruction execution compliance, including device response timeliness, resource allocation rationality, service quality compliance rate, etc., and identifies execution deviations and abnormal problems.

[0090] The fusion Bezier curve dynamic adjustment algorithm optimizes the adjustment logic, and the specific calculation process is as follows: first, collect the key state index data of the last 5 to 10 feedback periods, the historical adjustment strategy and the corresponding execution effect, establish the mapping data set of the adjustment parameter and the state deviation, and provide data support for the Bezier curve parameter configuration; the curve parameter rules are set for different execution deviation scenarios, the curve order is uniformly three-order Bezier curve, the curve shape is determined by four control points, and the control points are selected based on the historical optimal running state data, the current state data, the target state data and the stable threshold data; if the terminal feedback execution delay exceeds the threshold value, first calculate the delay deviation value, that is, the difference between the actual delay and the preset threshold value, select four control points based on the mapping data set, the first control point is the distribution priority corresponding to the historical minimum delay, the second control point is the current distribution priority, the third control point is the target priority calculated according to the delay deviation ratio, and the fourth control point is the priority safety threshold when the system runs stably; draw a three-order Bezier curve according to the coordinate values of the four control points, divide the adjustment period into 10 to 15 equal time steps, calculate the priority discrete value corresponding to each time step on the curve, and distribute the network bandwidth increment according to the curve slope, the greater the slope, the more obvious the delay deviation, and the greater the bandwidth allocation increment, so as to realize the smooth linkage adjustment of priority and bandwidth.

[0091] If the resource occupancy rate is too high, calculate the occupancy deviation value, that is, the difference between the actual occupancy rate and the preset safety threshold, and select four control points, respectively, as the resource allocation ratio corresponding to the historical optimal occupancy rate, the current resource allocation ratio, the target allocation ratio calculated according to the occupancy deviation, and the minimum allocation ratio threshold corresponding to the safe running of resources; draw a three-order Bezier curve, divide the time steps according to the adjustment period, calculate the resource allocation ratio discrete value corresponding to each step, gradually reduce the resource occupancy ratio of non-critical tasks according to the curve trend, ensure the stability of critical task resource supply, and avoid task interruption caused by sudden change of allocation ratio; if an equipment abnormal alarm occurs, select four control points, respectively, as the normal instruction sequence position before the abnormality occurs, the current instruction sequence position, the target position after the emergency instruction is inserted, and the sequence connection position after the normal recovery; draw a two-order Bezier curve, simplify the number of control points to improve the response speed, calculate the smooth transition path of the emergency instruction insertion, determine the instruction execution interval and insertion opportunity of each time step, insert the emergency adjustment instruction at the front end of the instruction sequence according to the curve planning path, and adjust the execution time sequence of the subsequent instructions, to ensure the smooth connection of emergency processing and regular tasks.

[0092] After each adjustment cycle, the adjustment effect is checked based on the new state feedback data, the control point coordinates of the Bezier curve are corrected, the next round of adjustment curve is recalculated, and a dynamic calibration mechanism is formed; the distribution strategy after adjustment calculated according to the Bezier curve is updated to the scheduling configuration center of the distributed message middleware, and the middleware executes subsequent instruction distribution according to the new strategy, and the process of continuously circulating receiving feedback data, evaluating the execution state, and adjusting the distribution strategy through the Bezier curve is realized, so that the real-time dynamic scheduling of global resources and the cooperative publishing of control instructions are realized.

[0093] In the embodiment of the application, the semantic analysis and instruction conversion are performed on the collaborative scheduling scheme to generate a plurality of executable instruction sequences; the abstract decision scheme is converted into structured and concrete execution instructions to strengthen the adaptability of the instructions to devices, resources and services; the standardized instruction data packet is encapsulated based on a preset communication protocol; the instruction transmission format and specification are unified to break the protocol barriers of different control terminals; the instruction data packet is distributed in parallel through the distributed message middleware; the parallel processing advantage of the distributed architecture is exerted to shorten the time consumption of multi-node instruction issuing; the state feedback data is received and the instruction distribution strategy is dynamically adjusted; the closed-loop control mechanism of decision, execution and feedback is constructed to make the instruction distribution adapt to the real-time running state, improve the real-time adaptive ability of the artificial intelligence system, and strengthen the dynamic optimization level of global resource scheduling.

[0094] The above mainly introduces the scheme provided by the embodiments of the application from the perspective of the method. To realize the above functions, it contains the hardware structure and / or software module corresponding to the execution of each function. The technical person skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed in the present text, the application can be realized in the form of hardware or the combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical scheme. The professional technical person skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application.

[0095] The embodiment of the application further provides a perception collaborative decision system based on multi-modal heterogeneous data fusion, as shown in the figure. Figure 2 As shown in the figure, the system includes: A multi-source acquisition module is configured to acquire multi-modal heterogeneous data, and the multi-modal heterogeneous data includes images, texts, time series data and graph structure data; and the multi-modal heterogeneous data is preprocessed to obtain standardized data flow. The fusion analysis module is configured to parse and extract features from the standardized data stream, and perform alignment and coding to obtain a multi-source feature vector; the multi-source feature vector is weighted fused to obtain a preliminary index set; based on the preliminary index set, a predefined key area is grid processed, and dynamic correlation characteristics of each spatial grid unit are analyzed to obtain a spatial dynamic weight matrix; the spatial dynamic weight matrix and the preliminary index set are weighted fused, and a spatial context calibration is performed to obtain an optimized comprehensive state index; and based on the optimized comprehensive state index, a global environment situation awareness map is generated; The collaborative decision module is configured to input the global environment situation awareness map into a pre-trained multi-target collaborative decision model; the multi-target collaborative decision model analyzes resource entities and associated relationships in the global environment situation awareness map to form a multi-target decision feature set; based on the multi-target decision feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme; The execution module is configured to perform instruction parsing and packaging on the comprehensive collaborative scheduling scheme to obtain an executable instruction sequence; and the executable instruction sequence is parallelly distributed to corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative distribution of control instructions.

[0096] Through the description of the above embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional modules is taken as an example for illustration, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the perception collaborative decision system based on multi-modal heterogeneous data fusion is divided into different functional modules to complete all or part of the above described functions.

[0097] It should be noted that the system corresponds to the above method, and all implementation manners in the above method embodiments are applicable to this embodiment, and the same technical effects can be achieved.

[0098] The embodiments of the present application further provide a computer readable storage medium. All or part of the processes in the above method embodiments can be instructed by computer instructions to complete relevant hardware, and the program can be stored in the above computer readable storage medium. When the program is executed, the program can include the processes of the above method embodiments. The computer readable storage medium can be the memory of any of the above embodiments. The above computer readable storage medium can also be an external storage device of the above authentication device, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the above computer readable storage medium can include both the internal storage unit of the above authentication device and the external storage device. The above computer readable storage medium is used to store the above computer program and other programs and data required by the above authentication device. The above computer readable storage medium can also be used to temporarily store data that has been output or will be output.

[0099] The embodiments of the present application further provide a computer program product, which contains a computer program, and when the computer program product runs on a computer, the computer program product makes the computer execute any one of the perception collaborative decision-making methods based on multi-modal heterogeneous data fusion provided in the above embodiments.

[0100] The above is the specific embodiment of the present application. It should be noted that, for those skilled in the art, without departing from the principles of the present application, a number of improvements and refinements can be made, and these improvements and refinements should also be considered as the protection scope of the present application.

Claims

1. A multi-modal heterogeneous data fusion based perception collaborative decision method, characterized in that, The method comprises: Step 100, collecting multi-modal heterogeneous data, the multi-modal heterogeneous data comprising images, texts, time series data and graph structure data; preprocessing the multi-modal heterogeneous data to obtain standardized data streams; Step 200, analyzing and extracting features of the standardized data streams, and performing alignment and encoding to obtain multi-source feature vectors; performing weighted fusion on the multi-source feature vectors to obtain a preliminary index set; based on the preliminary index set, performing grid processing on a predefined key area, and analyzing the dynamic correlation characteristics of each spatial grid unit to obtain a spatial dynamic weight matrix; performing weighted fusion on the spatial dynamic weight matrix and the preliminary index set, and performing spatial context calibration to obtain an optimized comprehensive state index; based on the optimized comprehensive state index, generating a global environment situation awareness map; Step 300, inputting the global environment situation awareness map into a pre-trained multi-target collaborative decision model; the multi-target collaborative decision model forms a multi-target decision feature set by analyzing resource entities and associated relationships in the global environment situation awareness map; based on the multi-target decision feature set, decision optimization is performed to obtain a comprehensive collaborative scheduling scheme; Step 400, performing instruction analysis and packaging on the comprehensive collaborative scheduling scheme to obtain an executable instruction sequence; the executable instruction sequence is parallelly issued to corresponding decision nodes and control terminals through a distributed communication architecture to complete real-time scheduling of resources and collaborative publishing of control instructions.

2. The method of claim 1, wherein, The step 100 comprises: parallelly collecting the multi-modal heterogeneous data from a plurality of heterogeneous data sources, the plurality of heterogeneous data sources comprising at least one of the following: monitoring images, sensor nodes, mobile units and environmental information data sources; performing format analysis and data cleaning on the multi-modal heterogeneous data to remove invalid and redundant data and obtain structured data; performing outlier detection and missing data repair processing on the structured data to obtain reduced data; performing time series alignment and spatial position matching on the reduced data to establish a unified space-time reference to obtain space-time aligned data; performing normalization processing on the space-time aligned data to eliminate dimension differences to obtain the standardized data streams.

3. The method of claim 1, wherein, The step 200 comprises: performing time dimension analysis on the standardized data streams to divide the continuous standardized data streams into an equal-interval time slice sequence; based on the equal-interval time slice sequence, performing spatial registration processing on the multi-modal heterogeneous data in each time slice to establish spatial reference data under a unified coordinate system, the spatial reference data comprising mobile object trajectory data, fixed sensor data and mobile sensor data; based on the mobile object trajectory data in the spatial reference data, determining the object distribution density in each grid unit to obtain a space-time feature matrix; based on the fixed sensor data, constructing a flow feature vector by statistically analyzing the correlation between event intervals and the number of event occurrences per unit time; using the mobile sensor data, performing outlier rejection and mean value calculation on a dynamic parameter sampling sequence to obtain a regional average dynamic feature; The spatiotemporal feature matrix, the traffic feature vector and the area average dynamic feature are aligned in feature dimension, and a multi-source feature vector is obtained through vectorization splicing.

4. The method of claim 3, wherein, The step 200 further includes: Based on the multi-source feature vector, an attention weight matrix of each feature vector is determined through a preset fully connected layer and a normalized exponential softmax function; The attention weight matrix and the multi-source feature vector are weighted and summed to obtain an attention weighted fusion feature; The attention weighted fusion feature is processed by feature dimension reduction and standardization to obtain a preliminary index set; Based on the preliminary index set, a regular grid topology structure covering the area is constructed in combination with a pre-defined key area boundary; Based on the regular grid topology structure, the preliminary index set is mapped to the spatial grid cells to determine the statistical distribution characteristics of the environmental state indicators in each grid cell; Based on the statistical distribution characteristics of the environmental state indicators in each grid cell, the dynamic correlation strength between adjacent grid cells is calculated through spatial autocorrelation analysis; The statistical distribution characteristics of the environmental state indicators in each grid cell and the dynamic correlation strength between adjacent grid cells are integrated to obtain the spatial dynamic weight matrix through weighted fusion calculation.

5. The method of claim 4, wherein, The step 200 further includes: The spatial dynamic weight matrix and the preliminary index set are multiplied element by element to realize feature weighted fusion in the spatial domain and obtain a weighted feature mapping; Based on the weighted feature mapping, the feature propagation of adjacent network nodes is spatially context calibrated to obtain context-aware features; the context-aware features are processed by feature compression and standardization to obtain the optimized comprehensive state indicators; Based on the optimized comprehensive state indicators, feature upsampling and spatial resolution enhancement are performed to obtain a high-resolution feature mapping; Based on the high-resolution feature mapping, a continuous spatial distribution feature field is obtained through a spatial interpolation algorithm; Based on the continuous spatial distribution feature field, the feature field values are mapped to corresponding color space values to obtain a raster image; based on the raster image, the gradient distribution of the feature field is calculated to obtain contour line vector data; the raster image and the contour line vector data are subjected to layer superposition and semantic labeling processing to obtain the global environmental situation awareness map.

6. The method of claim 1, wherein, The step 300 includes: The global environmental situation awareness map is parsed to extract key state indicators representing task processing efficiency, resource utilization balance and service demand satisfaction, and a multi-objective decision feature set is constructed; Based on the multi-objective decision feature set, a global multi-objective optimization function that simultaneously optimizes operation efficiency, resource cost and service quality is constructed, and operation constraints of each functional unit are set. The global multi-objective optimization function is decomposed into a local value function associated with each agent; each agent generates a local strategy based on its local value function and the environment state; all local strategies are integrated by a centralized coordinator, which evaluates its global coordination utility using an attention mechanism, and obtains a non-dominated solution set as an initial coordination strategy set by iterative search within the constraint solution space; The initial coordination strategy set is evaluated and analyzed in multiple dimensions, and the comprehensive utility value of each candidate strategy is calculated; the candidate strategies are sorted according to the comprehensive utility value, and a comprehensive utility ranking is obtained; Based on the comprehensive utility ranking, a dominant strategy is selected from the initial coordination strategy set to obtain the comprehensive coordination scheduling scheme.

7. The method of claim 6, wherein, The step 400 includes: The semantic analysis and instruction conversion of the comprehensive coordination scheduling scheme are performed to generate the executable instruction sequence including the device control instruction set, the resource scheduling instruction set and the service publishing instruction set; Based on the preset communication protocol, the executable instruction sequence is packaged into a standardized instruction data packet recognizable by each control terminal; The standardized instruction data packet is distributed in parallel to the corresponding decision node and control terminal through the distributed message middleware; Receive the state feedback data returned by each control terminal, and dynamically adjust the distribution strategy of subsequent instructions based on the state feedback data through the feedback control mechanism to complete the real-time scheduling of global resources and the coordinated publishing of control instructions.

8. A multi-modal heterogeneous data fusion based perception collaborative decision system, the system implements the method of any one of claims 1 to 7, characterized in that, It includes: A multi-source acquisition module is used to acquire multi-modal heterogeneous data, and the multi-modal heterogeneous data includes images, texts, time series data and graph structure data; The multi-modal heterogeneous data is preprocessed to obtain a standardized data stream; A fusion analysis module is used to analyze and extract features of the standardized data stream, align and encode the standardized data stream, obtain a multi-source feature vector, weight fuse the multi-source feature vector, obtain a preliminary index set, grid process a pre-defined key area based on the preliminary index set, analyze dynamic correlation characteristics of each spatial grid unit, obtain a spatial dynamic weight matrix, weight fuse the spatial dynamic weight matrix and the preliminary index set, and calibrate through a spatial context to obtain an optimized comprehensive state index, and generate a global environment situation awareness map based on the optimized comprehensive state index; A collaborative decision module is used to input the global environment situation awareness map into a pre-trained multi-objective collaborative decision model; the multi-objective collaborative decision model forms a multi-objective decision feature set by analyzing resource entities and correlation relationships in the global environment situation awareness map; Based on the multi-objective decision feature set, decision optimization is performed to obtain a comprehensive coordination scheduling scheme; An execution module is used to perform instruction analysis and packaging on the comprehensive coordination scheduling scheme to obtain an executable instruction sequence; the executable instruction sequence is distributed in parallel to the corresponding decision node and control terminal through a distributed communication architecture to complete real-time scheduling of resources and coordinated publishing of control instructions.

9. A computing device, comprising: It includes: One or more processors; A storage device for storing one or more programs, which when executed by the one or more processors, cause the one or more processors to implement the method of any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a program, which when executed by a processor, implements the method of any one of claims 1-7.

Images