Artificial intelligence based multi-modal environment perception adaptive regulation system

Abstract

The present application relates to the technical field of artificial intelligence, in particular to a multi-modal environment perception adaptive regulation system based on artificial intelligence, the system comprises a model establishing module, a training module, a perception module, a central control module and a mechanical regulation module, the model establishing module is used for establishing and training multiple environment perception neural network models, obtaining network structure and parameters, the training module iteratively adjusts the model, the perception module obtains environment perception information, the central control module issues regulation instructions according to the information, and the mechanical regulation module executes the instructions, the model establishing module comprises data acquisition, processing, input, conversion, splicing and prediction units, uses an attention module and a specialized neural network matrix, and the accuracy of environment identification is significantly improved, and the identification accuracy is improved by 30%-40% compared with traditional general models.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to a multi-modal environment perception adaptive regulation system based on artificial intelligence. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, intelligent environment perception and regulation systems have been widely applied in intelligent buildings, intelligent driving, industrial production and other fields. Traditional environment perception systems usually use single sensors or single algorithm models for environment information collection and processing, which is difficult to cope with complex and variable actual environments. Especially in the scene where multiple environmental factors act together, the single modal perception method is often difficult to accurately identify the environment state, resulting in insufficient regulation accuracy, affecting system performance and user experience.

[0003] The existing environment perception system in the prior art mainly has the following problems: first, the perception mode is single, which is difficult to capture comprehensive environment information; second, the model generalization is insufficient, which is difficult to adapt to different types of environmental changes; third, the perception and regulation are separated, lacking a closed-loop feedback mechanism; fourth, the feature extraction capability is limited, which is difficult to identify complex patterns in the environment. In addition, the existing system generally lacks special model design for specific types of environment, which cannot fully utilize the feature differences of various environments, resulting in poor recognition accuracy and regulation effect. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal environment perception adaptive regulation system based on artificial intelligence, which can realize accurate perception of different types of environments through a multi-environment neural network model, and intelligently regulate based on the perception results, thereby solving the technical problems of single perception mode, insufficient model generalization, and separation of perception and regulation in the prior art.

[0005] The present application discloses a multi-modal environment perception adaptive regulation system based on artificial intelligence, comprising:

[0006] A model establishment module is used to establish a light environment perception neural network model, a radiation environment perception neural network model, a gas environment perception neural network model, a sound environment perception neural network model and a mechanical environment perception neural network model respectively, and to obtain the network structure and parameters of each environment perception neural network model;

[0007] A training module is connected with the model establishment module, and is used to receive the multiple environment perception neural network models transmitted by the model establishment module, and to perform iterative adjustment and parameter optimization on the multiple environment perception neural network models;

[0008] The perception module is connected with the training module, and is used to acquire multi-modal environment perception information of an environment where the to-be-tested device is located, and identify an environment type and a state based on the trained environment perception neural network model;

[0009] The central control module is connected with the perception module, and is used to receive the environment perception information transmitted by the perception module, and issue a targeted regulation instruction according to the environment perception information;

[0010] The mechanical regulation module is connected with the central control module, and is used to receive the regulation instruction issued by the central control module, and regulate the operation of the environment device according to the regulation instruction;

[0011] The model establishing module comprises:

[0012] The data acquisition unit is used to acquire multi-environment training data including optical information and spectral information;

[0013] The data processing unit is used to perform targeted preprocessing on the training data;

[0014] The first input unit is used to acquire an observation image sequence of a to-be-tested environment, and generate four-dimensional tensor input data;

[0015] The conversion unit is used to construct an image conversion network structure based on an attention module, and perform attention enhancement processing on the four-dimensional tensor input data;

[0016] The splicing unit is used to perform multi-modal fusion on the output of the conversion unit and environment state data;

[0017] The prediction unit is used to generate environment perception information of a to-be-tested device in the environment image sequence according to the fusion data in the splicing unit;

[0018] The prediction unit sequentially comprises an adaptive spatial unit, an adaptive spectral unit, a fusion space-time unit and an adaptive frequency spectrum unit, and realizes layer-by-layer feature extraction and time sequence modeling from space, spectrum to frequency spectrum.

[0019] Preferably, the model establishing module acquires environment image sequences and environment states of a light environment, a radiation environment, a gas environment, a sound environment and a mechanical environment in a to-be-tested environment as special training data of an environment perception neural network model corresponding to each environment, and forms an environment-specific neural network matrix, wherein:

[0020] The light environment perception neural network model is used to process light-related environment features, including light intensity, color temperature and light direction;

[0021] The radiation environment perception neural network model is used to process electromagnetic radiation features, especially radiation information in an invisible spectrum;

[0022] The gas environment perception neural network model is used to identify the gas composition and concentration in the environment.

[0023] The sound environment perception neural network model processes sound wave features, including sound frequency, intensity, and spatial distribution.

[0024] The mechanical environment perception neural network model identifies physical features such as mechanical vibration and impact.

[0025] Preferably, the data processing unit includes a first preprocessing module and a second preprocessing module.

[0026] The first preprocessing module is used to remove noise from the environment images in the training data and divide the environment image sequence.

[0027] The second preprocessing module is used to normalize the environment state in the training data and use embedding methods for dimension compression.

[0028] The training data obtained by the data acquisition unit includes environment images and environment states corresponding to one or more test environments in the light environment, radiation environment, gas environment, sound environment, and mechanical environment.

[0029] Preferably, the input of the adaptive spatial unit is a four-dimensional tensor, represented as (m, n, p, q), where m is the number of frames of the training environment image sequence, n is the number of channels of the training environment image sequence, p is the height of the training environment image sequence, and q is the width of the training environment image sequence.

[0030] The adaptive spatial unit uses a spatio-temporal coordination processing mechanism: first, it extracts spatio-temporal features from the four-dimensional tensor, modeling the inter-frame temporal dependence relationship; then it calculates the spatial attention weight to generate the attention distribution in the spatial dimension; finally, it multi-level fuses the temporal and spatial features to output the enhanced features containing temporal and spatial information.

[0031] Preferably, the convolution kernel of the adaptive spatial unit is 1x1, the convolution kernel of the adaptive spectral unit is designed by parameterization, and the convolution kernel of the adaptive spectral unit is designed by parameterization, wherein:

[0032] The adaptive spectral unit uses a 3x3 or 5x5 convolution kernel to capture the spectral features of the local area.

[0033] The adaptive spectral unit uses a 3x1 or 5x1 convolution kernel to adapt to the one-dimensional characteristics of the spectral data, and is used to analyze the frequency distribution characteristics in the environment.

[0034] Preferably, the training module includes a target acquisition unit.

[0035] The target acquisition unit is configured to acquire a loss function between a predicted environment state and an actual environment state, and a loss function between a predicted environment image and an actual environment image;

[0036] The loss functions acquired by the target acquisition unit include a spatial loss function, a spectral loss function, a spectrum loss function and a total loss function, forming a quadruple loss function system;

[0037] The target acquisition unit calculates the spatial loss function according to the error between the prediction result of the prediction unit and the actual environment state, calculates the spectral loss function and the spectrum loss function according to the error between the predicted target extracted by the prediction unit and the actual environment image, and combines the three loss functions to form the total loss function.

[0038] Preferably, the perception module comprises a second input unit, a perception unit and an adjustment unit;

[0039] The second input unit is configured to acquire optical information and spectral information of the device under test in the environment under test, wherein the optical information comprises an environment image sequence of the device under test in the environment under test, and the spectral information comprises an environment state of the device under test in the environment under test;

[0040] The perception unit uses a prediction unit in the trained environment perception neural network model for environment recognition;

[0041] The adjustment unit uses an adaptive region division mechanism, first maintains the global features of the complete image, then performs adaptive region division for semantic perception based on attention weights, ensures that the objects with complete semantics are not segmented, and finally determines the environment type of each region according to the prediction result of the perception unit.

[0042] Preferably, the model establishment module obtains new training data by changing the observation distance, fuses the new training data with the original training data, trains the environment perception neural network model, and obtains the final parameters of the environment perception neural network model when the test result converges;

[0043] During the training process of iteratively adjusting multiple environment perception neural network models, the model establishment module respectively uses observation images and environment states of multiple devices under test for training, thereby improving the device adaptability and generalization ability of the model.

[0044] Preferably, the data acquisition unit comprises an optical intensity analyzer, an optical intensity sensor, a radiometer, an image acquisition device and a sound sensor, forming a multi-sensor cooperative acquisition system;

[0045] The light intensity analyzer is used for providing accurate spectral analysis data, the light intensity sensor is used for monitoring light intensity changes in real time, the radiometer is used for measuring electromagnetic radiation intensity, the image acquisition device is used for collecting visual information of the environment, and the sound sensor is used for collecting sound wave information in the environment.

[0046] As preferred, the attention module of the conversion unit adopts a processing sequence of attention first and convolution later, so as to avoid information loss caused by convolution operation.

[0047] The fusion space-time unit has a true time sequence modeling capability, adopts a long short-term memory network or a gated recurrent unit to model the time sequence dependence between frames, and performs convolution processing on the spatial features of each frame, so as to perform multi-level fusion of the time sequence features and the spatial features.

[0048] The system adopts a multi-scale feature processing strategy, including complete environment feature extraction in a global scale, fine-grained feature recognition in a local scale, and spatial relationship modeling in a context scale, so as to realize the unity of fine environment perception and global understanding through multi-scale feature fusion.

[0049] The beneficial effects of the present application include:

[0050] By constructing an environment-specific neural network matrix, a special model is designed for different environment types, which significantly improves the accuracy of environment recognition, and improves the recognition accuracy by 30%-40% compared with traditional general models.

[0051] A four-dimensional tensor space-time collaborative processing mechanism and a multi-level adaptive perception unit chain design are adopted, so as to realize layer-by-layer feature extraction and enhancement from space, spectrum to frequency, reduce information loss, and improve feature representation capability.

[0052] A fine-grained environment recognition mechanism of adaptive region division is introduced, which solves the problem of environment heterogeneity, supports the recognition of multiple environments in the same scene, and refines the environment perception granularity.

[0053] A quadruple loss function system is designed, which constrains model training from multiple dimensions, balances the learning weights of different dimensional features, and improves the robustness of the model.

[0054] A complete perception-prediction-regulation closed-loop system is constructed, which realizes adaptive response from environment perception to prediction and regulation, and improves the adaptability of the system to environmental changes. BRIEF DESCRIPTION OF DRAWINGS

[0055] Figure 1 It is a whole structure schematic view of the multi-modal environment perception adaptive regulation system based on artificial intelligence of the present application;

[0056] Figure 2A structural schematic diagram of the model establishment module of the present application;

[0057] Figure 3 A structural schematic diagram of the prediction unit of the present application;

[0058] Figure 4 A structural schematic diagram of the adaptive space unit of the present application;

[0059] Figure 5 A workflow schematic diagram of the training module of the present application;

[0060] Figure 6 A workflow schematic diagram of the perception module of the present application. DETAILED DESCRIPTION

[0061] The specific embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0062] As shown in the drawings, Figure 1 The present application provides a multi-modal environment perception adaptive regulation system based on artificial intelligence, which includes a model establishment module 1, a training module 2, a perception module 3, a central control module 4, and a mechanical regulation module 5. These five modules form a complete closed-loop collaborative architecture, realizing the whole-chain adaptive response from environment perception to intelligent regulation.

[0063] The model establishment module 1 is used to establish a light environment perception neural network model, a radiation environment perception neural network model, a gas environment perception neural network model, a sound environment perception neural network model, and a mechanical environment perception neural network model, respectively. This environment-specific neural network matrix design is the core innovation of the present application, which can fully utilize the characteristic differences of various environments compared with traditional single general model. In an embodiment of the present application, for the intelligent building environment control scene, the light environment perception neural network model mainly processes the change of illumination intensity in the range of 0 to 2000 lux, the adjustment of color temperature in the range of 2700K to 6500K, and the recognition of light direction. The radiation environment perception neural network model focuses on processing electromagnetic radiation detection with a wavelength range of 100nm to 1mm, and is particularly suitable for equipment thermal radiation monitoring in industrial environments.

[0064] The training module 2 is connected with the model establishment module 1, and uses an advanced training strategy to iteratively adjust the multiple environment perception neural network models. Preferably, the training module 2 uses the Adam optimizer to update the parameters, with an initial learning rate set to 0.001 and a batch size set to 32. During the training process, when the validation set loss change rate is less than 0.1% for 10 consecutive epochs, the system automatically triggers the early stopping mechanism to avoid overfitting phenomenon.

[0065] The perception module 3 is connected with the training module 2, and is responsible for acquiring multi-modal perception information of the environment in real time. In an embodiment of the present application, the sampling frequency of the perception module 3 is set to 1 Hz, that is, the environmental data is collected once per second, which can capture the dynamic changes of the environment without generating excessive computational burden. For a rapidly changing environment, such as an intelligent driving scene, the sampling frequency can be increased to 10 Hz.

[0066] The central control module 4 receives the environmental perception information from the perception module 3, and issues targeted control instructions according to these information. The response time of the central control module 4 is preferably set to within 200 milliseconds, ensuring that the system can respond to environmental changes in a timely manner. The mechanical control module 5 receives the control instructions from the central control module 4, controls the operation of the environmental equipment, and forms a complete closed-loop control system.

[0067] As shown in Figure 2 The model establishment module 1 includes a data acquisition unit 11, a data processing unit 12, a first input unit 13, a conversion unit 14, a splicing unit 15, and a prediction unit 16. This six-unit chain architecture ensures the gradual refinement and enhancement of data from raw collection to final prediction.

[0068] The data acquisition unit 11 includes a light intensity analyzer, a light intensity sensor, a radiometer, an image acquisition device, and a sound sensor, forming a multi-sensor cooperative acquisition system. Preferably, the sampling accuracy of the light intensity analyzer is set to ±2%, and the wavelength range covers the visible light range of 380-780 nm; the image acquisition device uses a camera with a resolution not less than 1080p, and the frame rate is set to 30 fps; the sampling rate of the sound sensor is set to 44.1 kHz, ensuring that all sound frequencies in the range of 20 Hz to 20 kHz audible to the human ear can be captured. These parameter settings are determined based on human perception range and actual application requirements, which not only ensure data quality but also control system cost.

[0069] The data processing unit 12 includes a first preprocessing module and a second preprocessing module. The first preprocessing module removes noise from the environmental image, preferably using a 5x5 Gaussian filter with a standard deviation of 1.0, which can effectively remove image noise without excessively blurring image details. The second preprocessing module normalizes the environmental state using the minimum-maximum normalization method:

[0070] ,

[0071] where x' is the normalized data value, x is the original data value, min(X) is the minimum value in the data set, and max(X) is the maximum value in the data set. Through this normalization processing, data of different dimensions such as temperature range 15°C to 30°C, humidity range 20% to 90%, and light intensity range 0 to 2000 lux are mapped to the [0, 1] interval.

[0072] The first input unit 13 acquires a sequence of observation images of the environment to be measured and generates four-dimensional tensor input data. In the intelligent building management system, 15 images are preferably set as a sequence unit, corresponding to 15 seconds of environment observation data, and standard RGB images are used, with an image resolution of 256x256. Therefore, the dimensions of the four-dimensional tensor are (15, 3, 256, 256), where 15 is the number of frames in the time dimension, 3 is the number of RGB channels, and 256x256 is the spatial resolution.

[0073] The conversion unit 14 constructs an image conversion network structure based on an attention module, using a processing order of attention first and then convolution. This order design avoids information loss caused by convolution operations and improves the accuracy of the attention mechanism. The conversion unit 14 first calculates the spatial attention weight:

[0074]

[0075] where: is the spatial attention weight, F is the input feature map with dimensions (b, c, h, w), b is the batch size, c is the number of channels, h and w are the height and width of the feature map, respectively, AvgPool is the average pooling operation, MaxPool is the maximum pooling operation, represents a 7x7 convolution layer, represents a sigmoid activation function.

[0076] Then the channel attention weight is calculated:

[0077]

[0078] where: is the channel attention weight, is the weight matrix of the dimension reduction fully connected layer, is the weight matrix of the dimension increasing fully connected layer, and ReLU is the rectified linear unit activation function. In actual implementation, the dimension reduction ratio is set to 16, i.e., if the input channel number is 256, then the channel number is reduced to 16, and then restored to 256.

[0079] ​​The concatenation unit 15 fuses the output of the conversion unit 14 with the environmental state in a multi-modal manner. In the fusion process, special attention needs to be paid to the matching of data types and dimensions. The image feature dimension is , and the environmental state vector dimension is . To achieve effective fusion, the environmental state vector is first converted into a feature map of by a copy expansion operation:

[0080] ,

[0081] Then, the concatenation is performed in the channel dimension:

[0082] ,

[0083] wherein: is the fused feature, the dimension is , is the image feature, is the expanded environmental state feature, is the channel dimension concatenation operation.

[0084] As shown in Figure 3 , the prediction unit 16 is composed of an adaptive spatial unit 161, an adaptive spectral unit 162, a fusion space-time unit 163, and an adaptive frequency spectrum unit 164 in sequence. This four-level design realizes feature extraction and time series modeling from space, spectrum to frequency layer by layer.

[0085] As shown in Figure 4 , the input of the adaptive spatial unit 161 is a four-dimensional tensor , which is specifically in the intelligent building application scenario. The adaptive spatial unit 161 adopts a space-time collaborative processing mechanism, and first performs 3D convolution processing:

[0086] ,

[0087] wherein: is the 3D convolution output feature, is the input four-dimensional tensor, Conv3D is a three-dimensional convolution operation, and the convolution kernel size is set to , that is, a convolution kernel of is used in both the time dimension and the space dimension. Then, the time series attention weight is calculated to model the inter-frame dependency:

[0088] ,

[0089] wherein: is the attention weight of the frame to the frame, is the query vector of the frame, the i-th key vector of the frame, dimension of the key vector, total number of frames, is an exponential function. In practical applications, is preferably set to 64, which can ensure sufficient expression ability without causing excessively high computational complexity. Then, the spatial attention weight is calculated:

[0090] ,

[0091] ,

[0092] wherein: weight value of the i-th row, weight value of the j-th column, activation value of the feature map at position , height of the feature map, width of the feature map.

[0093] The combined spatial attention is calculated by the outer product of the row weight and the column weight:

[0094] ,

[0095] Finally, the temporal feature and the spatial feature are fused at multiple levels:

[0096] ,

[0097] wherein enhanced feature after fusion, element-level multiplication, original input feature.

[0098] The adaptive spectral unit 162 adopts a parameterized design of a convolution kernel, preferably a 3x3 convolution kernel, for capturing the spectral features of a local area. In light environment perception, the 3x3 convolution kernel can effectively identify the spatial distribution pattern of light rays of different wavelengths. The convolution operation is represented as:

[0099]

[0100] wherein: output feature map, input feature map, convolution kernel weight, batch index, output channel index, For input channel index, Spatial location coordinates, This represents the relative offset within the convolution kernel.

[0101] The fusion spatiotemporal unit 163 possesses true temporal modeling capabilities, employing a long short-term memory (LSTM) network to model the temporal dependencies between frames.

[0102] The adaptive spectrum unit 164 employs a 3×1 or 5×1 convolutional kernel to adapt to the one-dimensional characteristics of the spectrum data. For gas environment sensing, a 5×1 convolutional kernel is preferred to capture a longer range of frequency features.

[0103] ,

[0104] The convolution kernel has 5 elements in the vertical direction and only 1 element in the horizontal direction, making it particularly suitable for processing frequency features in spectral data.

[0105] like Figure 5 As shown, training module 2 includes target acquisition unit 21, which constructs a four-fold loss function system. The spatial loss function is calculated using mean squared error:

[0106] ,

[0107] in: Here, N represents the spatial loss function value, and N is the number of samples. This represents the true environmental state value of the i-th sample. Let be the predicted environmental state value for the i-th sample.

[0108] The spectral loss function is calculated using the Fourier transform:

[0109] ,

[0110] in: Here, K represents the spectral loss function value, and K is the total number of frequency components. Let be the weighting coefficient of the k-th frequency component. FFT stands for Fast Fourier Transform, and |·| represents the modulus operation. In practical applications, the weighting coefficient of the low-frequency component is preferably set to 0.8, and that of the high-frequency component is set to 0.2, because the low-frequency component contains the main characteristic information of the environment.

[0111] Spectral loss function calculation for prediction errors at different wavelengths:

[0112] ,

[0113] in: Here, M represents the spectral loss function value, and M represents the total number of spectral components. S represents the weighting coefficient of the m-th spectral component, and S(·) represents the spectral transformation operation.

[0114] The overall loss function is a weighted combination of the three losses:

[0115] The overall loss function is a weighted combination of the three losses:

[0116] ,

[0117] wherein: is the overall loss function value, is the weight coefficient and satisfies . In the initial stage of training, it is preferable to set , and as the training progresses, it can be adjusted to . The training module 2 obtains new training data by changing the observation distance, realizing data enhancement. In the intelligent building scenario, it is preferable to set 3 observation distances: 1 meter, 3 meters, and 5 meters, corresponding to close-range detail features, medium-range overall features, and long-range context features, respectively. The data fusion is represented as:

[0118] ,

[0119] wherein: is the enhanced training data set, is the original training data, are the data collected at distances of 1 meter, 3 meters, and 5 meters, respectively, represents the set union operation.

[0120] In addition, the training module 2 uses multiple devices under test for training, improving the device adaptability of the model. In an embodiment of the present application, 3 cameras with different resolutions are used for data collection: 720p camera data accounts for 30%, 1080p camera accounts for 50%, and 4K camera accounts for 20%. This proportion setting is determined based on the usage frequency of various devices in actual deployment scenarios.

[0121] As shown in Figure 6 , the perception module 3 includes a second input unit 31, a perception unit 32, and an adjustment unit 33. The second input unit 31 acquires optical information and spectral information of the device under test in the test environment, with a sampling frequency consistent with the training stage, both being 1 Hz, ensuring the matching of the data interface.

[0122] The perception unit 32 uses the trained environment perception neural network model for environment recognition. In actual deployment, the perception unit 32 can run on an edge computing device or a cloud server. For intelligent building applications, it is preferable to be deployed on a building central controller, with a processing delay controlled within 100 milliseconds, meeting the real-time requirements.

[0123] The adjusting unit 33 adopts an adaptive region division mechanism to solve the contradiction between fixed grid division and global feature preservation.

[0124] ,

[0125] Among them: is a global feature vector, GlobalPool is a global pooling operation, is a two-dimensional convolution, is a complete input image.

[0126] Then, adaptive division is performed based on the attention weight:

[0127] ,

[0128] Among them: Regions is a region set after division, AdaptiveSegment is an adaptive segmentation function, is an attention weight map.

[0129] In the intelligent building scene, for an image with a resolution of 1024x1024, the system may identify 3 to 8 regions of different sizes, instead of a fixed 4x4 grid. For example, the window area may be identified as a large region, and the various functional areas indoors are identified as several small regions, which not only maintains the integrity of the semantics, but also realizes fine-grained environment recognition.

[0130] The modules communicate with each other through the TCP / IP protocol, and the data transmission frequency is uniformly set to 1Hz to ensure the synchronization of the entire system. The response delay of the central control module 4 is controlled within 200 milliseconds, and the execution delay of the mechanical control module 5 is controlled within 500 milliseconds. The end-to-end delay of the entire system from perception to control is not more than 1 second, meeting the real-time requirements of actual applications.

[0131] Through the above specific embodiments, the multi-modal environment perception adaptive control system based on artificial intelligence can effectively solve the shortcomings of traditional systems in terms of perception accuracy, adaptability and control effect, and provides an advanced technical solution for intelligent buildings, intelligent driving, industrial production and other fields.

[0132] The above describes only the preferred embodiments of the present application, and does not limit the scope of the present application. Any non-essential improvement to the present application based on the technical solutions and concepts of the present application shall be included within the scope of protection of the present application.

Claims

1. An artificial intelligence-based multi-modal environment perception adaptive regulation system, characterized in that, The application relates to a multi-modal environment perception system, comprising: a model establishing module, which is used for establishing light environment perception neural network models, radiation environment perception neural network models, gas environment perception neural network models, sound environment perception neural network models and mechanical environment perception neural network models respectively, and acquiring network structures and parameters of each environment perception neural network model; a training module, which is connected with the model establishing module and is used for receiving multiple environment perception neural network models transmitted by the model establishing module, and performing iterative adjustment and parameter optimization on the multiple environment perception neural network models; a perception module, which is connected with the training module and is used for acquiring multi-modal environment perception information of an environment where a device to be measured is located, and identifying an environment type and a state based on the trained environment perception neural network model; a central control module, which is connected with the perception module and is used for receiving environment perception information transmitted by the perception module, and issuing a targeted regulation instruction according to the environment perception information; a mechanical regulation module, which is connected with the central control module and is used for receiving the regulation instruction issued by the central control module, and regulating the operation of an environment device according to the regulation instruction; wherein the model establishing module comprises: a data acquisition unit, which is used for acquiring multiple environment training data including optical information and spectral information; a data processing unit, which is used for performing targeted pretreatment on the training data; a first input unit, which is used for acquiring an observation image sequence of an environment to be measured, and generating four-dimensional tensor input data; a conversion unit, which is used for constructing an image conversion network structure based on an attention module, and performing attention enhancement processing on the four-dimensional tensor input data; a splicing unit, which is used for performing multi-modal fusion on the output of the conversion unit and environment state data; and a prediction unit, which is used for generating environment perception information of a device to be measured in an environment image sequence according to the fusion data in the splicing unit; the prediction unit is sequentially composed of an adaptive spatial unit, an adaptive spectral unit, a fusion space-time unit and an adaptive frequency spectrum unit, and realizes layer-by-layer feature extraction and time sequence modeling from space, spectrum to frequency spectrum; The model establishing module respectively acquires environment image sequences and environment states of light environment, radiation environment, gas environment, sound environment and mechanical environment in an environment to be measured, as special training data of an environment perception neural network model corresponding to each environment, and forms an environment-specific neural network matrix, wherein the light environment perception neural network model is used for processing environment characteristics related to illumination, including illumination intensity, color temperature and illumination direction; the radiation environment perception neural network model is used for processing electromagnetic radiation characteristics, and the electromagnetic radiation characteristics are radiation information in an invisible spectrum; the gas environment perception neural network model is used for identifying gas components and concentrations in an environment; the sound environment perception neural network model is used for processing sound wave characteristics, including sound frequency, intensity and spatial distribution; and the mechanical environment perception neural network model is used for identifying mechanical vibration and impact physical characteristics. The convolution kernel of the adaptive spatial unit is 1*1, the convolution kernel length and width of the adaptive spectral unit are parameterized and designed, and the convolution kernel length and width of the adaptive frequency unit are parameterized and designed, wherein: the adaptive spectral unit adopts a 3*3 or 5*5 convolution kernel to capture the spectral features of the local area; the adaptive frequency unit adopts a 3*1 or 5*1 convolution kernel to adapt to the one-dimensional characteristics of the spectral data, and is used to analyze the frequency distribution characteristics in the environment; The training module comprises a target acquisition unit; the target acquisition unit is used to acquire a loss function between a predicted environment state and a real environment state, and a loss function between a predicted environment image and a real environment image; the loss functions acquired by the target acquisition unit comprise a spatial loss function, a spectral loss function, a frequency loss function and a total loss function, forming a quadruple loss function system; the target acquisition unit calculates the spatial loss function according to the error between the prediction result of the prediction unit and the real environment state, calculates the spectral loss function and the frequency loss function according to the error between the predicted target extracted by the prediction unit and the real environment image, and combines the three loss functions to form the total loss function. The attention module of the conversion unit adopts a processing sequence of attention first and then convolution, avoiding information loss caused by convolution operation; the fusion space-time unit has a true time sequence modeling capability, uses a long short-term memory network or a gated recurrent unit to model the time sequence dependence between frames, and simultaneously performs convolution processing on the spatial features of each frame, to perform multi-level fusion of the time sequence features and the spatial features; the system adopts a multi-scale feature processing strategy, including complete environment feature extraction in a global scale, fine-grained feature recognition in a local scale, and spatial relationship modeling in a context scale, to realize the unity of fine environment perception and global understanding through multi-scale feature fusion.

2. The artificial intelligence based multi-modal environment perception adaptive regulation system according to claim 1, wherein, The data processing unit comprises a first preprocessing module and a second preprocessing module; the first preprocessing module is used to remove noise from the environment images in the training data and divide the environment image sequence; the second preprocessing module is used to perform normalization processing on the environment states in the training data, and perform dimension compression by using an embedding method; the training data acquired by the data acquisition unit comprises environment images and environment states corresponding to a to-be-tested environment in one or more of a light environment, a radiation environment, a gas environment, a sound environment and a mechanical environment. 3.The artificial intelligence based multi-modal environment perception adaptive regulation system according to claim 1, wherein, The input of the adaptive spatial unit is a four-dimensional tensor, represented as (m, n, p, q), wherein m is the frame number of the training environment image sequence, n is the channel number of the training environment image sequence, p is the height of the training environment image sequence, and q is the width of the training environment image sequence; the adaptive spatial unit adopts a space-time collaborative processing mechanism: firstly, space-time feature extraction is performed on the four-dimensional tensor to model the time sequence dependence between frames; then, spatial attention weights are calculated to generate an attention distribution in the spatial dimension; finally, multi-level fusion is performed on the time sequence features and the spatial features to output enhanced features containing time sequence and spatial information. 4.The artificial intelligence based multi-modal environment perception adaptive regulation system according to claim 1, wherein, The perception module comprises a second input unit, a perception unit and an adjustment unit; the second input unit is used to acquire optical information and spectral information of the device to be tested in the environment to be tested, wherein the optical information comprises an environment image sequence of the device to be tested in the environment to be tested, and the spectral information comprises an environment state of the device to be tested in the environment to be tested; the perception unit adopts a prediction unit in a trained environment perception neural network model to perform environment recognition; the adjustment unit adopts an adaptive region division mechanism, first maintains global features of a complete image, then performs adaptive region division for semantic perception based on attention weights, ensures that a semantic complete object is not segmented, and finally determines an environment type of each region according to a prediction result of the perception unit. 5.The artificial intelligence based multi-modal environment perception adaptive regulation system according to claim 1, wherein, The model establishing module obtains new training data by changing an observation distance, fuses the new training data with original training data, trains the environment perception neural network model, and obtains final parameters of the environment perception neural network model when a test result tends to converge; in the training process of iterative adjustment of the multiple environment perception neural network models, the model establishing module respectively uses observation images and environment states of multiple devices to be tested for training, and improves device adaptability and generalization capability of the model. 6.The artificial intelligence based multi-modal environment perception adaptive regulation system according to claim 1, wherein, The data acquisition unit comprises a light intensity analyzer, a light intensity sensor, a radiometer, an image acquisition device and a sound sensor, and constitutes a multi-sensor cooperative acquisition system; the light intensity analyzer is used to provide accurate spectral analysis data, the light intensity sensor is used to monitor real-time light intensity changes, the radiometer is used to measure electromagnetic radiation intensity, the image acquisition device is used to acquire visual information of the environment, and the sound sensor is used to acquire sound wave information in the environment.

Images