A flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion

Abstract

The application relates to the technical field of flight simulator cockpit monitoring, and discloses a flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion, which comprises the following steps: collecting temperature and humidity, gas concentration and display screen image data of a multi-source sensor, recording a time stamp and a position coordinate, establishing a coordinate mapping relationship based on display screen corner points, and matching the data to a cockpit partition and a display area; after data drift correction, calculating a dynamic weight according to an error rate and data timeliness, weighting and fusing the same data in the partition, obtaining a regional effective value and a regional confidence degree; identifying a display screen abnormal type and a target position, inputting the regional effective value and the regional confidence degree into a cross-modal correlation rule library, adjusting a matching strength, calculating a continuous risk value and mapping an early warning level, generating a linkage instruction, sending the linkage instruction to a cockpit execution system and pushing the linkage instruction to an operation and maintenance terminal. The application realizes unified fusion of cockpit multi-source heterogeneous data and intelligent early warning of composite faults.

Description

Technical Field ,

[0009] , ,

[0008] , ,

[0007] , ,

[0006] , ,

[0010] ,

[0001] The present invention relates to the technical field of flight simulator cockpit monitoring, and particularly relates to a flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion. Background Art

[0002] Flight simulators and real aircraft cockpits generally use decentralized single sensors to complete environmental monitoring, separately arranging independent temperature and humidity probes, smoke gas detectors, relying on single-point data to achieve over-limit alarms, without the ability of cross-device and cross-parameter linkage analysis. This problem also exists in the field of flight simulator cockpits, and the long-term high-frequency operation of simulators puts higher requirements on equipment reliability and rapid fault location.

[0003] The existing technologies mainly have the following defects:

[0004] The monitoring error of single sensors is large, and the false and missed alarms are relatively high. The temperature drift of traditional single-point temperature and humidity sensors can reach ±2°C, and the humidity error is ±3%RH; the alarm delay of ordinary smoke detectors is greater than 5s, and the failure of a single component is extremely likely to cause missed alarms, and the overall alarm false alarm rate is about 15%.

[0005] Data islands, without the ability of cross-parameter correlation analysis. Environmental parameters such as temperature and humidity and harmful gases, equipment operating states such as DU screen alarms and key failures are isolated from each other, and it is impossible to judge the root cause of faults according to multi-condition combinations. For example, the condensation failure of the display screen induced by a high-humidity environment cannot be automatically associated and identified, and the fault location is cumbersome and time-consuming.

[0006] The real-time performance cannot meet the aviation safety specifications. The aviation cockpit safety control requires a millisecond-level response to fault warnings. The existing discrete monitoring systems are restricted by data asynchronization and lack of unified fusion operations, and cannot simultaneously achieve high-precision acquisition and rapid alarm within 2 seconds.

[0007] The weights are fixed and the adaptability is poor. Traditional multi-parameter monitoring uses fixed weighted parameters and cannot dynamically adjust the data adoption weights according to the real-time working errors and aging degrees of sensors. After the sensors age and drift, the system accuracy drops rapidly.

[0008] The above defects make it difficult for the existing technologies to meet the comprehensive requirements of flight simulator cockpits for monitoring accuracy, real-time performance, and composite fault location.

[0009] Therefore, the present application provides a flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion to solve the above technical problems. Summary of the Invention

[0010] The purpose of this invention is to provide a flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion, in order to solve the technical problems in the prior art where temperature and humidity data, gas concentration data and display screen images in flight simulator cockpits lack unified spatiotemporal correlation, sensor data drift and timeliness changes are difficult to be incorporated into fusion calculations, and environmental data and display screen anomalies are difficult to perform zoned joint early warning.

[0011] To address the aforementioned technical problems, this invention provides a flight simulator cockpit monitoring method based on multi-sensor heterogeneous fusion, comprising: Acquire temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and record timestamps and location coordinates; Establish coordinate mapping relationships using the corner points of the display screen, and match temperature and humidity data and gas concentration data to the cockpit partitions and display areas; Drift corrections are applied to temperature and humidity data and gas concentration data. The error rate between observed and corrected values ​​and the timeliness of the data are calculated. Dynamic weights are then calculated based on the error rate and the timeliness of the data. The data of the same type in the partition are weighted and merged according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and regional confidence level. Identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image of the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate the continuous risk value and map the warning level; Based on the warning level and anomaly type, a linkage instruction is generated and sent to the ventilation system and lighting alarm system. The warning level, continuous risk value, anomaly type, and target location are also sent to the operation and maintenance terminal.

[0012] In some specific embodiments, a coordinate mapping relationship is established using the corner points of the display screen to match temperature and humidity data and gas concentration data to the cockpit partition and display area, further including: The edge computing device is set as the master clock, and a synchronization clock is sent to the acquisition nodes of the temperature and humidity sensor, gas sensor and industrial camera. The timestamps of the temperature and humidity data, the gas concentration data and the display screen image are recorded based on hardware timestamps. Using the geometric center of the cockpit as the origin of the world coordinate system, the three-dimensional physical coordinates of each temperature and humidity sensor and gas sensor are measured. The industrial camera is calibrated using a checkerboard pattern, and the mapping relationship from world coordinates to image coordinates is calculated based on the corner points of the display screen. Based on the three-dimensional physical coordinates, the mapping relationship, and the timestamp, the temperature and humidity data and gas concentration data are associated with the cockpit partition and the display area of ​​the screen image.

[0013] In some specific embodiments, drift correction of temperature and humidity data and gas concentration data further includes: An independent Kalman filter is maintained for each channel of temperature and humidity data and gas concentration data. The sensor observations are input into the Kalman filter, and the drift-corrected estimated values ​​are output. The relative deviation between the sensor observation and the estimated value is calculated as the error rate, and the timeliness of the data is determined based on the delay between the timestamp and the current processing time. When the error rate is lower than the preset minimum error rate, the preset minimum error rate is used to replace the error rate. When the error rate is not lower than the preset minimum error rate, the error rate is retained; The dynamic weight is formed by multiplying the reciprocal of the error rate and the timeliness of the data by preset coefficients and then adding them together.

[0014] In some specific embodiments, the weighted fusion of similar data in partitions according to dynamic weights further includes: Multiply each temperature and humidity data or each gas concentration data in the same cockpit zone by the dynamic weight of each sensor. The sum of the multiplication results is divided by the sum of the dynamic weights to obtain the effective value of the temperature and humidity of the region or the effective value of the gas in the region. The dynamic weights participating in the weighted fusion are summed, and the sum of the dynamic weights forms the region confidence score.

[0015] In some specific embodiments, adjusting the matching strength of the rule conditions further includes: Set the first preset reliability threshold to be less than the second preset reliability threshold; Based on the data timestamp and the cockpit partition to which it belongs, the effective values ​​of temperature and humidity in the region, the effective values ​​of gas in the region, the anomaly type of the display screen image, and the confidence level of the region are matched with the rule conditions item by item; When the confidence level of the region is lower than the first preset confidence threshold, the rule condition is marked as a weak match. When all the remaining rule conditions of the same rule are met, the weak match result is included in the calculation of the continuous risk value. When not all the remaining rule conditions of the same rule are met, the rule warning is not triggered. When the confidence level of the region is higher than the second preset confidence threshold and the rule condition meets the threshold range, the rule condition is marked as a strong match, the weight of the strong match result is adjusted according to the confidence level of the region and included in the calculation of continuous risk value; When the confidence level of the region is between the first preset confidence threshold and the second preset confidence threshold, the rule matching result is included in the calculation of the continuous risk value according to the normal matching state.

[0016] In some specific embodiments, identifying the anomaly type and target location of the display image further includes: Convert the images captured by the industrial camera on the display screen from the three primary color space to the hue, saturation, and lightness color space; Based on the target detection model, target recognition is performed on the red alarm area, amber alarm area, screen flicker area, noise area and display color deviation area in the converted display screen image.

[0017] In some specific embodiments, calculating continuous risk values ​​and mapping early warning levels further includes: Based on the threshold range of each rule condition, the matching results for the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and abnormal image types on the display screen are determined respectively; The weight of each matching result is adjusted according to the regional confidence level, and the matching results of the same rule are weighted and calculated to obtain the continuous risk value; The continuous risk value is compared with the first risk threshold and the second risk threshold in the rule base, and the normal level, the first-level early warning level or the second-level emergency alarm level is output. After the continuous risk values ​​are mapped to the warning level, the rule warning of the second-level emergency alarm level is written into the priority position of the warning output queue; Ventilation control commands and lighting alarm commands are generated according to the aforementioned warning level.

[0018] In some specific embodiments, the updating of dynamic weights further includes: The detection sensor checks whether the continuously output data from the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions. The preset abnormal conditions are that the error rate of the continuously output data is higher than the preset error threshold or the timeliness of the data is lower than the preset timeliness threshold. When the data continuously output by the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions, the corresponding sensor is deweighted in the dynamic weight calculation, and a self-test command is sent to the temperature and humidity sensor or the gas sensor that meets the preset abnormal conditions. Upon receiving the weight reduction request, the dynamic weight of the corresponding sensor is reduced. When the self-test result indicates that the sensor is in normal condition, the dynamic weight is updated according to the error rate and the data timeliness. When the self-test result indicates that the sensor is in an abnormal state, the reduced dynamic weight is maintained and a sensor abnormality alarm is output. When the data continuously output by the temperature and humidity sensor or the gas sensor does not meet the preset abnormal conditions, the dynamic weight is updated according to the error rate and the timeliness of the data.

[0019] In some specific embodiments, acquiring temperature and humidity data, gas concentration data, and display screen images further includes: Temperature and humidity sensors are installed in the cockpit instrument control area, the ceiling area, the electrical equipment cabinet area, and the rear auxiliary area. The smoke sensor and the infrared gas sensor are arranged side by side on the center console and the top of the cockpit. Two industrial cameras were mounted on the upper left and upper right rear corners of the cockpit, respectively, to capture images of the display screen with a cross field of view.

[0020] Based on the same concept, the present invention also provides a flight simulator cockpit monitoring system based on multi-sensor heterogeneous fusion, comprising: The multi-source data synchronous acquisition module is configured to acquire temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and record timestamps and location coordinates. The spatiotemporal alignment and coordinate mapping module is configured to establish a coordinate mapping relationship based on the corner points of the display screen, and match the temperature and humidity data and gas concentration data to the cockpit partition and display area. The drift correction and dynamic weight calculation module is configured to perform drift correction on temperature and humidity data and gas concentration data, calculate the error rate between observed values ​​and corrected values ​​as well as data timeliness, and calculate dynamic weights based on the error rate and data timeliness. The same type of data weighted fusion module is configured to weight and fuse similar data in the partition according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, regional gas effective values ​​and regional confidence scores. The visual recognition and cross-modal risk mapping module is configured to identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image in the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate continuous risk values ​​and map the warning level; The early warning classification and linkage output module is configured to generate linkage instructions based on the early warning level and anomaly type, send the linkage instructions to the ventilation system and lighting alarm system, and send the early warning level, continuous risk value, anomaly type and target location to the operation and maintenance terminal.

[0021] Compared with existing technologies, its advantages are as follows: This invention discloses a flight simulator cockpit monitoring method and system based on multi-sensor heterogeneous fusion. By writing timestamps and location coordinates into temperature and humidity data, gas concentration data and display screen images, and establishing a coordinate mapping relationship between cockpit partitions and display areas, the environmental data and abnormal information on the display screen that enter the rule matching have a unified time and space attribution.

[0022] By performing drift correction on each channel of temperature and humidity data and gas concentration data, and forming dynamic weights based on error rate and data timeliness, the current deviation and delay state of sensor data are incorporated into the data fusion process of the same type.

[0023] By generating effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and regional confidence levels according to dynamic weights, and classifying weak matching, strong matching, and ordinary matching states based on regional confidence levels, the matching strength of the rule conditions changes with the regional data state.

[0024] By inputting the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, abnormal image types on the display screen, and target locations into the cross-modal association rule base, continuous risk values, early warning levels, ventilation control instructions, and lighting alarm instructions are generated.

[0025] By performing a weight reduction request and self-test on sensors that meet preset abnormal conditions, the self-test results are returned to the dynamic weight calculation process, and a sensor abnormality alarm is generated. Attached Figure Description

[0026] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart illustrating some specific embodiments of the flight simulator cockpit monitoring method based on multi-sensor heterogeneous fusion of the present invention; Figure 2 This is a schematic diagram of the structure of a flight simulator cockpit monitoring system based on multi-sensor heterogeneous fusion in some specific embodiments of the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms "a," "the," and "the" as used in the embodiments of this application are also intended to include the plural forms, unless the context clearly indicates otherwise, and "multiple" generally includes at least two.

[0029] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0030] It should be understood that although the terms first, second, third, etc., may be used in the embodiments of this application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, first may also be referred to as second without departing from the scope of the embodiments of this application, and similarly, second may also be referred to as first.

[0031] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to detection.” Similarly, depending on the context, the phrases “if determination” or “if detection (of the stated condition or event)” can be interpreted as “when determination” or “in response to determination” or “when detection (of the stated condition or event)” or “in response to detection (of the stated condition or event).”

[0032] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that an article or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such an article or device. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the article or device that includes said element.

[0033] It should be noted that any symbols and / or numbers present in the specification that are not marked in the accompanying drawings are not reference numerals.

[0034] Reference Figure 1 A method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion, comprising: S101 acquires temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and records timestamps and location coordinates. S102 establishes a coordinate mapping relationship using the corner points of the display screen, and matches the temperature and humidity data and gas concentration data to the cockpit partition and display area; S103 performs drift correction on temperature and humidity data and gas concentration data, calculates the error rate between observed values ​​and corrected values ​​as well as data timeliness, and calculates dynamic weights based on error rate and data timeliness. S104, weighted and fused similar data in the partition according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and regional confidence level; S105, identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image of the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate the continuous risk value and map the warning level; S106 generates linkage instructions based on the warning level and anomaly type, sends the linkage instructions to the ventilation system and lighting alarm system, and sends the warning level, continuous risk value, anomaly type and target location to the operation and maintenance terminal.

[0035] To provide a clearer explanation, the steps in the embodiments of the present invention are described in detail below: In this embodiment, the following parameter symbols are defined: Road sensor in the first The observation values ​​at each acquisition time are denoted as: ;No. Road sensor in the first The drift correction value at each acquisition time is denoted as: ;No. The error rate of the road sensor is denoted as: ;No. The timeliness of data from road sensors is denoted as: ;No. The dynamic weights of the road sensors are denoted as: Within the same cockpit section The drift correction value for similar sensors is denoted as: Valid values ​​for the same type of data in a region are denoted as: Regional confidence level, denoted as: In the cross-modal association rule, the first The matching results of each rule condition are denoted as: Continuous risk value, denoted as: .

[0036] S101 acquires temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and records timestamps and location coordinates. Furthermore, temperature and humidity sensors are installed in the cockpit instrument control area, the ceiling area, the electrical equipment cabinet area, and the rear auxiliary area; The smoke sensor and the infrared gas sensor are arranged side by side on the center console and the top of the cockpit. Two industrial cameras were mounted on the upper left and upper right rear corners of the cockpit, respectively, to capture images of the display screen with a cross field of view.

[0037] In this embodiment, a PT100 platinum resistance temperature sensor can be used for temperature detection, and a capacitive humidity sensor can be used for humidity detection. In an integrated data acquisition method, an SHT45 temperature and humidity sensor can also be used to simultaneously acquire temperature and humidity data. The temperature detection accuracy can reach ±0.5℃, and the humidity detection accuracy can reach ±1%RH.

[0038] Temperature and humidity sensors are deployed in zones based on the distribution of heat sources, airflow paths, and temperature field gradients within the cockpit. The deployment density of temperature and humidity sensors can be no less than 1 sensor per square meter.

[0039] Taking a flight simulator with a cockpit area of ​​4㎡ to 6㎡ as an example, eight temperature and humidity monitoring points can be set up. Among them, three temperature and humidity monitoring points are set up in the instrument control area, two in the top ceiling area, two in the electrical equipment cabinet area, and one in the rear auxiliary area.

[0040] Temperature and humidity collection points in the instrument control area are used to monitor the temperature and humidity status near the display screen, control panel, and electronic equipment; temperature and humidity collection points in the top ceiling area are used to monitor the accumulation of hot air in the upper part of the cockpit; temperature and humidity collection points in the electrical equipment cabinet area are used to monitor the heat dissipation status of electronic equipment; and temperature and humidity collection points in the rear auxiliary area are used to supplement and cover the monitoring area formed by seats, bulkheads, or equipment.

[0041] Gas sensors can include electrochemical smoke sensors and infrared gas sensors based on non-spectral infrared detection. Electrochemical smoke sensors are used to detect smoke particles or CO produced by combustion, while infrared gas sensors are used to detect at least one of CO and CO2.

[0042] Parallel deployment refers to placing smoke sensors and infrared gas sensors in the same cockpit compartment, so that smoke concentration data and gas concentration data correspond to the same local spatial location.

[0043] In one embodiment, two electrochemical smoke sensors and two infrared gas sensors are provided, wherein one electrochemical smoke sensor and one infrared gas sensor are located on the center console, and the other electrochemical smoke sensor and the other infrared gas sensor are located on the top of the cockpit.

[0044] The smoke sensor has a smoke detection limit of 0.001%obs / m and a response time of no more than 2 seconds. The smoke concentration unit %obs / m represents the percentage shading rate per meter of optical path.

[0045] The visual monitoring system includes two industrial cameras. These cameras are mounted at the upper left and upper right rear corners of the cockpit, respectively, allowing their fields of view to complement each other. Each display screen is captured entirely by at least one industrial camera.

[0046] The industrial camera can be positioned at a downward angle of 30° to 45° relative to the display screen, thereby reducing obstruction of the display screen by the driver, seat, and control components. The industrial camera can be a 1280×1024 pixel resolution camera with a frame rate of 30 FPS, a lens focal length of 6mm, and an aperture of F1.4.

[0047] The data frames output by each temperature and humidity sensor and gas sensor should include at least the sensor identifier, sensor type, sensor observation value, acquisition timestamp, and sensor location coordinates. The image data output by the industrial camera should include at least the camera identifier, display image, image acquisition timestamp, and camera installation location.

[0048] Temperature and humidity data, gas concentration data, and display images are transmitted to the edge computing device via wired or wireless communication links. The edge computing device can be an NVIDIA Jetson AGX Orin, or other edge computing devices capable of performing multi-sensor data fusion and target detection model inference.

[0049] S102 establishes a coordinate mapping relationship using the corner points of the display screen, and matches the temperature and humidity data and gas concentration data to the cockpit partition and display area; Furthermore, the edge computing device is set as the master clock, and a synchronization clock is sent to the acquisition nodes of the temperature and humidity sensor, gas sensor and industrial camera, and the timestamps of the temperature and humidity data, the gas concentration data and the display screen image are recorded based on hardware timestamps; Using the geometric center of the cockpit as the origin of the world coordinate system, the three-dimensional physical coordinates of each temperature and humidity sensor and gas sensor are measured. The industrial camera is calibrated using a checkerboard pattern, and the mapping relationship from world coordinates to image coordinates is calculated based on the corner points of the display screen. Based on the three-dimensional physical coordinates, the mapping relationship, and the timestamp, the temperature and humidity data and gas concentration data are associated with the cockpit partition and the display area of ​​the screen image.

[0050] In this embodiment, the edge computing device can use the IEEE 1588 PTPv2 precise time protocol as the time synchronization method, and the edge computing device is set as the master clock. Each acquisition node is configured with a communication chip that supports hardware timestamps, and receives the synchronization information sent by the master clock through Ethernet PTP messages.

[0051] Each acquisition node corrects its local clock based on the received synchronization information and writes a hardware timestamp when temperature and humidity data, gas concentration data, and display images are generated. The hardware timestamp allows data from different acquisition frequencies and transmission links to be converted to a unified timeline.

[0052] In one implementation, the time synchronization error of each acquisition node is no greater than ±1ms.

[0053] With the geometric center of the cockpit as the origin of the world coordinate system Establish a world coordinate system - . The shaft can be set along the length of the cockpit. The shaft can be set along the width of the cockpit. The shaft can be set along the height of the cockpit.

[0054] During the installation phase, a laser rangefinder was used to measure the temperature, humidity, and gas sensors relative to the world coordinate origin. The system obtains the three-dimensional physical coordinates and writes the sensor identifier, sensor type, three-dimensional physical coordinates, and cockpit partition to which it belongs into the system configuration file.

[0055] For industrial cameras, the checkerboard calibration method is used to obtain the intrinsic parameters and distortion parameters of the industrial camera, and then the extrinsic parameters of the industrial camera are calculated using the four corner points of the display screen, the seat marking points, or other reference markers in known spatial positions in the cockpit.

[0056] World coordinates Image pixels The mapping relationship between them can be represented as: ; in, As a scale factor, Let be the coordinates of the image pixels in the horizontal direction of the image. These are the coordinates of the image pixels in the vertical direction of the image. This is the intrinsic parameter matrix of an industrial camera. This is the rotation matrix from the world coordinate system to the industrial camera coordinate system. Let be the translation vector from the world coordinate system to the industrial camera coordinate system. For world coordinates at Coordinates along the axial direction, For world coordinates at Coordinates along the axial direction, For world coordinates at Coordinates along the axis.

[0057] By substituting the four corner points of the display screen into the above mapping relationship, the display area of ​​the corresponding display screen can be determined in the image captured by the industrial camera. The display area can be represented by a polygon enclosed by the four corner points.

[0058] Temperature and humidity data, gas concentration data, and display screen images are grouped according to preset time windows. The timestamps of the numerical data are used as the basis for grouping. Image acquisition timestamp satisfy: At that time, the corresponding numerical data and the display image are determined to be data within the same time window. Among them, For timestamps of numerical data, For image acquisition timestamps, This is a preset time matching window.

[0059] The cockpit partition to which the numerical data belongs is determined based on the three-dimensional physical coordinates of the temperature and humidity sensors and the gas sensor, and the display area in the display screen image and its corresponding cockpit partition are determined based on the coordinate mapping relationship established by the corner points of the display screen.

[0060] Establish a spatiotemporal correlation between temperature and humidity data, gas concentration data, and display screen images that are in the same time window and correspond to the same cockpit partition.

[0061] S103 performs drift correction on temperature and humidity data and gas concentration data, calculates the error rate between observed values ​​and corrected values ​​as well as data timeliness, and calculates dynamic weights based on error rate and data timeliness. Furthermore, drift corrections are performed on the temperature and humidity data and gas concentration data, including: An independent Kalman filter is maintained for each channel of temperature and humidity data and gas concentration data. The sensor observations are input into the Kalman filter, and the drift-corrected estimated values ​​are output. The relative deviation between the sensor observation and the estimated value is calculated as the error rate, and the timeliness of the data is determined based on the delay between the timestamp and the current processing time. When the error rate is lower than the preset minimum error rate, the preset minimum error rate is used to replace the error rate. When the error rate is not lower than the preset minimum error rate, the error rate is retained; The dynamic weight is formed by multiplying the reciprocal of the error rate and the timeliness of the data by preset coefficients and then adding them together.

[0062] Furthermore, the dynamic weight updates include: The detection sensor checks whether the continuously output data from the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions. The preset abnormal conditions are that the error rate of the continuously output data is higher than the preset error threshold or the timeliness of the data is lower than the preset timeliness threshold. When the data continuously output by the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions, the corresponding sensor is deweighted in the dynamic weight calculation, and a self-test command is sent to the temperature and humidity sensor or the gas sensor that meets the preset abnormal conditions. Upon receiving the weight reduction request, the dynamic weight of the corresponding sensor is reduced. When the self-test result indicates that the sensor is in normal condition, the dynamic weight is updated according to the error rate and the data timeliness. When the self-test result indicates that the sensor is in an abnormal state, the reduced dynamic weight is maintained and a sensor abnormality alarm is output. When the data continuously output by the temperature and humidity sensor or the gas sensor does not meet the preset abnormal conditions, the dynamic weight is updated according to the error rate and the timeliness of the data.

[0063] In this embodiment, an independent Kalman filter instance is maintained for each temperature and humidity sensor and each gas sensor. The state equation of the Kalman filter can be expressed as: ; in, For the first The actual sensor value at any given time. Here is the state transition matrix. For the first The actual sensor value at any given time. To control the input matrix, For external control input, This is process noise.

[0064] The observation equation of the Kalman filter can be expressed as: ; in, For the first Sensor observations at time [time] For the observation matrix, For the first The actual sensor value at any given time. To observe noise.

[0065] For single-channel temperature, humidity, or gas concentration sensors It can be a one-dimensional state variable corresponding to the physical quantity. When the sensor output changes slowly between adjacent acquisition times, It can be set to , It can be set to , It can be set to .

[0066] Process noise Follow the mean variance is The distribution This can be obtained statistically from long-term operating data of the corresponding sensors. Observation noise. Follow the mean variance is The distribution It can be determined by the square of the sensor's factory-calibrated accuracy value.

[0067] Kalman filter receives the first Sensor observations at time And output the estimated value Estimated value As the correction value after drift correction.

[0068] No. Error rate of road sensor It can be represented as: ; in, For the first Error rate of road sensor For the first Road sensor in the first The observed value at time, For the first The estimated value of the Kalman filter output corresponding to the road sensor.

[0069] When the sensor observation value is close to In such cases, the sensor range normalized value or a preset non-zero reference value can be used to replace the denominator to avoid the denominator being zero. .

[0070] The preset minimum error rate is denoted as In one implementation, It can be set to When the calculated error rate Less than season: ; in, For the first Error rate of road sensor The preset minimum error rate.

[0071] When the calculated error rate Not less than At that time, retain the calculated error rate. Error rate It can be limited to ~ Within the range.

[0072] Data timeliness is determined based on the delay between the current processing time and the data timestamp. The [number]th... The data delay of the road sensor is denoted as Then the timeliness of the data It can be represented as: ; in, For the first Timeliness of data from road sensors This is the lower limit for data timeliness. For the first Data delay of road sensors.

[0073] In one implementation, It can be set to .

[0074] When the data is the latest data Pick Data latency increases by 100%. , reduce When the timeliness of the calculated data is lower than hour, Keep as .

[0075] No. Dynamic weights of road sensors The following formula can be used for calculation: ; in, For the first Dynamic weights of road sensors For the error rate term coefficient, For the first Error rate of road sensor For timeliness coefficients, For the first The timeliness of data from road sensors.

[0076] In one implementation, Pick , Pick ,Right now: ; in, For the first Dynamic weights of road sensors For the first Error rate of road sensor For the first The timeliness of data from road sensors.

[0077] Error rate The lower the value, the smaller the deviation between the sensor observation and the drift correction value, and the higher the corresponding dynamic weight. The larger the value, the lower the data latency and the faster the data timeliness. The higher the value, the higher the corresponding dynamic weight. The larger.

[0078] The Kalman filter is used to pre-correct sensor drift and provide an estimate for error rate calculation. Error rate and data timeliness are further used to calculate dynamic weights. The processing order for these two methods is as follows: Kalman filter correction; Calculate the error rate and data timeliness ; Calculate dynamic weights ; Dynamic weights Used for weighted fusion of similar data and calculation of regional confidence.

[0079] In this embodiment, the error rate and data timeliness of each sensor are stored according to a sliding time window. When the error rate of the same sensor exceeds a preset error threshold or the data timeliness falls below a preset timeliness threshold in multiple consecutive processing cycles, it is determined that the data continuously output by that sensor meets a preset abnormal condition.

[0080] The number of consecutive processing cycles can be set according to the sensor acquisition frequency and alarm real-time requirements. For example, if a preset abnormal condition is met for 3 to 5 consecutive processing cycles, the corresponding sensor can be marked as a sensor to be downweighted.

[0081] Pre-deweighting processing can be performed on sensors to be deweighted, and the abnormal sensor status can be sent to the intelligent early warning application layer. The intelligent early warning application layer returns a deweighting request to the fusion engine based on the abnormal status. After receiving the deweighting request, the fusion engine maintains or further reduces the dynamic weight of the corresponding sensor.

[0082] Dynamic weights after weight reduction It can be represented as: ; in, For the first Dynamic weights of road sensors after weight reduction For weighting coefficients, For the first Dynamic weights of road sensors before weight reduction.

[0083] The self-test command is used to cause the corresponding sensor to perform at least one of the following: zero-point detection, communication status detection, power supply status detection, data acquisition channel detection, or internal fault code reading.

[0084] When the self-test result indicates that the sensor is in normal condition, cancel the weighting coefficient. The dynamic weights are recalculated based on the current error rate and the timeliness of the current data.

[0085] When the self-test result indicates an abnormal sensor condition, the reduced dynamic weight is maintained, and a sensor abnormality alarm is output. The sensor abnormality alarm includes at least the sensor identifier, sensor type, installation location, current error rate, current data timeliness, and self-test result.

[0086] When the data continuously output by the sensor does not meet the preset abnormal conditions, the weight reduction process is not performed, and the dynamic weights continue to be updated according to the error rate and data timeliness.

[0087] S104, weighted and fused similar data in the partition according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and regional confidence level; Furthermore, weighted fusion of similar data across partitions is performed using dynamic weights, including: Multiply each temperature and humidity data or each gas concentration data in the same cockpit zone by the dynamic weight of each sensor. The sum of the multiplication results is divided by the sum of the dynamic weights to obtain the effective value of the temperature and humidity of the region or the effective value of the gas in the region. The dynamic weights participating in the weighted fusion are summed, and the sum of the dynamic weights forms the region confidence score.

[0088] In this embodiment, for the same cockpit partition Road-type sensors, area-type data valid values It can be represented as: ; in, For valid values ​​of the same type of data in the region, This refers to the number of similar sensors participating in fusion within the same cockpit section. For the first Drift correction values ​​for similar sensors in the road, For the first Dynamic weights of sensors of the same type in the road.

[0089] For temperature data, the effective value of the regional temperature. It can be represented as: ; in, This represents the effective value of the regional temperature. This refers to the number of temperature sensors participating in the fusion process within this cockpit section. For the first Dynamic weighting of road temperature sensors, For the first The drift correction value of the road temperature sensor.

[0090] For humidity data, the effective value of regional humidity. It can be represented as: ; in, This represents the effective value of the regional humidity. This represents the number of humidity sensors participating in the fusion process within this cockpit section. For the first Dynamic weighting of road humidity sensors, For the first Drift correction value for road humidity sensor.

[0091] For gas concentration data, the effective value of regional gases It can be represented as: ; in, This represents the effective value of the regional gas. This refers to the number of gas sensors participating in the fusion within this cockpit section. For the first Dynamic weighting of the gas sensor in the road, For the first The drift correction value of the gas sensor.

[0092] Smoke concentration, CO concentration, and CO2 concentration can be fused using the same data method described above.

[0093] In order to make the regional confidence level comparable to the first preset confidence threshold and the second preset confidence threshold, the sum of the dynamic weights involved in the fusion can be normalized relative to the sum of the reference weights under normal conditions.

[0094] The first The road sensor has an error rate of a preset minimum error rate and a data timeliness of [missing information]. The reference weight at that time is denoted as ,but: ; in, For the first Reference weights for road sensors, For the error rate term coefficient, To preset the minimum error rate, This is the coefficient for the timeliness term.

[0095] Regional confidence It can be represented as: ; in, For regional confidence, This refers to the number of similar sensors participating in fusion within the same cockpit section. For the first Dynamic weights of road sensors For the first Reference weights for road sensors.

[0096] Regional confidence The range of values ​​for can be limited to: ~ When the calculation yields Greater than At that time, Set as .

[0097] When the sensor error rate is low, the data latency is small, and the sensor is not in a deweighted state within the same cockpit zone, the regional confidence level is... High. When multiple sensors experience increased error rates, data delays, or self-test anomalies, the regional confidence level is [high / high]. reduce.

[0098] The effective values ​​of regional temperature, regional humidity, regional gas, and regional confidence are sent together to the cross-modal association rule base.

[0099] For example, three temperature sensors are installed in a certain cockpit section. The normalized fusion weights of the three temperature sensors are as follows: , and The drift-corrected temperature values ​​are respectively , and Then the effective value of the regional temperature can be expressed as: ; in, This represents the effective value of the regional temperature. This is the drift correction value for the first temperature sensor. This is the drift correction value for the second temperature sensor. This is the drift correction value for the third temperature sensor.

[0100] Humidity data and gas concentration data are calculated using the same method.

[0101] S105, identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image of the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate the continuous risk value and map the warning level; Furthermore, adjust the matching strength of the rule conditions, including: Set the first preset reliability threshold to be less than the second preset reliability threshold; Based on the data timestamp and the cockpit partition to which it belongs, the effective values ​​of temperature and humidity in the region, the effective values ​​of gas in the region, the anomaly type of the display screen image, and the confidence level of the region are matched with the rule conditions item by item; When the confidence level of the region is lower than the first preset confidence threshold, the rule condition is marked as a weak match. When all the remaining rule conditions of the same rule are met, the weak match result is included in the calculation of the continuous risk value. When not all the remaining rule conditions of the same rule are met, the rule warning is not triggered. When the confidence level of the region is higher than the second preset confidence threshold and the rule condition meets the threshold range, the rule condition is marked as a strong match, the weight of the strong match result is adjusted according to the confidence level of the region and included in the calculation of continuous risk value; When the confidence level of the region is between the first preset confidence threshold and the second preset confidence threshold, the rule matching result is included in the calculation of the continuous risk value according to the normal matching state.

[0102] Furthermore, identifying the types of anomalies and target locations in the display screen images includes: Convert the images captured by the industrial camera on the display screen from the three primary color space to the hue, saturation, and lightness color space; Based on the target detection model, target recognition is performed on the red alarm area, amber alarm area, screen flicker area, noise area and display color deviation area in the converted display screen image.

[0103] Furthermore, the continuous risk value is calculated and mapped to the warning level, including: Based on the threshold range of each rule condition, the matching results for the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and abnormal image types on the display screen are determined respectively; The weight of each matching result is adjusted according to the regional confidence level, and the matching results of the same rule are weighted and calculated to obtain the continuous risk value; The continuous risk value is compared with the first risk threshold and the second risk threshold in the rule base, and the normal level, the first-level early warning level or the second-level emergency alarm level is output. After the continuous risk values ​​are mapped to the warning level, the rule warning of the second-level emergency alarm level is written into the priority position of the warning output queue; Ventilation control commands and lighting alarm commands are generated according to the aforementioned warning level.

[0104] Furthermore, marking the rule conditions as weak matches includes: Obtain the remaining rule conditions from the same rule; According to the data type of each remaining rule condition, the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, or anomaly types of display screen images are selected for matching respectively; When all the remaining rule conditions are met, the weights of the matching results of the rule conditions marked as weak matches and the remaining rule conditions are adjusted according to the regional confidence level, and the weighted calculation of each matching result is performed and the continuous risk value is updated. If not all remaining rule conditions are met, the rule conditions marked as weak matches are excluded.

[0105] Further, the calculation of continuous risk values ​​includes: Read the smoke concentration conditions, red malfunction screen conditions, and cockpit partition conditions from the rules associated with the Level 2 emergency alarm; The cockpit zone to which the display screen image belongs is determined based on the target location and the coordinate mapping relationship; The smoke concentration in the effective value of the gas in the cockpit zone, the abnormal type of the display screen image, and the cockpit zone are matched with the smoke concentration condition, the red fault screen condition, and the cockpit zone condition, respectively. The weights of each matching result are adjusted according to the regional confidence level, and a multi-dimensional weighted calculation is performed on each matching result to obtain the continuous fire risk value. The risk value of continuous fires is mapped to an early warning level according to the first risk threshold and the second risk threshold.

[0106] In this embodiment, the original display screen image captured by the industrial camera can be an RGB color space image. Distortion correction, display area cropping, and brightness normalization are performed on the display screen image, and the image is converted from the RGB color space to the HSV color space.

[0107] In the HSV color space, the hue component is used to distinguish between red and amber warning areas, the saturation component is used to distinguish between warning characters and gray-white backgrounds, and the lightness component is used to identify screen flicker, local brightness anomalies, and color cast.

[0108] The target detection model can employ an improved YOLOv7 model trained with abnormal images from the cockpit display screen. The training samples should include at least five types of abnormal images: red alarms, amber alarms, screen flickering, noise, and color cast.

[0109] The first The detection results of each target are recorded as follows: ; in, For the first Target detection results It is an exception type. For target detection bounding boxes or target locations, To identify confidence levels.

[0110] Exception types It can display red alarms, amber alarms, screen flicker, noise, or color cast. Target location. It can be represented by the center coordinates, width, and height of the target detection box, or by the coordinates of the four vertices of the target detection box.

[0111] According to the target location Based on the established corner mapping relationship of the display screen, the display screen to which the abnormal target belongs is determined, and the corresponding cockpit partition is further determined.

[0112] When the same abnormal target is identified by two industrial cameras at the same time, one of the identification results can be selected based on the identification confidence, target integrity and field of view, or the two identification results can be verified for consistency to form a joint identification result.

[0113] In this embodiment, the cross-modal association rule base can pre-store rules for temperature and humidity, gas, vision, and cross-modal association. The cross-modal association rule base can be stored using a JSON configuration file, allowing operations personnel to add or modify rules via an operations terminal.

[0114] A single rule can be represented using IF-THEN propositional logic. For example: If the effective humidity value in the IF area is greater than the humidity threshold, and the anomaly type of the display image is condensation-related, then the THEN outputs a Level 1 warning and generates a ventilation anti-condensation command.

[0115] A single rule must include at least a rule identifier, cockpit partition, rule conditions, condition thresholds, condition base weights, first risk threshold, second risk threshold, warning level, and linkage instructions.

[0116] The first preset reliability threshold is denoted as... The second preset reliability threshold is denoted as ,and: ; in, The first preset confidence threshold, Set the second pre-set confidence threshold.

[0117] In one implementation, It can be set to , It can be set to .

[0118] For the The first rule Each rule condition is used to determine the matching result of the rule condition. , The range of values ​​is ~ Numerical rule conditions can determine the matching result based on the degree to which the effective value of the region falls within the threshold range; image-based rule conditions can determine the matching result based on the recognition confidence of the target detection model.

[0119] The first The basic weight of each rule condition is denoted as . The weights adjusted according to the matching status will be denoted as .

[0120] When regional confidence Below the first preset confidence threshold When this happens, the corresponding rule condition is marked as a weak match.

[0121] A weak match condition participates in the continuous risk value calculation only when all remaining rule conditions of the same rule are satisfied. The adjustment weight of a weak match condition can be expressed as: ; in, For the first Rule number 1 The weights after adjusting the rules and conditions For the first Rule number 1 The basic weight of each rule condition. For regional confidence levels.

[0122] If not all remaining rule conditions of the same rule are met, the rule conditions marked as weak matches are excluded, and no rule warning is triggered.

[0123] When regional confidence Higher than the second preset confidence threshold If the corresponding rule condition meets the threshold range, then the rule condition is marked as a strong match. The adjustment weight of the strong match condition can be expressed as: ; in, For the first Rule number 1 The weights after adjusting the rules and conditions For the first Rule number 1 The basic weight of each rule condition. For regional confidence levels.

[0124] Strong matching conditions are included in the calculation of continuous risk values, and the response priority of the corresponding rules in the early warning output queue is increased.

[0125] When regional confidence At the first preset confidence threshold Second preset confidence threshold When this occurs, the corresponding rule condition is marked as a normal match. The adjustment weight for normal match conditions can be expressed as: ; in, For the first Rule number 1 The weights after adjusting the rules and conditions For the first Rule number 1 The basic weight of each rule condition.

[0126] For including The first rule condition Rule, continuous risk value It can be represented as: ; in, For the first The continuous risk value of the rule, For the first The number of rule conditions contained in a rule. For the first Rule number 1 The weights after adjusting the rules and conditions For the first Rule number 1 The matching results of each rule condition.

[0127] The first risk threshold is denoted as The second risk threshold is denoted as ,and: ; in, The first risk threshold, This is the second risk threshold.

[0128] The mapping relationship between warning levels can be represented as follows: ; in, The warning level is set at [level]. For the first The continuous risk value of the rule, The first risk threshold, The second risk threshold, Normal level It is at the highest level of warning. It is a Level 2 emergency alarm.

[0129] When a continuous risk value is mapped to a Level 2 emergency alarm, the corresponding warning is written to the priority position in the warning output queue. The warning output queue is first sorted according to the warning level, and when the warning levels are the same, they can be sorted from high to low according to the continuous risk value.

[0130] When a rule condition is marked as a weak match, the remaining rule conditions are obtained from the same rule, and the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, or anomaly types of display screen images are selected for matching according to the data type of each remaining rule condition.

[0131] When all remaining rule conditions are met, the weights of the matching results of weak matching rule conditions and remaining rule conditions are adjusted according to the regional confidence level, and the weighted calculation of each matching result is performed and the continuous risk value is updated.

[0132] If not all remaining rule conditions are met, exclude the rule conditions marked as weak matches and do not output the warning corresponding to that rule.

[0133] This avoids the situation where a single sensor condition with low regional confidence level independently triggers an early warning, while allowing the condition to participate in risk calculation as an auxiliary observation when other cross-modal conditions are met together.

[0134] In one implementation, the fire risk rules associated with the Level 2 emergency alarm include smoke concentration conditions, red malfunction screen conditions, and cockpit partition conditions.

[0135] The cockpit zone to which the display screen image belongs is determined based on the abnormal target location and coordinate mapping relationship on the display screen. The effective value of smoke in the cockpit zone, the abnormal type of the display screen image, and the cockpit zone are then matched with the smoke concentration conditions, the red fault screen conditions, and the cockpit zone conditions, respectively.

[0136] The matching result of the smoke concentration condition is denoted as The matching result of the red fault screen condition is recorded as The matching result of the cockpit partition conditions is recorded as The risk value of continuous fires It can be represented as: ; in, This represents the risk value for continuous fires. The weights for smoke concentration conditions are adjusted based on regional confidence and matching status. The results show the matching of smoke concentration conditions. The conditional weights for the red faulty image are adjusted based on regional confidence and matching status. The matching results for the red fault screen condition. The cockpit partitioning conditions are weighted according to the regional confidence level and matching status. The matching results for the cockpit partition conditions.

[0137] When the risk value of continuous fire If the level is greater than or equal to the corresponding second risk threshold, output a level 2 emergency alarm.

[0138] In one implementation, the smoke concentration reaches... When the conditions for a Level 1 smoke warning are met, the smoke concentration reaches [a certain level]. The conditions for a Level 2 smoke alarm are met.

[0139] When the effective value of the regional temperature is greater than At that time, the conditions for a high-temperature warning were met.

[0140] When the effective value of regional humidity is greater than Furthermore, when condensation-related anomalies appear on the display screens within the same cockpit partition, the cross-modal association rule base generates a ventilation anti-condensation warning.

[0141] When the smoke concentration exceeds the smoke threshold and a red malfunction image appears on the display screen within the same cockpit compartment, the continuous fire risk value is calculated. When the continuous fire risk value is greater than or equal to... At that time, a level 2 emergency alarm can be output.

[0142] The The risk threshold is determined by dynamically calculating the risk value of continuous fire based on the current effective gas value in the area, the abnormal identification result of the display screen, the area confidence level, and the weight of the rule conditions.

[0143] S106 generates linkage instructions based on the warning level and anomaly type, sends the linkage instructions to the ventilation system and lighting alarm system, and sends the warning level, continuous risk value, anomaly type and target location to the operation and maintenance terminal.

[0144] In this embodiment, ventilation control commands and lighting warning commands are generated based on the warning level, anomaly type, and cockpit partition.

[0145] For a Level 1 warning, a linked command can be generated to increase the ventilation level, activate the corresponding cockpit zone ventilation, or activate the Level 1 lights flashing.

[0146] For Level 2 emergency alarms, it can generate linkage commands to activate the maximum ventilation setting, activate emergency lighting alarms, maintain alarm output, or send emergency status linkage commands to the higher-level control system.

[0147] The linkage command is sent to the ventilation system and lighting alarm system via a dual-redundant CAN bus. The CAN bus can use the CAN 2.0B protocol, and the communication rate can be 250kbps.

[0148] Ventilation control commands can use the following message formats: The message identifier is 0x181; The data length code DLC is 2; DATA[0] is used to indicate the ventilation setting; DATA[1] is used to represent the target cockpit partition or control mode.

[0149] Light alarm commands can use the following message format: The message identifier is 0x182; The data length code (DLC) is 1; DATA[0] is used to represent the light color or flashing mode.

[0150] The transmission period of CAN commands can be no more than 10ms.

[0151] The same alarm information can also be sent to the remote operation and maintenance HMI terminal via the Wi-Fi 6 wireless communication link using the MQTT protocol.

[0152] The alarm information sent to the operation and maintenance terminal shall include at least the cockpit partition, warning level, continuous risk value, anomaly type, target location, effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, alarm generation time and linkage command.

[0153] The maintenance terminal displays the screen indicating the anomaly based on the target location and marks the corresponding area on the cockpit layout diagram according to the cockpit partitions. Maintenance personnel can use the maintenance terminal to query sensor dynamic weights, area confidence levels, rule matching status, and the execution results of linkage commands.

[0154] Through the above steps, the credibility formed by dynamic weights is passed from the same-data fusion stage to the cross-modal association stage. Regional confidence not only represents the overall credibility of sensor data within the same partition, but also controls the weak, normal, and strong matching states of rule conditions, thus forming a continuous data processing link between same-data fusion and cross-modal fault diagnosis.

[0155] The following describes this embodiment in conjunction with application scenarios: In one exemplary implementation, the monitored object is an area of The cockpit of a flight simulator.

[0156] Eight temperature and humidity monitoring points are installed in the cockpit. Among them, three temperature and humidity monitoring points are installed in the instrument control area, two in the ceiling area, two in the electrical equipment cabinet area, and one in the rear auxiliary area.

[0157] One electrochemical smoke sensor and one infrared gas sensor are installed on the center console and the top of the cockpit, respectively. One industrial camera is installed on the upper left corner and the upper right rear corner of the cockpit.

[0158] PTPv2 is used to synchronize the time of temperature and humidity sensors, gas sensors and industrial cameras, and a world coordinate system is established with the geometric center of the cockpit.

[0159] For three temperature sensors within a specific cockpit zone, the drift correction values ​​output by the Kalman filter are respectively... , and After calculating the dynamic weights based on the error rate and data timeliness, the dynamic weights are then normalized by fusion of similar data to obtain the corresponding fusion coefficients. , and .

[0160] The effective temperature value for this cockpit zone is: ; in, This represents the effective value of the regional temperature.

[0161] The calculated effective value of the regional temperature is approximately .

[0162] When the timeliness of data from a certain temperature sensor decreases due to increased communication delay, or when the error rate increases due to increased drift, the dynamic weight of that temperature sensor decreases, and its influence on the effective value of the regional temperature decreases accordingly.

[0163] At another detection time, the effective humidity value of the instrument control area was... The regional confidence level is The image recognition results from the display screen show anomalies related to condensation. The confidence level of the affected area is higher than the second preset confidence threshold. The regional humidity conditions were marked as a strong match.

[0164] The cross-modal association rule base calculates continuous risk values ​​based on regional humidity conditions, abnormal condensation conditions on the display screen, and cockpit zoning conditions. When the continuous risk value reaches the first-level warning threshold, a ventilation control command is generated to increase the ventilation level in the instrument control area, and a condensation risk warning is displayed on the maintenance terminal.

[0165] At another testing time, the smoke concentration in the top ceiling area reached A red malfunction image appeared on the display screen associated with this zone. The risk value of continuous fire was calculated based on smoke concentration conditions, red malfunction image conditions, and cockpit zone conditions.

[0166] When the risk value of continuous fires is greater than or equal to When the alarm is triggered, a level 2 emergency alarm is output, the alarm is written to the priority position of the warning output queue, and a maximum ventilation level control command and an emergency light alarm command are generated.

[0167] In one testing method, the temperature and humidity detection test lasted for 72 hours, and the test conditions included a normal temperature condition of 25℃, a high temperature condition of 55℃, a low temperature condition of 0℃, and a variable temperature cycle condition. No less than 1000 sets of data were collected for each condition. The root mean square error before and after multi-sensor fusion was compared with the measurement results of a standard temperature and humidity calibrator.

[0168] Under the above test conditions, the temperature detection error can be controlled within ±0.5℃ and the humidity detection error can be controlled within ±1%RH after adopting the method of this embodiment.

[0169] The display anomaly identification test set can include 12,800 images, of which 8,000 are fault images and 4,800 are normal images. The fault images include five types of anomalies: red alarm, amber alarm, screen flicker, noise, and color distortion, and cover bright environment, dark cabin environment, and reverse cabin environment.

[0170] In a set of tests, the object detection model achieved an overall precision of 95.2%, a recall of 93.8%, and a mean precision (mAP@0.5) of 96.1%.

[0171] For the purpose of simplicity, the method steps disclosed in the above embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0172] like Figure 2 As shown, the present invention also provides a flight simulator cockpit monitoring system based on multi-sensor heterogeneous fusion, comprising: The multi-source data synchronous acquisition module 201 is configured to acquire temperature and humidity data, gas concentration data, and display screen images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and record timestamps and location coordinates. The spatiotemporal alignment and coordinate mapping module 202 is configured to establish a coordinate mapping relationship based on the corner points of the display screen, and match the temperature and humidity data and gas concentration data to the cockpit partition and display area. The drift correction and dynamic weight calculation module 203 is configured to perform drift correction on temperature and humidity data and gas concentration data, calculate the error rate between observed values ​​and corrected values ​​and data timeliness, and calculate dynamic weights according to the error rate and data timeliness. The similar data weighted fusion module 204 is configured to perform weighted fusion of similar data in the partition according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, regional gas effective values ​​and regional confidence levels. The visual recognition and cross-modal risk mapping module 205 is configured to identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level and the anomaly type and target location of the display screen image of the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate continuous risk values ​​and map the warning level; The early warning classification and linkage output module 206 is configured to generate linkage instructions based on the early warning level and the anomaly type, send the linkage instructions to the ventilation system and the lighting alarm system, and send the early warning level, continuous risk value, anomaly type and target location to the operation and maintenance terminal.

[0173] It is worth noting that although only some basic functional modules are disclosed in the embodiments of this invention, it does not mean that the composition of this system is limited to the above-mentioned basic functional modules. On the contrary, based on the above-mentioned basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with existing technology to form an infinite number of embodiments or technical solutions. That is to say, this system is open rather than closed. The fact that this embodiment only discloses a few basic functional modules should not be considered as the scope of protection of this invention being limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described separately according to their functions as various units and modules. Of course, in implementing this invention, the functions of each unit and module can be implemented in one or more software and / or hardware.

[0174] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion, characterized in that, include: Acquire temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and record timestamps and location coordinates; Establish coordinate mapping relationships using the corner points of the display screen, and match temperature and humidity data and gas concentration data to the cockpit partitions and display areas; Drift corrections are applied to temperature and humidity data and gas concentration data. The error rate between observed and corrected values ​​and the timeliness of the data are calculated. Dynamic weights are then calculated based on the error rate and the timeliness of the data. The data of the same type in the partition are weighted and merged according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and regional confidence level. Identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image of the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate the continuous risk value and map the warning level; Based on the warning level and anomaly type, a linkage instruction is generated and sent to the ventilation system and lighting alarm system. The warning level, continuous risk value, anomaly type, and target location are also sent to the operation and maintenance terminal.

2. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Establishing a coordinate mapping relationship using the corner points of the display screen, and matching temperature and humidity data and gas concentration data to the cockpit zones and display areas, further includes: The edge computing device is set as the master clock, and a synchronization clock is sent to the acquisition nodes of the temperature and humidity sensor, gas sensor and industrial camera. The timestamps of the temperature and humidity data, the gas concentration data and the display screen image are recorded based on hardware timestamps. Using the geometric center of the cockpit as the origin of the world coordinate system, the three-dimensional physical coordinates of each temperature and humidity sensor and gas sensor are measured. The industrial camera is calibrated using a checkerboard pattern, and the mapping relationship from world coordinates to image coordinates is calculated based on the corner points of the display screen. Based on the three-dimensional physical coordinates, the mapping relationship, and the timestamp, the temperature and humidity data and gas concentration data are associated with the cockpit partition and the display area of ​​the screen image.

3. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Drift corrections were applied to temperature, humidity, and gas concentration data, including: An independent Kalman filter is maintained for each channel of temperature and humidity data and gas concentration data. The sensor observations are input into the Kalman filter, and the drift-corrected estimated values ​​are output. The relative deviation between the sensor observation and the estimated value is calculated as the error rate, and the timeliness of the data is determined based on the delay between the timestamp and the current processing time. When the error rate is lower than the preset minimum error rate, the preset minimum error rate is used to replace the error rate. When the error rate is not lower than the preset minimum error rate, the error rate is retained; The dynamic weight is formed by multiplying the reciprocal of the error rate and the timeliness of the data by preset coefficients and then adding them together.

4. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Weighted fusion of similar data across partitions using dynamic weights further includes: Multiply each temperature and humidity data or each gas concentration data in the same cockpit zone by the dynamic weight of each sensor. The sum of the multiplication results is divided by the sum of the dynamic weights to obtain the effective value of the temperature and humidity of the region or the effective value of the gas in the region. The dynamic weights participating in the weighted fusion are summed, and the sum of the dynamic weights forms the region confidence score.

5. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Adjusting the matching strength of rule conditions further includes: Set the first preset reliability threshold to be less than the second preset reliability threshold; Based on the data timestamp and the cockpit partition to which it belongs, the effective values ​​of temperature and humidity in the region, the effective values ​​of gas in the region, the anomaly type of the display screen image, and the confidence level of the region are matched with the rule conditions item by item; When the confidence level of the region is lower than the first preset confidence threshold, the rule condition is marked as a weak match. When all the remaining rule conditions of the same rule are met, the weak match result is included in the calculation of the continuous risk value. When not all the remaining rule conditions of the same rule are met, the rule warning is not triggered. When the confidence level of the region is higher than the second preset confidence threshold and the rule condition meets the threshold range, the rule condition is marked as a strong match, the weight of the strong match result is adjusted according to the confidence level of the region and included in the calculation of continuous risk value; When the confidence level of the region is between the first preset confidence threshold and the second preset confidence threshold, the rule matching result is included in the calculation of the continuous risk value according to the normal matching state.

6. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Identifying the type of anomaly and the target location in the display screen image, further including: Convert the images captured by the industrial camera on the display screen from the three primary color space to the hue, saturation, and lightness color space; Based on the target detection model, target recognition is performed on the red alarm area, amber alarm area, screen flicker area, noise area and display color deviation area in the converted display screen image.

7. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Calculating continuous risk values ​​and mapping them to warning levels further includes: Based on the threshold range of each rule condition, the matching results for the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, and abnormal image types on the display screen are determined respectively; The weight of each matching result is adjusted according to the regional confidence level, and the matching results of the same rule are weighted and calculated to obtain the continuous risk value; The continuous risk value is compared with the first risk threshold and the second risk threshold in the rule base, and the normal level, the first-level early warning level or the second-level emergency alarm level is output. After the continuous risk values ​​are mapped to the warning level, the rule warning of the second-level emergency alarm level is written into the priority position of the warning output queue; Ventilation control commands and lighting alarm commands are generated according to the aforementioned warning level.

8. The method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, The updating of dynamic weights further includes: The detection sensor checks whether the continuously output data from the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions. The preset abnormal conditions are that the error rate of the continuously output data is higher than the preset error threshold or the timeliness of the data is lower than the preset timeliness threshold. When the data continuously output by the temperature and humidity sensor or the gas sensor meets the preset abnormal conditions, the corresponding sensor is deweighted in the dynamic weight calculation, and a self-test command is sent to the temperature and humidity sensor or the gas sensor that meets the preset abnormal conditions. Upon receiving the weight reduction request, the dynamic weight of the corresponding sensor is reduced. When the self-test result indicates that the sensor is in normal condition, the dynamic weight is updated according to the error rate and the data timeliness. When the self-test result indicates that the sensor is in an abnormal state, the reduced dynamic weight is maintained and a sensor abnormality alarm is output. When the data continuously output by the temperature and humidity sensor or the gas sensor does not meet the preset abnormal conditions, the dynamic weight is updated according to the error rate and the timeliness of the data.

9. A method for monitoring the cockpit of a flight simulator based on multi-sensor heterogeneous fusion according to claim 1, characterized in that, Acquiring temperature and humidity data, gas concentration data, and display screen images, further including: Temperature and humidity sensors are installed in the cockpit instrument control area, the ceiling area, the electrical equipment cabinet area, and the rear auxiliary area. The smoke sensor and the infrared gas sensor are arranged side by side on the center console and the top of the cockpit. Two industrial cameras were mounted on the upper left and upper right rear corners of the cockpit, respectively, to capture images of the display screen with a cross field of view.

10. A flight simulator cockpit monitoring system based on multi-sensor heterogeneous fusion, characterized in that, include: The multi-source data synchronous acquisition module is configured to acquire temperature and humidity data, gas concentration data, and display images collected by temperature and humidity sensors, gas sensors, and industrial cameras, and record timestamps and location coordinates. The spatiotemporal alignment and coordinate mapping module is configured to establish a coordinate mapping relationship based on the corner points of the display screen, and match the temperature and humidity data and gas concentration data to the cockpit partition and display area. The drift correction and dynamic weight calculation module is configured to perform drift correction on temperature and humidity data and gas concentration data, calculate the error rate between observed values ​​and corrected values ​​as well as data timeliness, and calculate dynamic weights based on the error rate and data timeliness. The same type of data weighted fusion module is configured to weight and fuse similar data in the partition according to dynamic weights to obtain the effective values ​​of regional temperature and humidity, regional gas effective values ​​and regional confidence scores. The visual recognition and cross-modal risk mapping module is configured to identify the anomaly type and target location of the display screen image, input the effective values ​​of regional temperature and humidity, effective values ​​of regional gas, regional confidence level, and the anomaly type and target location of the display screen image in the same zone into the cross-modal association rule base, adjust the matching strength of the rule conditions, calculate continuous risk values ​​and map the warning level; The early warning classification and linkage output module is configured to generate linkage instructions based on the early warning level and anomaly type, send the linkage instructions to the ventilation system and lighting alarm system, and send the early warning level, continuous risk value, anomaly type and target location to the operation and maintenance terminal.

Images