Method and system for monitoring on-duty state of meteorological observer based on multi-modal fusion and hardware-software collaboration

Abstract

This invention discloses a method and system for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware-software collaboration. In this method, within a monitoring cycle, an interaction operation data of computer peripherals is collected through a behavior detection engine to determine the effective operation frequency, and facial image data is collected through a visual detection engine to calculate the percentage of effective face frames. The effective operation frequency and the percentage of face frames are input into a preset cross-judgment matrix, and the on-duty status type is output. When an abnormal status occurs, a two-level alarm is executed on an external communication terminal via serial port AT commands. Simultaneously, on-site images are captured, timestamped, and written to an encrypted log to solidify the evidence. This invention employs visual and behavioral dual-modal cross-verification, combined with a remote strong alert mechanism through hardware-software collaboration, effectively reducing the false alarm rate, ensuring timely handling of abnormal statuses, and improving the reliability and intelligence level of meteorological observation post supervision.

Description

Technical Field

[0001] This invention relates to the field of intelligent monitoring technology, and in particular to a method and system for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware-software collaboration. Background Technology

[0002] Key positions such as meteorological observation require staff to remain continuously on duty to ensure the timeliness and accuracy of observational data. Existing personnel on-duty monitoring technologies mainly rely on single visual signals, such as facial detection or posture recognition algorithms, to determine whether personnel have left their posts or are fatigued. However, these purely visual solutions have significant shortcomings in actual operational scenarios: when observers lose facial features due to factors such as looking down to record, consulting paper documents, insufficient lighting, or temporary obstruction of the camera, the system is prone to false alarms; at the same time, existing alarm mechanisms are mostly local pop-ups or background records, which are difficult to effectively remind staff when they are fatigued or during nighttime; in addition, high-precision visual algorithms have high hardware resource requirements, and running them on low-spec operational computers at grassroots meteorological stations can cause system lag; furthermore, existing technologies lack evidence-based mechanisms for abnormal triggering, which is not conducive to subsequent accountability.

[0003] Therefore, how to achieve multi-dimensional, low false alarm, strong alert, and hardware-compatible on-duty status monitoring is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] This invention provides a method and system for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware-software collaboration. It can solve the problems of single monitoring dimensions leading to false alarms, passive and ineffective alarm methods, poor hardware compatibility, and lack of evidence consolidation in existing meteorological observer on-duty status monitoring technologies.

[0005] The first aspect of this invention provides a method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, comprising: Within a monitoring period, the behavior detection engine collects interactive operation data of computer peripherals, and determines the effective operation frequency of the on-duty personnel based on the interactive operation data; the visual detection engine collects facial image data of the on-duty personnel, and calculates the proportion of effective face frames to the total number of frames in the monitoring period based on the facial image data. Input the effective operation frequency and the percentage of effective face frames into the preset cross-judgment matrix, and output the on-duty status type of the on-duty personnel. The on-duty status type includes normal status, off-duty risk, violation risk and fatigue risk. When the on-duty status is classified as risk of leaving the post, risk of violation, or risk of fatigue, the external communication terminal is driven to execute the first-level alarm and the second-level alarm via serial port AT commands. The first-level alarm includes sending a text message containing the type of abnormality, the time of occurrence, and the job identifier to the preset management personnel. The second-level alarm includes automatically dialing the phone number of the preset management personnel and keeping it ringing for a preset duration. At the same time as triggering the alarm, the evidence solidification operation is performed, which includes capturing the current scene, embedding a timestamp watermark in the scene, and writing the detection process data, judgment results and alarm records into an encrypted log.

[0006] Optionally, the interaction operation data of computer peripherals can be collected through a behavior detection engine, and the effective operation frequency of the on-duty personnel can be determined based on the interaction operation data, including: It uses low-level hook technology to monitor mouse movement, clicks, scroll wheel operations, and keyboard key presses in real time to obtain interactive operation data; The total number of mouse and keyboard operations during the monitoring period is counted as the effective operation frequency.

[0007] Optionally, facial image data of on-duty personnel is collected through a visual inspection engine, and the percentage of valid face frames in the total number of frames within the monitoring period is calculated based on the facial image data, including: Use a high-definition camera to capture video streams; A hybrid detection algorithm is used to process the video stream, in which a deep neural network is used to quickly capture facial contours, and a multi-task convolutional neural network is used to perform fine feature verification under complex lighting conditions. The proportion of video frames in which a valid face was successfully identified within the monitoring period is calculated as the percentage of valid face frames.

[0008] Optionally, the effective operation frequency and the percentage of effective face frames are input into a preset cross-determination matrix, and the output on-duty status type of the staff includes: When continuous computer operation is detected during the monitoring period and the percentage of valid face frames is greater than or equal to a preset threshold, a normal status is output. If no computer operation is detected during the monitoring period and the percentage of valid face frames is less than a preset threshold, the risk of leaving the post is output. If continuous computer operation is detected during the monitoring period and the percentage of valid face frames is less than a preset threshold, a violation risk will be output. When no computer operation is detected within the preset no-operation time threshold during the monitoring period and the percentage of valid face frames is greater than or equal to the preset threshold, fatigue risk is output.

[0009] Optionally, driving an external communication terminal to execute first-level and second-level alarms via serial port AT commands includes: Generate a PDU-formatted text message containing the anomaly type, occurrence time, and job identifier, and send the message to multiple pre-defined mobile phone numbers of management personnel via an external communication terminal; After the SMS message is sent, the system automatically executes the dialing command, dials the preset administrator's phone number, and keeps ringing for a preset duration.

[0010] Optionally, the method further includes: When the hardware performance of the local terminal is detected to meet the preset conditions, the full-function mode is enabled, and the behavior detection engine and the visual detection engine are activated at the same time. When the hardware performance of the local terminal is detected to be lower than the preset threshold, switch to lightweight mode, turn off the visual detection engine, and rely on the behavior detection engine to monitor the on-duty status.

[0011] Optionally, the method further includes: After determining the risk of being absent from work or fatigue, and before triggering the first-level alarm, a full-screen confirmation window will pop up on the monitoring terminal screen and an audio prompt will be played. If a confirmation response from the on-duty personnel is received within the preset waiting time, it will be determined as a false alarm and the monitoring cycle will be reset. If no acknowledgment response is received within the preset waiting time, the first-level alarm and the second-level alarm will be activated.

[0012] A second aspect of this invention provides a meteorological observer on-duty status monitoring system based on multimodal fusion and hardware / software collaboration, comprising: The multimodal data acquisition module is used to collect interactive operation data of computer peripherals through the behavior detection engine within a monitoring cycle, and determine the effective operation frequency of the on-duty personnel based on the interactive operation data; and to collect facial image data of the on-duty personnel through the vision detection engine, and calculate the proportion of effective face frames to the total number of frames in the monitoring cycle based on the facial image data. The status determination module is used to input the effective operation frequency and the proportion of effective face frames into a preset cross-determination matrix and output the on-duty status type of the on-duty personnel. The on-duty status types include normal status, off-duty risk, violation risk and fatigue risk. The alarm module is used to drive an external communication terminal to execute a first-level alarm and a second-level alarm when the on-duty status is classified as off-duty risk, violation risk, or fatigue risk. The first-level alarm includes sending an SMS containing the anomaly type, occurrence time, and job identifier to a preset manager. The second-level alarm includes automatically dialing the phone number of a preset manager and keeping it ringing for a preset duration. The evidence consolidation module is used to perform evidence consolidation operations when an alarm is triggered. The evidence consolidation operations include capturing the current scene, embedding a timestamp watermark into the scene, and writing the detection process data, judgment results, and alarm records into an encrypted log.

[0013] A third aspect of this invention provides a meteorological observer on-duty status monitoring device based on multimodal fusion and hardware / software collaboration, comprising: One or more processors; A memory on which one or more programs are stored; When the one or more programs are executed by the one or more processors, the one or more processors implement the meteorological observer on-duty status monitoring method based on multimodal fusion and hardware-software collaboration as described above.

[0014] The fourth aspect of the present invention provides a computer storage medium for storing a program, which, when executed, is used to implement the meteorological observer on-duty status monitoring method based on multimodal fusion and hardware-software collaboration as described in any of the preceding claims.

[0015] Compared with the prior art, the present invention has the following beneficial effects: Multimodal cross-validation significantly reduces the false alarm rate. This invention integrates computer peripheral operation behavior with facial visual features to construct a cross-determination matrix, effectively distinguishing normal scenarios such as looking down at a computer or changes in lighting from genuine absence from work or fatigue behavior.

[0016] Strong alerts through hardware and software collaboration ensure no alarm is missed. This invention employs a two-tiered remote alarm mechanism combining SMS and telephone ringing. Leveraging the strong physical penetration of the telephone ringtone, it ensures timely intervention by management personnel, preventing passive alerts from being ignored.

[0017] The operating mode is adaptive, and it has strong hardware compatibility. This invention supports switching between full-featured mode and lightweight mode. On low-end terminals, the visual engine can be turned off and behavior monitoring can be relied upon alone to ensure the smooth operation of core business systems.

[0018] Evidence is automatically preserved, and responsibility is traceable. When an alarm is triggered, this invention automatically captures images, overlays timestamps and watermarks, and encrypts and stores logs to form a complete chain of evidence, facilitating post-event review and responsibility determination. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 A flowchart illustrating a method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, provided as an embodiment of the present invention; Figure 2 A flowchart illustrating another method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, provided as an embodiment of the present invention; Figure 3 A schematic diagram of a meteorological observer on-duty status monitoring system based on multimodal fusion and hardware / software collaboration is provided for an embodiment of the present invention. Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention. Detailed Implementation

[0021] This invention provides a method and system for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware-software collaboration, which addresses the problems of existing meteorological observer on-duty status monitoring technologies, such as single monitoring dimensions leading to false alarms, passive and ineffective alarm methods, poor hardware compatibility, and lack of evidence consolidation.

[0022] See Figure 1 This figure is a flowchart illustrating a method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, according to an embodiment of the present invention. The method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration provided in this embodiment of the present invention can be implemented, for example, through the following steps S101-104.

[0023] S101: Within a monitoring cycle, the behavior detection engine collects interactive operation data of computer peripherals and determines the effective operation frequency of the on-duty personnel based on the interactive operation data; the visual detection engine collects facial image data of the on-duty personnel and calculates the proportion of effective face frames to the total number of frames in the monitoring cycle based on the facial image data.

[0024] In this embodiment of the invention, a low-level hook technology is used to monitor mouse movement, clicks, scroll wheel operations, and keyboard key presses in real time to obtain interactive operation data; the total number of mouse and keyboard operations within the monitoring period is counted as the effective operation frequency.

[0025] The system uses a high-definition camera to acquire video streams; it employs a hybrid detection algorithm to process the video streams, where a deep neural network is used to quickly capture facial contours, and a multi-task convolutional neural network is used to perform fine feature verification under complex lighting conditions; the system calculates the proportion of video frames in which valid faces are successfully identified within the monitoring period to the total number of frames in the monitoring period, which is taken as the percentage of valid face frames.

[0026] Specifically, two independent sub-engines run synchronously to acquire real-time status data of on-duty personnel. The behavior detection engine uses low-level hook technology to monitor the interaction commands of computer peripherals in real time, including mouse movement, clicks, scroll wheel operations, and keyboard key presses. The frequency of valid operations is counted within a set monitoring period (e.g., 50 seconds). The visual detection engine utilizes a high-definition camera and a hybrid detection algorithm (DNN deep neural network + MTCNN multi-task convolutional neural network). DNN is used to quickly capture facial contours, while MTCNN is used for fine-grained feature verification under complex lighting conditions to ensure the robustness of face recognition.

[0027] In one implementation of this invention, the visual inspection engine is not limited to using an RGB camera; it can also integrate an infrared (IR) camera or a depth sensor (such as a Kinect or ToF lens). In extremely low-light environments at night, the system automatically switches to infrared mode for facial feature extraction, or uses a depth sensor to perceive the three-dimensional contours and liveness features of personnel on duty, thereby solving the monitoring failure caused by extremely poor ambient light and enhancing the system's robustness in all-weather environments.

[0028] In one implementation of this invention, a wearable device interface can be added. A smart bracelet worn by on-duty personnel is connected via Bluetooth or ZigBee to acquire biometric data such as heart rate and blood oxygen saturation. This physiological data is introduced as a third dimension (in addition to facial features and behavior) into the decision matrix to monitor and alert on extreme physical abnormalities such as cardiac arrest.

[0029] In one implementation of this invention, when the hardware performance of the local terminal is detected to meet preset conditions, the full-function mode is enabled, and the behavior detection engine and the visual detection engine are activated simultaneously; when the hardware performance of the local terminal is detected to be lower than a preset threshold, the system switches to lightweight mode, the visual detection engine is turned off, and the on-duty status is monitored by the behavior detection engine.

[0030] In one implementation of this invention, microphone array sound analysis can also be integrated. By recognizing audio signals in the environment such as speech, page turning sounds, and walkie-talkie communication sounds, it helps determine whether personnel are on duty, thus providing supplementary judgment basis when there are blind spots in the visual range.

[0031] Specifically, to ensure compatibility with different hardware environments, this invention designs two operating modes: Full-featured mode: Enables all visual and behavioral engines, suitable for terminals equipped with dedicated graphics cards or high-performance CPUs.

[0032] Lightweight mode: For low-spec business terminals, the system supports disabling camera access and visual algorithm modules, relying solely on underlying peripheral hooks to determine "behavior on duty" logic, reducing CPU usage to an extremely low level and ensuring no interference with the operation of core business software.

[0033] In one implementation of this invention, the system can be designed with a distributed architecture. Within a large meteorological observation hall, multiple cameras (main view + side view) simultaneously connect to a central control server to perform rolling monitoring of multiple duty stations. The system dynamically adjusts the monitoring frequency and alarm thresholds based on the different job responsibility levels of each station.

[0034] S102: Input the effective operation frequency and the percentage of effective face frames into the preset cross-judgment matrix, and output the on-duty status type of the duty personnel.

[0035] In this embodiment of the invention, the on-duty status types include normal status, risk of absence from duty, risk of violation, and risk of fatigue. When continuous computer operation is detected within the monitoring period and the percentage of valid face frames is greater than or equal to a preset threshold, a normal status is output; when no computer operation is detected within the monitoring period and the percentage of valid face frames is less than the preset threshold, a risk of absence from duty is output; when continuous computer operation is detected within the monitoring period and the percentage of valid face frames is less than the preset threshold, a risk of violation is output; when no computer operation is detected within a preset no-operation time threshold within the monitoring period and the percentage of valid face frames is greater than or equal to the preset threshold, a risk of fatigue is output.

[0036] Specifically, a cross-judgment matrix based on the "operation-face" dimension is established. Using a preset effective percentage threshold (e.g., effective face frame percentage ≥ 60%), the on-duty status of staff is categorized into the following four types: Normal state: During the monitoring period, continuous computer operation is detected and facial recognition is successful; Risk of being absent from duty: During the monitoring period, there is neither computer interaction nor the ability to identify a valid face; Risk of violation: During the monitoring period, continuous computer operation was detected, but no valid face could be identified (suspected to be not operated by the person in question or obstructed by the monitoring system). Fatigue risk: During the monitoring period, a valid face is identified, but there is no computer interaction for an extended period of time.

[0037] S103: When the on-duty status type is off-duty risk, violation risk, or fatigue risk, the external communication terminal is driven to execute the first-level alarm and the second-level alarm through serial port AT commands.

[0038] In this embodiment of the invention, the first-level alarm includes sending a text message containing the anomaly type, occurrence time, and job identifier to preset management personnel, generating a PDU-formatted text message containing the anomaly type, occurrence time, and job identifier, and sending the text message to the mobile phone numbers of multiple preset management personnel via an external communication terminal. The second-level alarm includes automatically dialing the phone number of a preset management personnel and maintaining a ringing time for a preset duration. After the text message is sent, the alarm automatically executes a dialing command to dial the phone number of the preset management personnel and maintains a ringing time for a preset duration.

[0039] Specifically, once an abnormal state is determined (absence from duty, violation of regulations, or fatigue), an alarm process based on serial AT command communication is immediately initiated: Level 1 Alarm (Information Delivery): The system automatically generates a PDU-formatted SMS message containing the anomaly type, occurrence time, and job ID, and sends it to the mobile phones of multiple preset management personnel via an external communication terminal (such as a text messaging modem).

[0040] Level 2 Alarm (Physical Wake-up): After the SMS is sent, the system automatically triggers a dialing command (e.g., ATD command) to call the administrator. The system is set to maintain the ringing duration (e.g., 30 seconds), using the strong physical penetration of the telephone ring to forcibly wake up or alert the administrator.

[0041] In one implementation of this invention, before triggering the first-level alarm after determining the risk of being absent from work or fatigue risk, a full-screen confirmation window pops up on the monitoring terminal screen and an audio prompt is played; if a confirmation response from the on-duty personnel is received within a preset waiting time, it is determined to be a false alarm and the monitoring cycle is reset; if no confirmation response is received within the preset waiting time, the first-level alarm and the second-level alarm are activated.

[0042] Specifically, a local interactive confirmation phase is added before triggering a remote alarm. When the initial determination is "off-duty" or "fatigue," a full-screen confirmation window pops up on the business computer screen, accompanied by a specific audio prompt. If the on-duty personnel click the confirmation button or enter a voice command within a preset waiting time (e.g., 10 seconds), it is considered a false alarm and the monitoring cycle is reset; if there is no response, SMS and telephone alarms are immediately activated, thereby further reducing the frequency of false alarms affecting management personnel and improving the system's intelligent interaction level.

[0043] In one implementation of the present invention, the present invention is not limited to using an external communication terminal (e.g., a modem), but may also employ the following communication structure: VoIP / voice gateway connects to a VoIP server via a local area network to enable automatic voice dialing based on network protocols; Narrowband IoT modules (NB-IoT / LoRa) use independent IoT communication links to send alarm data, ensuring that alarm signals can still be delivered via independent wireless frequency bands even when the business LAN is paralyzed. Instant messaging push (Webhook) pushes alarm messages with captured images to office platforms such as DingTalk and WeChat Work through API interfaces.

[0044] In one implementation of this invention, the system can interact in real time with the background data of the meteorological reporting software. The system reads the logs of the reporting software in real time. For example, if the current time is close to the hourly reporting time (e.g., XX:55), and the system detects that it is currently in an "unattended" state, it automatically increases the monitoring sensitivity and shortens the alarm trigger threshold.

[0045] In one implementation of this invention, to further demonstrate the system's ability to perceive meteorological observation business logic, the technical solution can introduce a dynamic weighted algorithm based on meteorological task time windows. The system pre-configures task time windows synchronized with meteorological observation operations, such as 10 minutes before hourly reporting, encrypted observation periods, and radar calibration periods. Within different time windows, the weight coefficients of each dimension in the cross-judgment matrix are dynamically adjusted: in the period approaching the reporting window, the weight threshold of the behavior detection engine is increased, requiring more frequent computer peripheral operations; simultaneously, the constraint on the proportion of effective face frames is reduced to avoid false alarms due to head-down data recording. During regular monitoring windows, the default weights are restored. By real-time reading of the task status of the business system or local timetable, the judgment thresholds for effective operation frequency and the proportion of effective face frames are dynamically corrected, allowing monitoring sensitivity to adaptively change with the urgency of the business, reducing false alarm rates during non-critical periods and improving response rigidity during critical periods.

[0046] In one implementation of this invention, to ensure the reliability of the physical closed-loop alarm link, alarm link self-healing and switching logic based on serial port return status codes is added. When the system drives an external communication terminal to send an SMS message via serial port AT commands, it parses the communication status codes returned by the serial port in real time: if "+CMGS: xxx" is returned, it indicates that the SMS message was sent successfully, and the log is recorded normally; if "ERROR" or "CMS ERROR" or other abnormal codes are returned, it is determined that the current communication terminal or link has failed. The system then automatically performs link self-healing operations: first, it switches to a backup serial port or backup communication terminal (such as a second SMS modem or 4G DTU); if the backup link is also unavailable, it skips the SMS alarm, immediately triggers a second-level alarm (voice dialing), and sends a preset voice prompt indicating the abnormality type to the administrator during the ringing process. At the same time, the link abnormality and switching process are recorded in an encrypted log, forming a complete physical closed loop from abnormality detection to alarm delivery, ensuring that administrators can receive mandatory reminders under any communication failure.

[0047] S104: At the same time as triggering the alarm, perform the evidence solidification operation, which includes capturing the current scene, embedding a timestamp watermark in the scene, and writing the detection process data, judgment results and alarm records into the encrypted log.

[0048] In this embodiment of the invention, the following operations are performed simultaneously with triggering the alarm: Synchronous capture: Immediately invokes the vision module to capture the current scene.

[0049] Digital watermarking: embedding high-precision timestamp watermarks into images.

[0050] Log auditing: The detection process data, judgment results, SMS sending status and call records are written to the local encrypted log to establish a complete evidence chain from "anomaly detection" to "alarm completion".

[0051] Beneficial effects: 1. Unlike traditional solutions that rely solely on image recognition, this invention incorporates a low-level system hook module to collect real-time physical interaction data from peripherals such as keyboards and mice. Visual judgment results (facial feature percentage) and behavioral judgment results (operation frequency) are logically fused to establish a multi-dimensional on-duty judgment matrix. This significantly reduces false alarms caused by purely visual monitoring (such as judgment failures in scenarios like observers looking down at documents, drastic changes in ambient light, or temporary camera obstruction). Through dual-dimensional comparison, it can accurately distinguish between different risk types such as "fatigue and drowsiness" and "absence from duty," improving the robustness and accuracy of monitoring.

[0052] 2. This invention adds a hardware control module based on serial communication to the system architecture, driving the external communication terminal through the AT command set. Upon triggering an anomaly, it not only sends a detailed status SMS message but also automatically dials a pre-registered phone number and maintains a ringing frequency for a specific duration. This solves the drawback of existing technologies where local pop-ups or simple SMS alerts are easily ignored. The phone ringing has stronger physical penetration and alert intensity, ensuring that management personnel can perceive and intervene in risks immediately even when asleep or not under monitoring, achieving a true closed loop in risk management.

[0053] 3. A "lightweight operation mode" that adapts to both algorithm depth and running efficiency is introduced. For different hardware configurations, this invention designs a switchable hierarchical monitoring structure. In full-function mode, a hybrid visual algorithm combining DNN and MTCNN is used; in lightweight mode, the visual engine can be disabled, relying entirely on underlying peripheral interaction logic for job status determination. This resolves the conflict between high-precision visual algorithms and business software competing for system resources. On business terminals without high-performance GPUs / CPUs, the system can still maintain basic on-duty monitoring functions with extremely low resource consumption (low CPU consumption), ensuring the smooth operation of core business systems such as meteorological observation and improving the system's versatility.

[0054] 4. This invention implements a "proof solidification and automatic tagging" structure at the instant of anomaly triggering. Within milliseconds of logical anomaly detection, it structurally integrates the linked operations of "automatic image capture, timestamp watermark overlay, and local log encryption writing." Compared to the tedious manual extraction of surveillance footage in existing solutions, this invention can automatically provide image evidence with clear time dimensions and status determination criteria. This not only reduces the workload of manual review but also provides an immutable and objective basis for subsequent security responsibility definition and duty performance evaluation.

[0055] Now combined Figure 2 illustrate, Figure 2 This is a flowchart illustrating another method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, provided as an embodiment of the present invention. After system startup, an operating mode selection is performed, automatically switching to full-function mode or lightweight mode based on hardware performance configuration.

[0056] In full-function mode, both the visual detection engine and the behavior detection engine are activated simultaneously. The visual detection engine uses a hybrid algorithm combining deep neural networks and multi-task convolutional neural networks to acquire facial feature frames in real time, while the behavior detection engine uses low-level hook technology to acquire keyboard interaction data in real time. The system is set with a fixed monitoring cycle, inputting the acquired dual-modal data into a state fusion judgment matrix and outputting the status type of the on-duty personnel. When the judgment result is a normal status, the system continues monitoring for the next cycle; when the judgment result is a risk of leaving the post, a risk of violation, or a risk of fatigue, the anomaly handling and alarm subsystem is triggered.

[0057] During the anomaly handling process, the system simultaneously performs evidence solidification operations, including capturing on-site images, overlaying timestamp watermarks, and generating encrypted audit logs; at the same time, it executes two levels of remote alarms in sequence: the first level alarm sends a PDU format SMS, the second level alarm triggers an ATD dialing command and keeps ringing for 30 seconds, and finally the alarm ends and waits for the system to reset.

[0058] In lightweight mode, the system disables the visual detection engine and relies solely on the behavior detection engine to determine on-duty status.

[0059] Based on the methods provided in the above embodiments, this invention also provides a meteorological observer on-duty status monitoring system based on multimodal fusion and software-hardware collaboration. The following describes the meteorological observer on-duty status monitoring system based on multimodal fusion and software-hardware collaboration in conjunction with the accompanying drawings.

[0060] See Figure 3 The figure is a schematic diagram of the structure of a meteorological observer on-duty status monitoring system based on multimodal fusion and hardware-software collaboration provided by an embodiment of the present invention.

[0061] The meteorological observer on-duty status monitoring system 300 based on multimodal fusion and hardware-software collaboration provided in this embodiment of the invention includes: a multimodal data acquisition module 301, a status determination module 302, an alarm module 303, and an evidence solidification module 304.

[0062] The multimodal data acquisition module 301 is used to collect interactive operation data of computer peripherals through the behavior detection engine within a monitoring cycle, and determine the effective operation frequency of the on-duty personnel based on the interactive operation data; and to collect facial image data of the on-duty personnel through the vision detection engine, and calculate the proportion of effective face frames to the total number of frames in the monitoring cycle based on the facial image data. The status determination module 302 is used to input the effective operation frequency and the proportion of effective face frames into a preset cross-determination matrix and output the on-duty status type of the on-duty personnel. The on-duty status types include normal status, off-duty risk, violation risk and fatigue risk. The alarm module 303 is used to drive an external communication terminal to execute a first-level alarm and a second-level alarm through serial port AT commands when the on-duty status type is off-duty risk, violation risk or fatigue risk. The first-level alarm includes sending a text message containing the abnormality type, occurrence time and job identifier to the preset management personnel. The second-level alarm includes automatically dialing the phone number of the preset management personnel and keeping it ringing for a preset duration. The evidence consolidation module 304 is used to perform evidence consolidation operations when an alarm is triggered. The evidence consolidation operations include capturing the current scene, embedding a timestamp watermark in the scene, and writing the detection process data, judgment results, and alarm records into an encrypted log.

[0063] In one possible implementation, the multimodal data acquisition module 301 is specifically used for: It uses low-level hook technology to monitor mouse movement, clicks, scroll wheel operations, and keyboard key presses in real time to obtain interactive operation data; The total number of mouse and keyboard operations during the monitoring period is counted as the effective operation frequency.

[0064] In one possible implementation, the multimodal data acquisition module 301 is specifically used for: Use a high-definition camera to capture video streams; A hybrid detection algorithm is used to process the video stream, in which a deep neural network is used to quickly capture facial contours, and a multi-task convolutional neural network is used to perform fine feature verification under complex lighting conditions. The proportion of video frames in which a valid face was successfully identified within the monitoring period is calculated as the percentage of valid face frames.

[0065] In one possible implementation, the state determination module 302 is specifically used for: When continuous computer operation is detected during the monitoring period and the percentage of valid face frames is greater than or equal to a preset threshold, a normal status is output. If no computer operation is detected during the monitoring period and the percentage of valid face frames is less than a preset threshold, the risk of leaving the post is output. If continuous computer operation is detected during the monitoring period and the percentage of valid face frames is less than a preset threshold, a violation risk will be output. When no computer operation is detected within the preset no-operation time threshold during the monitoring period and the percentage of valid face frames is greater than or equal to the preset threshold, fatigue risk is output.

[0066] In one possible implementation, the alarm module 303 is specifically used for: Generate a PDU-formatted text message containing the anomaly type, occurrence time, and job identifier, and send the message to multiple pre-defined mobile phone numbers of management personnel via an external communication terminal; After the SMS message is sent, the system automatically executes the dialing command, dials the preset administrator's phone number, and keeps ringing for a preset duration.

[0067] In one possible implementation, a runtime mode switching module is also included, specifically for: When the hardware performance of the local terminal is detected to meet the preset conditions, the full-function mode is enabled, and the behavior detection engine and the visual detection engine are activated at the same time. When the hardware performance of the local terminal is detected to be lower than the preset threshold, switch to lightweight mode, turn off the visual detection engine, and rely on the behavior detection engine to monitor the on-duty status.

[0068] In one possible implementation, a local interactive confirmation module is also included, specifically for: After determining the risk of being absent from work or fatigue, and before triggering the first-level alarm, a full-screen confirmation window will pop up on the monitoring terminal screen and an audio prompt will be played. If a confirmation response from the on-duty personnel is received within the preset waiting time, it will be determined as a false alarm and the monitoring cycle will be reset. If no acknowledgment response is received within the preset waiting time, the first-level alarm and the second-level alarm will be activated.

[0069] Since the meteorological observer on-duty status monitoring system 300 based on multimodal fusion and software-hardware collaboration is the system corresponding to the meteorological observer on-duty status monitoring method based on multimodal fusion and software-hardware collaboration provided in the above method embodiments, the specific implementation of each module of the meteorological observer on-duty status monitoring system 300 based on multimodal fusion and software-hardware collaboration is based on the same concept as in the above method embodiments. Therefore, for the specific implementation of each module of the meteorological observer on-duty status monitoring system 300 based on multimodal fusion and software-hardware collaboration, please refer to the description section of the meteorological observer on-duty status monitoring method based on multimodal fusion and software-hardware collaboration in the above method embodiments, and will not be repeated here.

[0070] This invention also provides a meteorological observer on-duty status monitoring device based on multimodal fusion and hardware / software collaboration, the device comprising: a processor and a memory; The memory is used to store instructions; The processor is used to execute the instructions in the memory to perform the meteorological observer on-duty status monitoring method based on multimodal fusion and hardware-software collaboration mentioned in the above embodiments.

[0071] It should be noted that the hardware structure of the meteorological observer on-duty status monitoring device based on multimodal fusion and hardware / software collaboration provided in this embodiment of the invention can be as follows: Figure 4 The structure shown, Figure 4 This is a schematic diagram of the structure of a device provided in an embodiment of the present invention.

[0072] Please see Figure 4 As shown, device 400 includes: processor 410, communication interface 420, and memory 430. The number of processors 410 in device 400 can be one or more. Figure 4 Taking a processor as an example, in this embodiment of the invention, the processor 410, communication interface 420, and memory 430 can be connected via a bus system or other means. Figure 4 Taking the connection between China and Israel via the bus system 440 as an example.

[0073] Processor 410 may be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP. Processor 410 may further include hardware chips. These hardware chips may be application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or combinations thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0074] The memory 430 may include volatile memory, such as random-access memory (RAM); the memory 430 may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 430 may also include a combination of the above types of memory.

[0075] Optionally, the memory 430 stores an operating system and programs, executable modules, or data structures, or subsets thereof, or extended sets thereof. The programs may include various operation instructions for implementing various operations. The operating system may include various system programs for implementing various basic services and handling hardware-based tasks. The processor 410 can read the programs in the memory 430 to implement the data acquisition method provided in this embodiment of the invention.

[0076] The bus system 440 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus system 440 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0077] This invention also provides a computer-readable storage medium including instructions that, when executed on a computer, cause the computer to perform the meteorological observer on-duty status monitoring method based on multimodal fusion and hardware / software collaboration mentioned in the above embodiments.

[0078] This invention also provides a computer program product containing instructions that, when run on a computer, causes the computer to execute the meteorological observer on-duty status monitoring method based on multimodal fusion and hardware-software collaboration mentioned in the above embodiments.

[0079] Although the invention has been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that various changes in form and detail may be made to the invention without departing from the spirit and scope of the invention as defined in the appended claims, all of which shall be within the scope of protection of the invention.

Claims

1. A method for monitoring the on-duty status of meteorological observers based on multimodal fusion and hardware / software collaboration, characterized in that, include: Within a monitoring cycle, interactive operation data of computer peripherals are collected through a behavior detection engine, and the effective operation frequency of the on-duty personnel is determined based on the interactive operation data. The visual detection engine collects facial image data of the on-duty personnel, and calculates the percentage of valid face frames in the total number of frames in the monitoring period based on the facial image data. The effective operation frequency and the effective face frame ratio are input into a preset cross-determination matrix to output the on-duty status type of the on-duty personnel. The on-duty status type includes normal status, off-duty risk, violation risk and fatigue risk. When the on-duty status type is off-duty risk, violation risk, or fatigue risk, the external communication terminal is driven to execute the first-level alarm and the second-level alarm through serial port AT commands. The first-level alarm includes sending a text message containing the abnormality type, occurrence time, and job identifier to the preset manager. The second-level alarm includes automatically dialing the preset manager's phone number and keeping it ringing for a preset duration. Simultaneously with triggering the alarm, an evidence consolidation operation is performed, which includes capturing the current scene, embedding a timestamp watermark in the scene, and writing the detection process data, judgment results, and alarm records into an encrypted log.

2. The method according to claim 1, characterized in that, The step of collecting interactive operation data from computer peripherals through a behavior detection engine and determining the effective operation frequency of the on-duty personnel based on the interactive operation data includes: The underlying hook technology is used to monitor mouse movement, clicks, scroll wheel operations, and keyboard key presses in real time to obtain the interactive operation data; The total number of mouse and keyboard operations within the monitoring period is counted and used as the effective operation frequency.

3. The method according to claim 1, characterized in that, The process of collecting facial image data of on-duty personnel through a visual detection engine and calculating the percentage of valid face frames out of the total number of frames in the monitoring period based on the facial image data includes: Use a high-definition camera to capture video streams; The video stream is processed using a hybrid detection algorithm, in which a deep neural network is used to quickly capture facial contours, and a multi-task convolutional neural network is used to perform fine feature verification under complex lighting conditions. The proportion of video frames in which a valid face was successfully identified within the monitoring period is calculated as the percentage of valid face frames.

4. The method according to claim 1, characterized in that, The step of inputting the effective operation frequency and the percentage of effective face frames into a preset cross-determination matrix to output the on-duty status type of the staff includes: When continuous computer operation is detected during the monitoring period and the percentage of valid face frames is greater than or equal to a preset threshold, the normal state is output. When no computer operation is detected during the monitoring period and the percentage of valid face frames is less than the preset threshold, the risk of leaving the post is output. When continuous computer operation is detected during the monitoring period and the percentage of valid face frames is less than the preset threshold, the violation risk is output. When no computer operation is detected within the preset no-operation time threshold during the monitoring period and the percentage of valid face frames is greater than or equal to the preset threshold, the fatigue risk is output.

5. The method according to claim 1, characterized in that, The step of driving an external communication terminal to execute the first-level alarm and the second-level alarm via serial port AT commands includes: Generate a PDU-formatted text message containing the anomaly type, occurrence time, and job identifier, and send the text message to multiple preset management personnel's mobile phone numbers via an external communication terminal; After the text message is sent, the dialing command is automatically executed to dial the preset administrator's phone number and the ringing continues for a preset duration.

6. The method according to claim 1, characterized in that, The method further includes: When the hardware performance of the local terminal is detected to meet the preset conditions, the full-function mode is enabled, and the behavior detection engine and the vision detection engine are activated at the same time. When the hardware performance of the local terminal is detected to be lower than a preset threshold, the system switches to lightweight mode, shuts down the visual detection engine, and relies on the behavior detection engine to monitor the on-duty status.

7. The method according to claim 1, characterized in that, The method further includes: Before triggering the first-level alarm after determining the risk of leaving the post or the risk of fatigue, a full-screen confirmation window pops up on the monitoring terminal screen and plays an audio prompt. If a confirmation response from the on-duty personnel is received within the preset waiting time, it is determined to be a false alarm and the monitoring cycle is reset. If the confirmation response is not received within the preset waiting time, the first-level alarm and the second-level alarm will be activated.

8. A meteorological observer on-duty status monitoring system based on multimodal fusion and hardware / software collaboration, characterized in that, include: The multimodal data acquisition module is used to collect interactive operation data of computer peripherals through a behavior detection engine within a monitoring period, and determine the effective operation frequency of the on-duty personnel based on the interactive operation data; and to collect facial image data of the on-duty personnel through a visual detection engine, and calculate the proportion of effective face frames to the total number of frames in the monitoring period based on the facial image data. The status determination module is used to input the effective operation frequency and the proportion of effective face frames into a preset cross-determination matrix and output the on-duty status type of the on-duty personnel. The on-duty status type includes normal status, off-duty risk, violation risk and fatigue risk. The alarm module is used to drive an external communication terminal to execute a first-level alarm and a second-level alarm via serial port AT commands when the on-duty status type is off-duty risk, violation risk, or fatigue risk. The first-level alarm includes sending a text message containing the abnormality type, occurrence time, and job identifier to a preset manager. The second-level alarm includes automatically dialing the phone number of a preset manager and keeping it ringing for a preset duration. The evidence consolidation module is used to perform evidence consolidation operations when an alarm is triggered. The evidence consolidation operations include capturing the current scene, embedding a timestamp watermark in the scene, and writing the detection process data, judgment results, and alarm records into an encrypted log.

9. An electronic device, characterized in that, The device includes: a processor and a memory; The memory is used to store instructions; The processor is configured to execute the instructions in the memory to perform the method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, Including instructions that, when run on a computer, cause the computer to perform the method described in any one of claims 1-7 above.

Images