Lightweight system and method based on image snapshot and multi-modal AI recognition

Abstract

The application relates to the technical field of intelligent monitoring and safe operation, in particular to a lightweight system and method based on image snapshot and multi-modal AI recognition, which comprises a monitoring module for monitoring motion information, positioning information and environmental information; a collecting module for collecting sound information and image information; a main control module for controlling the opening and closing and working state of each module, which is provided with two-stage wake-up mechanisms and finally controls the terminal to enter a working mode; in the working mode, the main control module controls the collecting module to periodically collect sound information and image information through a second preset collecting frequency, and uploads the collected sound information and image information, the motion information and the environmental information collected by the monitoring module to the cloud; the cloud analyzes the operation state and the risk level according to the uploaded image information, sound information, motion information and environmental information, and triggers the terminal to perform early warning. The scheme can comprehensively monitor and analyze the operation process of multi-source data, perform multi-aspect scene perception, and reduce power consumption and network dependency.

Description

Technical Field

[0001] The invention relates to the field of intelligent monitoring and safe operation technology, specifically a lightweight system and method based on image capture and multimodal AI recognition. Background Technology

[0002] With the deepening of the intelligent transformation of the power industry, on-site operation safety monitoring has become a key link in ensuring the stable operation of the power system. The power operation environment is complex and has many risk factors. Traditional safety supervision methods mainly rely on manual inspections and fixed-point video surveillance, which have problems such as large blind spots, slow response, and difficulty in covering mobile and dispersed operation points.

[0003] To address the aforementioned issues, smart wearable monitoring devices (such as smart safety helmets) that integrate video capture, positioning, and communication functions are gradually being applied to high-risk industries such as power and construction, enabling real-time monitoring and recording of the work process.

[0004] Existing smart wearable monitoring devices are mainly divided into two categories: the first category adopts a continuous video recording mode, which uploads the entire operation process to the monitoring center in real time; the second category adopts a local storage + post-event analysis mode, which exports the video for manual or semi-automatic review after the operation is completed.

[0005] Both methods have drawbacks: The first method has high power consumption and short battery life. Continuous video recording or high-frequency image uploading leads to extremely high power consumption, and the battery life is usually only 2-4 hours, which is difficult to meet the monitoring needs of power operations around the clock (more than 8 hours). Frequent charging or battery replacement seriously affects the actual user experience and work efficiency. In addition, it is too dependent on the network. Continuous video streaming requires extremely high network bandwidth. In remote mountainous areas, underground pipe corridors, substations and other environments with poor signal coverage, video stuttering and interruption occur frequently, resulting in loss of monitoring data, delayed early warning, and a significant reduction in the actual availability of the equipment.

[0006] The second type of approach lacks scene perception capabilities. Existing solutions mostly collect data indiscriminately around the clock, failing to intelligently identify whether personnel have entered the work area or what stage of work they are in. They collect and transmit data at a fixed frequency, which not only wastes energy but also generates a large amount of redundant data, increasing the burden on backend analysis.

[0007] Therefore, there is an urgent need for a lightweight system and method based on image capture and multimodal AI recognition, which can monitor and analyze the operation process of multi-source data, perform multi-faceted scene perception, and reduce power consumption and network dependence, thus solving the problems of existing technologies. Summary of the Invention

[0008] One of the objectives of this invention is to provide a lightweight system based on image capture and multimodal AI recognition, which can monitor and analyze the operation process of multi-source data, perform multi-faceted scene perception, and reduce power consumption and network dependence.

[0009] The basic solution provided by this invention is a lightweight system based on image capture and multimodal AI recognition, comprising: a terminal and a cloud;

[0010] The terminal includes: a main control module, a monitoring module, a data acquisition module, and a communication module;

[0011] The monitoring module is used to monitor motion information, location information, and environmental information;

[0012] The acquisition module is used to acquire sound and image information;

[0013] The communication module is used to upload motion information, environmental information, sound information, and image information to the cloud;

[0014] The main control module controls the start-up, shutdown, and operational status of the monitoring, data acquisition, and communication modules. It employs a two-level wake-up mechanism, including:

[0015] First-level wake-up step: Start the monitoring module, and determine whether the preset work area buffer has been entered by receiving motion and positioning information collected by the monitoring module. If yes, execute the second-level wake-up step; otherwise, continue to execute the first-level wake-up step.

[0016] Second-level wake-up step: Start the acquisition module, periodically acquire image information at the first preset acquisition frequency, and upload it to the cloud through the communication module;

[0017] The cloud is used to analyze the current scene based on image information. If the current scene is a preset working scene, the control terminal enters the working mode. In the working mode, the main control module controls the acquisition module to periodically collect sound information and image information through the second preset acquisition frequency. The motion information and environmental information collected by the monitoring module are then uploaded to the cloud through the communication module.

[0018] In the cloud, it is also used to analyze the operation status and risk level based on uploaded image, sound, motion and environmental information, and trigger the terminal to issue an early warning based on the operation status and risk level.

[0019] Furthermore, the cloud platform is also used to perform state analysis on motion information to obtain the motion status of personnel; wherein the motion status of personnel includes: stationary, walking, and climbing.

[0020] The second preset sampling frequency is adjusted according to the movement status of the personnel, including:

[0021]

[0022] in The adjusted current second preset sampling frequency; For adjustment coefficients, It is a motion factor, determined based on the movement status of the personnel.

[0023] Furthermore, the cloud-based deployment includes multimodal AI models, such as: object detection model, text conversion model, speech emotion recognition model, multimodal fusion layer, job status analysis model, and risk level analysis model.

[0024] The object detection model detects objects in image information, outputs object bounding boxes and class confidence scores, and extracts scene semantic feature vectors. and output the corresponding scene confidence score. The objectives include those related to power facilities and those related to personnel behavior.

[0025] The text conversion model converts audio information into text, performs keyword matching, and outputs text features. ;

[0026] Speech emotion recognition models detect emotions in speech information, output emotion labels and emotion confidence scores, and form emotion features. .

[0027] Image features, text features, and sentiment features are concatenated and input into an LSTM temporal network to perform job status analysis and output the probability distribution of the current job status, including but not limited to: waiting, ready, operating, and finished.

[0028] The risk level analysis model is used to analyze the risk level based on the violations and operational status of the detected targets.

[0029] Furthermore, the aforementioned risk level These include: Level 1 risk, Level 2 risk, Level 3 risk, and Level 4 risk.

[0030] Level 1 risk This indicates that there were no violations and the work was carried out in accordance with regulations;

[0031] Level 2 risk This indicates a minor traffic violation;

[0032] Level 3 risk This indicates a general violation;

[0033] Level 4 risk This indicates a serious violation;

[0034] A base weight is assigned to each type of violation using a predefined database of violations. ;

[0035] Different work conditions have different tolerance levels for violations; therefore, risk coefficients are defined for different work conditions. ;

[0036] If the scene confidence score output by the object detection model is lower than the confidence score threshold, the violation will be ignored.

[0037] If the number of violations committed by the same user within the predicted detection period exceeds a preset threshold, the risk level should be accumulated, and a corresponding penalty factor should be set. ;

[0038] The weighted risk value method is used to analyze the risk level based on the violations and operational status of the detected targets, including:

[0039] For each violation currently identified Calculate the individual risk value: ;

[0040] Take the highest single-item risk value and adjust it based on the risk coefficient and penalty factor: ;

[0041] Map the maximum individual risk value to discrete levels:

[0042] ;

[0043] in This represents the threshold corresponding to the risk level.

[0044] Furthermore, the cloud platform is also used to extract voltage levels from work orders. Calculate the near-field warning distance :

[0045] ;

[0046] It is also used to determine that no work is being done at height if the personnel's movement status is not detected as being at height and the work status is not being operated, and to set a height warning for working at height. rice;

[0047] It is also used to push operational status, risk level, voltage level, proximity warning distance, and height warning altitude to the terminal;

[0048] The terminal is used to issue warnings based on one or more of the following information: operational status, risk level, voltage level, proximity warning distance, and height warning altitude.

[0049] Furthermore, the terminal also includes: a storage module;

[0050] The storage module is used to store motion information, environmental information, sound information, and image information;

[0051] It is also used to store operation status, risk level, voltage level, proximity warning distance, and height warning height; among which the proximity warning distance and height warning height are used as current thresholds and compared with the actual distance and relative height, respectively. If the actual distance is less than the proximity warning distance and continues for a preset time, an alarm is triggered; if the relative height is greater than the height warning height, an alarm is triggered.

[0052] Furthermore, the terminal is also used to determine whether the wearer has fallen based on motion information; if so, the information is uploaded to the cloud.

[0053] In the cloud, the judgment results are also used to combine the collected motion information, sound information and image information to analyze the specific location and type of injury of the wearer, and to provide movement suggestions, which are then pushed to the terminal.

[0054] Furthermore, the terminal calculates the composite acceleration and composite angular velocity from the acceleration and angular velocity in the motion information;

[0055] Based on acceleration and angular velocity, a fall detection model is used to detect whether the wearer of the terminal falls.

[0056] The fall detection model is a two-level triggering model. The first level performs a rapid threshold judgment: if the synthesized acceleration exceeds the preset acceleration threshold, the second level is performed; or the current attitude angle is calculated, and if the attitude angle changes more than the preset angle within the preset time interval after the impact, the second level is performed.

[0057] The second stage involves machine learning to determine whether the impact was a real fall.

[0058] If the classification result is "fall down", then emergency mode will be triggered immediately:

[0059] High-frequency image and continuous sound acquisition are performed on all terminals within the preset distance range of the terminal worn by the person who fell.

[0060] Upload motion, image, and sound information of the most recent preset time period to the cloud.

[0061] Furthermore, the cloud-based system extracts image pose features and target detection results based on image information;

[0062] Image pose features, target detection results, audio features, and motion features are concatenated into a vector, which is then input into a multimodal fusion network to output the injured location and injury type.

[0063] In the cloud, there is a pre-set rescue knowledge base. The injured part, injury type and location information are used as input, and the knowledge base rules are matched to generate suggested text, which is pushed to the terminal for broadcast to assist in the rescue.

[0064] The second objective of this invention is to provide a lightweight method based on image capture and multimodal AI recognition, which can comprehensively monitor and analyze the operation process of multi-source data, perform multi-faceted scene perception, and reduce power consumption and network dependence.

[0065] The present invention provides a second basic solution: a lightweight method based on image capture and multimodal AI recognition, which adopts the aforementioned lightweight system based on image capture and multimodal AI recognition.

[0066] Beneficial effects: This solution employs a two-level wake-up mechanism. In the first-level wake-up step, the monitoring module is activated. By receiving motion and positioning information collected by the monitoring module, it determines whether the system has entered the preset work area buffer zone. If so, the second-level wake-up step is executed; otherwise, the first-level wake-up step continues. This avoids wasted data collection in non-working areas.

[0067] In the second-level wake-up step, the acquisition module is activated, and image information is periodically acquired at the first preset acquisition frequency and uploaded to the cloud through the communication module. The cloud analyzes the current scene based on the image information. If the current scene is a preset working scene, the control terminal enters the working mode. In the working mode, the main control module controls the acquisition module to periodically acquire sound information and image information at the second preset acquisition frequency, along with motion information and environmental information acquired by the monitoring module, and uploads them to the cloud through the communication module.

[0068] By adopting key image capture technology to replace traditional video recording methods, the device's power consumption is effectively reduced, solving the problem of high power consumption and alleviating the pressure on channel transmission. This allows the device to operate stably even in environments with poor network conditions, overcoming the shortcomings of poor network adaptability in existing smart safety helmets. At the same time, the lightweight design reduces interference with operators, improves their user experience, and increases actual work efficiency.

[0069] In particular, this solution effectively reduces device power consumption and solves the problem of high power consumption, thereby enabling lightweight design during hardware implementation. In particular, the battery design does not simply involve selecting a small-capacity battery, but rather, through the synergy of system-level power consumption optimization and battery selection, the battery weight is reduced as much as possible while ensuring battery life.

[0070] In the cloud, it is also used to analyze the operation status and risk level based on uploaded image, sound, motion and environmental information, triggering the terminal to issue early warnings based on the operation status and risk level. This improves the on-site operation safety monitoring and malicious violation detection capabilities, fills the gaps in the monitoring of some violations that existing equipment cannot cover, promotes the development of power operation safety monitoring technology towards a more intelligent and accurate direction, and plays a demonstrative and leading role in the safety technology upgrade of the entire industry.

[0071] In summary, this solution can comprehensively monitor and analyze the operation process using multi-source data, perform multi-faceted scenario perception, and reduce power consumption and network dependence. Attached Figure Description

[0072] Figure 1 This is a logic block diagram of an embodiment of the lightweight system based on image capture and multimodal AI recognition of the present invention;

[0073] Figure 2 This is a logic block diagram of the terminal in an embodiment of the lightweight system based on image capture and multimodal AI recognition of the present invention. Detailed Implementation

[0074] The following detailed description illustrates the specific implementation methods:

[0075] Example 1

[0076] This embodiment provides a lightweight system based on image capture and multimodal AI recognition, as shown in the attached figure. Figure 1 As shown, this includes: terminals and the cloud;

[0077] In this embodiment, the terminal is a smart safety helmet, including: a main control module, a monitoring module, a data acquisition module, a communication module, and a storage module, such as... Figure 2 As shown, all of these are installed on the smart safety helmet;

[0078] The monitoring module, data acquisition module, communication module, and storage module are all connected to the main control module;

[0079] The main control module is used to control the start-up, shutdown, and working status of the monitoring module, data acquisition module, communication module, and storage module; in this embodiment, the main control module is a CPU.

[0080] The monitoring module is used to monitor motion information, positioning information, and environmental information. This monitoring module includes, but is not limited to, a motion sensor, a barometric pressure sensor, and a positioning sensor, which are used to collect motion information, ambient air pressure, and positioning information, respectively. It is connected to the main control module via a sensor bus and also includes a lithium battery and a battery management unit for power supply and management. In this embodiment, the motion sensor uses an IMU (accelerometer and gyroscope), and the positioning sensor uses a GPS.

[0081] The acquisition module is used to acquire sound and image information; the acquisition module includes, but is not limited to, a microphone and a camera, which are connected to the main control module through the IIS interface and the DVP interface, respectively; in this embodiment, a silicon microphone is used.

[0082] The communication module is used to upload motion information, environmental information, sound information, and image information to the cloud; the communication module communicates with the CPU via radio frequency; in this embodiment, the communication module includes, but is not limited to, WIFI and Bluetooth;

[0083] The storage module is used to store motion information, positioning information, environmental information, sound information, and image information; the storage module is connected to the main control module via a storage bus and includes PSRAM and FLASH.

[0084] The main control module is used to control the start-up, shutdown, and working status of the monitoring module, data acquisition module, communication module, and storage module.

[0085] Specifically, the main control module is equipped with a two-level wake-up mechanism, including:

[0086] The first-level wake-up step involves activating the monitoring module. By receiving motion and location information collected by the monitoring module, it determines whether the user has entered a preset work area buffer zone. If yes, the second-level wake-up step is executed; otherwise, the first-level wake-up step continues. In this embodiment, the motion sensor collects motion information at a frequency of 1Hz and calculates the motion intensity. ,in To synthesize the acceleration variance; the positioning module performs positioning every 10 minutes to obtain location information, i.e., location coordinates. ;like (Preset threshold), or If the data falls within the preset work area buffer zone, the second-level wake-up step is executed, triggering the initial wake-up acquisition module.

[0087] The second-level wake-up step involves activating the acquisition module, periodically acquiring image information at a first preset acquisition frequency, and uploading it to the cloud via the communication module. In this embodiment, the first preset acquisition frequency... Once per minute (every 5 minutes), after waking up, the acquisition module takes a low-resolution image (320×240), compresses it, and uploads it to the cloud via the communication module, along with the current motion information and location information.

[0088] A lightweight scene analysis model is deployed in the cloud to analyze the current scene based on image information. If the current scene is a preset work scene, the control terminal enters the work mode and queries the work ticket system for a work ticket. If no work ticket is found, it is marked as no task to perform. In this embodiment, the lightweight scene analysis model uses the MobileNetV3-SSD model to perform fast inference on the image information, identify whether it contains power facility elements (such as: iron towers, power poles, ring main units, safety fences, etc.), and outputs the confidence level of containing power facility elements. , ,like If the current scenario is determined to be a preset work scenario, an instruction to enter the work area is sent to the main control module; upon receiving the instruction to enter the work area, the main control module switches to work mode.

[0089] In working mode, the main control module controls the acquisition module to periodically collect sound and image information at a second preset acquisition frequency. This information, along with the motion and environmental information collected by the monitoring module, is then uploaded to the cloud via the communication module. Simultaneously, the cloud performs status analysis on the motion information to obtain the personnel's motion status.

[0090] In this embodiment, the initial value of the second preset sampling frequency is Times / minute; personnel movement status, including but not limited to: stationary, walking, climbing.

[0091] The second preset sampling frequency is adjusted according to the movement status of the personnel, including:

[0092]

[0093] in The adjusted second preset sampling frequency, i.e., the current sampling frequency; This is the adjustment factor (usually taken as 0.5). As a motion factor, it is determined based on the person's movement state. If the person's movement state is climbing, then... If the person's movement is walking, then If the person's movement is stationary, then This ensures that within the work area, the more vigorous the movements, the more frequent the data collection.

[0094] Multimodal AI models are deployed in the cloud to analyze the operation status and risk level based on uploaded image, sound, motion, and environmental information.

[0095] The uploaded data includes a unique terminal ID for each terminal.

[0096] The multimodal AI model includes: object detection model, text conversion model, speech emotion recognition model, multimodal fusion layer, job status analysis model, and risk level analysis model. In this embodiment, the object detection model adopts the YOLOv8 model, the text conversion model adopts the Whisper model, the speech emotion recognition model adopts EmoNet, and the job status adopts the LSTM temporal network.

[0097] The object detection model detects objects in image information, outputs object bounding boxes and class confidence scores, and extracts scene semantic feature vectors. and output the corresponding scene confidence score. (0~1); the targets include power facility targets and personnel behavior targets; power facility targets include, but are not limited to: iron towers, power poles, transformers, insulators, and live signs; personnel behavior targets include, but are not limited to: wearing safety helmets, approaching live objects, climbing, and using tools.

[0098] The text conversion model converts audio information into text, performs keyword matching, and outputs text features. (BERT encoding); keywords include, but are not limited to: operation ticket, grounding, voltage testing.

[0099] The speech emotion recognition model detects emotions in speech information, such as tension, fright, and other abnormal emotions, and outputs emotion labels and emotion confidence scores to form emotion features. .

[0100] Image features, text features, and sentiment features are concatenated and input into an LSTM temporal network to perform job state analysis and output the probability distribution of the current job state, including but not limited to: waiting, ready, operating, and finished.

[0101] Based on the work orders retrieved from the work order system (such as work schedule time, voltage level, and whether the work is energized), the work status judgment is corrected.

[0102] The risk level analysis model is used to analyze the risk level based on violations (such as not wearing a safety helmet) and work status in the detected targets (personnel behavior targets);

[0103] Specifically, this embodiment sets four risk levels. These include: Level 1 risk, Level 2 risk, Level 3 risk, and Level 4 risk.

[0104] Level 1 risk This indicates that there were no violations and the work was carried out in accordance with regulations. Correspondingly, only records will be made and no alarm will be triggered.

[0105] Level 2 risk This indicates a minor traffic violation, and a corresponding voice prompt will be given to remind you.

[0106] Level 3 risk This indicates a general violation, for which a voice warning will be issued and the violation will be recorded in the cloud.

[0107] Level 4 risk This indicates a serious violation, and in response, an audible and visual alarm will be triggered, a notification will be sent to the cloud, and the machine will be forced to shut down, meaning it will stop working.

[0108] A base weight is assigned to each type of violation using a predefined database of violations. (0~1), as shown in Table 1 below:

[0109] Table 1: Weight Allocation Table for Violations

[0110]

[0111] Different work conditions have different tolerance levels for violations; therefore, risk coefficients are defined for different work conditions. Specifically, the risk coefficient for the waiting-to-work state is 0.2, which is a low-risk non-operation state; the risk coefficient for the preparation-to-work state is 0.5, which involves tool inspection and safety measure setup; the risk coefficient for the operation-to-work state is 1.0, which is the actual operation and has the highest risk; and the risk coefficient for the ending-to-work state is 0.3, which is the final stage and has a decreasing risk.

[0112] Scene confidence score output by the object detection model (0~1), if it is below the confidence threshold (e.g., 0.6), it is ignored.

[0113] If the number of violations committed by the same user within the predicted detection period exceeds a preset threshold, the risk level should be accumulated, and a corresponding penalty factor should be set. In this embodiment This refers to the number of traffic violations within the last 30 minutes, with a maximum of 3.

[0114] The weighted risk value method is used to analyze the risk level based on the violations and operational status of the detected targets. The specific process is as follows:

[0115] For each violation currently identified Calculate the individual risk value: ;

[0116] Take the highest single-item risk value and adjust it based on the risk coefficient and penalty factor: ;

[0117] Map the maximum individual risk value to discrete levels:

[0118] ;

[0119] in The threshold values ​​corresponding to the risk levels are set to 0.2, 0.4, and 0.7 in this embodiment.

[0120] The cloud is also used to extract voltage levels from work orders. (kV), calculate the near-field warning distance :

[0121]

[0122] The distance requirements in the State Grid safety regulations were fitted: 10kV→0.7m, 110kV→1.5m.

[0123] It is also used to determine that no work is being done at height if the personnel's movement status is not detected as being at height and the work status is not being operated, and to set a height warning for working at height. rice.

[0124] It is also used to encapsulate the operation status, risk level, voltage level, near-electricity warning distance, and high-altitude warning height into JSON and push it to the terminal via MQTT. In other embodiments, the operation status, risk level, voltage level, near-electricity warning distance, and high-altitude warning height are converted into voice prompt text, encapsulated into JSON, and pushed to the terminal via MQTT.

[0125] The terminal is used to issue warnings based on one or more of the following information: work status, risk level, voltage level, proximity warning distance, and height warning.

[0126] Specifically, warnings are issued according to the risk level, and the operation status, risk level, and voltage level are broadcast or displayed;

[0127] The system stores the operation status, risk level, voltage level, proximity warning distance, and height warning altitude in the storage module. The proximity warning distance and height warning altitude serve as current thresholds, which are compared with the actual distance and relative height, respectively. If the actual distance is less than the proximity warning distance and remains so for a preset time (e.g., 1 second), an alarm is triggered, such as a voice alarm saying "Approaching a live conductor, please move away immediately." If the relative height is greater than the height warning altitude, an alarm is triggered, such as an alarm saying "Height risk, please fasten your safety belt." In this embodiment, the monitoring module is equipped with an electric field strength sensor to measure the electric field strength, converting it to the actual distance through a calibration curve, and outputting the relative height in real time based on the ambient air pressure.

[0128] In other embodiments, the storage module stores the latest rules (voltage level-distance mapping table, height threshold, etc.) in the terminal flash memory, and uses the most recent valid rule when the network is disconnected; at the same time, the main control module can also run a lightweight image classification model (such as MobileNet) to make a preliminary judgment on the captured image locally (such as whether someone is approaching a live object) and issue an early warning when there is no network.

[0129] In addition, the terminal also includes a power module. In this embodiment, the power module uses a lithium cobalt oxide polymer battery, which has high energy density (typical value 250 Wh / kg), small size, and can be customized to fit the internal space. It can further utilize a multi-layer stacked battery packaged with flexible printed circuit boards (FPC), with a thickness controllable to 3-4mm and a weight of approximately 25-30g, which can be embedded within the smart module. Compared to existing smart safety helmets that mostly use fixed-capacity batteries (such as 5000mAh, weighing approximately 120g), relying on large batteries to "harden" power consumption, resulting in a heavy overall device, this solution reduces battery weight while maintaining the same battery life through a combination of lightweight batteries and system-level power consumption optimization (adjustment of key image capture and acquisition frequency), improving wearing comfort and operational flexibility.

[0130] Example 2

[0131] This embodiment is basically the same as the above embodiment, except that: in actual work, although the wearer is wearing a smart safety helmet, accidents such as falls may still occur. If the fall is serious, it may cause problems such as bone fractures, sprains, etc. Other people blindly moving may cause secondary injuries to the injured person. The smart safety helmet will collect data such as motion information and image information. After the wearer falls, the injury location can be initially obtained through the analysis of the motion trajectory, so as to provide assistance for the treatment of the injured person. Specifically, the terminal is also used to determine whether the wearer has fallen based on the motion information. If so, it will be uploaded to the cloud.

[0132] In the cloud, the judgment results are also used to combine the collected motion information, sound information and image information to analyze the specific location and type of injury of the wearer, and to provide movement suggestions, which are then pushed to the terminal.

[0133] The specific process is as follows:

[0134] The terminal performs sliding window filtering (such as median filtering) on ​​the acceleration and angular velocity in the motion information to remove noise and calculates the composite acceleration. and the resultant angular velocity ;

[0135] Based on acceleration and angular velocity, a fall detection model is used to detect whether the wearer of the terminal falls.

[0136] The fall detection model is a two-level triggering model. The first level performs a rapid threshold judgment: if the synthesized acceleration exceeds the preset acceleration threshold, such as 4g (a typical fall impact can reach 6-10g), then the second level is performed; or the current attitude angle (pitch angle, roll angle) is calculated, and if the attitude angle changes more than the preset angle (such as 60°) within a preset time interval (such as 2 seconds) after the impact, then the second level is performed.

[0137] The second level involves machine learning analysis: extracting motion features within a preset time window after the impact, including: peak acceleration, trough acceleration, mean acceleration, variance, integral of angular velocity (angle change), difference in attitude angle before and after the impact, and duration of free fall (when...). When the weight is less than 0.5g, input a lightweight classifier, such as a decision tree or random forest, to determine whether it is a real fall (distinguishing it from everyday actions such as bending over or jumping); the classifier runs on the terminal, such as using TensorFlow Lite Micro.

[0138] If the classification result is "fall down", then emergency mode will be triggered immediately:

[0139] All terminals within the preset distance range of the terminal worn by the person who fell were used to acquire high-frequency images (e.g., 5 images / second) and continuous sound (e.g., 16kHz).

[0140] Upload motion, image, and sound information for the most recent preset time period (e.g., 10 seconds) to the cloud.

[0141] In the cloud, based on image information, image pose features and target detection results are extracted;

[0142] Specifically, a lightweight pose estimation model (such as OpenPose) is used to analyze the image information captured after the fall and extract the coordinates of key points, including: head, shoulder, elbow, wrist, hip, knee and ankle.

[0143] Calculate the angles and spatial relationships between key points to identify abnormal postures as image pose features, such as: abnormal head tilt, which may indicate cervical spine injury; abnormal arm / leg bending, which may indicate fracture or dislocation; body curling up, which may indicate abdominal injury; inability to stand / move, which may indicate lower limb or spinal injury.

[0144] Object detection models (such as YOLOv8) are used to obtain object detection results, such as detecting bloodstains, signs of fractures (such as limb deformities), tools or obstacles on the ground; other embodiments also use semantic segmentation models to identify the surrounding environment (such as electrical equipment, ditches, stairs) to determine the cause of the fall and secondary risks.

[0145] Based on the sound information, extract audio features, including:

[0146] The ASR model (such as Whisper) is used to transcribe sound information and extract keywords for distress calls (such as "help" and "pain"). Emotional characteristics such as pain and fear are identified through acoustic features (MFCC, prosody). Sounds of falls, impacts, metallic collisions, and electrical noises are detected to help determine the context of the fall. All of these features are audio features.

[0147] Image pose features, target detection results, audio features, and motion features are concatenated into a vector, which is then input into a multimodal fusion network, such as the Transformer fusion layer. The output is the injured location and injury type.

[0148] Injured areas: head, neck, shoulder, arm, waist, leg, etc. (multi-label classification).

[0149] Injury type: fracture, bleeding, sprain, loss of consciousness, unresponsiveness, etc. (multi-label classification).

[0150] Data can be obtained by simulating fall experiments.

[0151] In the cloud, there is a pre-set rescue knowledge base. The injured part, injury type and location information are used as input, and the knowledge base rules are matched to generate suggested text, which is pushed to the terminal for broadcast to assist in the rescue.

[0152] The terminal detects falls in real time, and the cloud performs in-depth analysis of injuries, balancing real-time performance with analytical accuracy. It makes full use of complementary information from images, sound, and IMU to improve the accuracy of injury assessment. The rescue knowledge base generates actionable movement suggestions with interpretability.

[0153] The above descriptions are merely embodiments of the present invention. Commonly known structures and characteristics are not described in detail here. Those skilled in the art are aware of all common technical knowledge in the field prior to the application date or priority date, are aware of all existing technologies in that field, and have the ability to apply conventional experimental methods prior to that date. Those skilled in the art can, under the guidance of this application, improve and implement this solution in combination with their own capabilities. Some typical known structures or methods should not be obstacles for those skilled in the art to implement this application. It should be noted that those skilled in the art can make several modifications and improvements without departing from the structure of the present invention. These should also be considered within the scope of protection of the present invention, and will not affect the effectiveness of the implementation of the present invention or the practicality of the patent. The scope of protection claimed in this application should be determined by the content of its claims, and the specific embodiments described in the specification can be used to interpret the content of the claims.

Claims

1. A lightweight system based on image capture and multimodal AI recognition, characterized in that, include: Terminal and cloud; The terminal includes: a main control module, a monitoring module, a data acquisition module, and a communication module; The monitoring module is used to monitor motion information, location information, and environmental information; The acquisition module is used to acquire sound and image information; The communication module is used to upload motion information, environmental information, sound information, and image information to the cloud; The main control module controls the start-up, shutdown, and operational status of the monitoring, data acquisition, and communication modules. It employs a two-level wake-up mechanism, including: First-level wake-up step: Start the monitoring module, and determine whether the preset work area buffer has been entered by receiving motion and positioning information collected by the monitoring module. If yes, execute the second-level wake-up step; otherwise, continue to execute the first-level wake-up step. Second-level wake-up step: Start the acquisition module, periodically acquire image information at the first preset acquisition frequency, and upload it to the cloud through the communication module; The cloud is used to analyze the current scene based on image information. If the current scene is a preset working scene, the control terminal enters the working mode. In the working mode, the main control module controls the acquisition module to periodically collect sound information and image information through the second preset acquisition frequency. The motion information and environmental information collected by the monitoring module are then uploaded to the cloud through the communication module. In the cloud, it is also used to analyze the operation status and risk level based on uploaded image, sound, motion and environmental information, and trigger the terminal to issue an early warning based on the operation status and risk level.

2. The lightweight system based on image capture and multimodal AI recognition according to claim 1, characterized in that, The cloud platform is also used to perform state analysis on motion information and obtain the motion status of personnel. The movement states of people include: stationary, walking, and climbing. The second preset sampling frequency is adjusted according to the movement status of the personnel, including: in The adjusted current second preset sampling frequency; This indicates the second preset sampling frequency. For adjustment coefficients, It is a motion factor, determined based on the movement status of the personnel.

3. The lightweight system based on image capture and multimodal AI recognition according to claim 1, characterized in that, The cloud-based deployment includes multimodal AI models, such as: object detection model, text conversion model, speech emotion recognition model, multimodal fusion layer, job status analysis model, and risk level analysis model. The object detection model detects objects in image information, outputs object bounding boxes and class confidence scores, and extracts scene semantic feature vectors. and output the corresponding scene confidence score. The objectives include those related to power facilities and those related to personnel behavior. The text conversion model converts audio information into text, performs keyword matching, and outputs text features. ; Speech emotion recognition models detect emotions in speech information, output emotion labels and emotion confidence scores, and form emotion features. .

4. The lightweight system based on image capture and multimodal AI recognition according to claim 3, characterized in that, The risk level These include: Level 1 risk, Level 2 risk, Level 3 risk, and Level 4 risk. Level 1 risk This indicates that there were no violations and the work was carried out in accordance with regulations; Level 2 risk This indicates a minor traffic violation; Level 3 risk This indicates a general violation; Level 4 risk This indicates a serious violation; A base weight is assigned to each type of violation using a predefined database of violations. ; Different work conditions have different tolerance levels for violations; therefore, risk coefficients are defined for different work conditions. ; If the scene confidence score output by the object detection model is lower than the confidence score threshold, the violation will be ignored. If the number of violations committed by the same user within the predicted detection period exceeds a preset threshold, the risk level should be accumulated, and a corresponding penalty factor should be set. ;in To predict the number of violations within the detection period; The weighted risk value method is used to analyze the risk level based on the violations and operational status of the detected targets, including: For each violation currently identified Calculate the individual risk value: ; Take the highest single-item risk value and adjust it based on the risk coefficient and penalty factor: ; Map the maximum individual risk value to discrete levels: ； in The threshold corresponding to the risk level.

5. The lightweight system based on image capture and multimodal AI recognition according to claim 4, characterized in that, The cloud platform is also used to extract voltage levels from work orders. Calculate the near-field warning distance : ； It is also used to determine that no work is being done at height if the personnel's movement status is not detected as being at height and the work status is not being operated, and to set a height warning for working at height. rice; It is also used to push operational status, risk level, voltage level, proximity warning distance, and height warning altitude to the terminal; The terminal is used to issue warnings based on one or more of the following information: operational status, risk level, voltage level, proximity warning distance, and height warning altitude.

6. The lightweight system based on image capture and multimodal AI recognition according to claim 5, characterized in that, The terminal also includes: a storage module; The storage module is used to store motion information, environmental information, sound information, and image information; It is also used to store operation status, risk level, voltage level, proximity warning distance, and height warning height; among which the proximity warning distance and height warning height are used as current thresholds and compared with the actual distance and relative height, respectively. If the actual distance is less than the proximity warning distance and continues for a preset time, an alarm is triggered; if the relative height is greater than the height warning height, an alarm is triggered.

7. The lightweight system based on image capture and multimodal AI recognition according to claim 1, characterized in that, The terminal is also used to determine whether the wearer has fallen based on motion information; if so, the information is uploaded to the cloud. In the cloud, the judgment results are also used to combine the collected motion information, sound information and image information to analyze the specific location and type of injury of the wearer, and to provide movement suggestions, which are then pushed to the terminal.

8. The lightweight system based on image capture and multimodal AI recognition according to claim 7, characterized in that, The terminal calculates the composite acceleration and composite angular velocity from the acceleration and angular velocity in the motion information; Based on acceleration and angular velocity, a fall detection model is used to detect whether the wearer of the terminal falls. The fall detection model is a two-level triggering model. The first level performs a rapid threshold judgment: if the synthesized acceleration exceeds the preset acceleration threshold, the second level is performed; or the current attitude angle is calculated, and if the attitude angle changes more than the preset angle within the preset time interval after the impact, the second level is performed. The second stage involves machine learning to determine whether the impact was a real fall. If the classification result is "fall down", then emergency mode will be triggered immediately: High-frequency image and continuous sound acquisition are performed on all terminals within the preset distance range of the terminal worn by the person who fell. Upload motion, image, and sound information of the most recent preset time period to the cloud.

9. The lightweight system based on image capture and multimodal AI recognition according to claim 8, characterized in that, The cloud platform extracts image pose features and target detection results based on image information. Image pose features, target detection results, audio features, and motion features are concatenated into a vector, which is then input into a multimodal fusion network to output the injured location and injury type. In the cloud, there is a pre-set rescue knowledge base. The injured part, injury type and location information are used as input, and the knowledge base rules are matched to generate suggested text, which is pushed to the terminal for broadcast to assist in the rescue.

10. A lightweight method based on image capture and multimodal AI recognition, characterized in that, A lightweight system based on image capture and multimodal AI recognition using any one of claims 1-9.

Images